Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclusive jobs across sub process instances #4003

Closed
2 tasks done
Tracked by #316
danielkelemen opened this issue Dec 11, 2023 · 4 comments
Closed
2 tasks done
Tracked by #316

Exclusive jobs across sub process instances #4003

danielkelemen opened this issue Dec 11, 2023 · 4 comments
Assignees
@psavidis
Labels
type:feature Issues that add a new user feature to the project. version:7.21.0-alpha5 version:7.21.0

Comments

@danielkelemen
Copy link
Member

danielkelemen commented Dec 11, 2023 •
edited by psavidis

User Story (Required on creation)

As a developer, I want to use an exclusive continuation for tasks in a multi-instance sub-process use-case to ensure exclusive tasks are not executed in parallel.

Functional Requirements (Required before implementation)

  • Job executor acquisition is updated so that it can now work with the rootProcessInstanceId, allowing it to respect exclusiveness in tasks across sub-process instances.
  • A new engine configuration enables this extended exclusive behaviour for sub-processes.
  • The new behaviour is documented and tested.

Technical Requirements (Required before implementation)

See tickets.

Limitations of Scope

Hints

Links

Breakdown

Tasks
Beta Give feedback
  1. Extend job executor acquisition to work with rootProcessInstanceId #4004
    1 of 1
    scope:core-api type:subtask version:7.21.0 version:7.21.0-alpha5
    psavidis
  2. Document changes #4005
    1 of 1
    scope:documentation type:subtask version:7.21.0
    psavidis

Dev2QA handover

  • Does this ticket need a QA test and the testing goals are not clear from the description? Add a Dev2QA handover comment
@danielkelemen danielkelemen added type:feature Issues that add a new user feature to the project. potential:7.21.0 labels Dec 11, 2023
Open
@yanavasileva
Copy link
Member

I am interested in doing the review. :)

This was referenced Mar 5, 2024
Closed
Closed
@psavidis
Copy link
Contributor

psavidis commented Mar 15, 2024 •
edited

Notes for the Reviewer

  • Both the code and the documentation Pull Requests are ready for review.
  • I'd recommend to finish the review with the code first and then move on to the documentation, in order to avoid
    reference mismatches    , mainly between the new config and the documentation.
  • I've added conversations on the docs PR as reminders to be resolved after the code PR finishes, although i think the new configuration name already reflects very well its purpose, as suggested by @tasso94

Since @yanavasileva expressed already interest to conduct the code review, after agreeing also with @danielkelemen, it makes sense for another person to review the code besides the ones who took part in the design / implementation phases so i'm assigning the ticket to her for the code review 🚀

@psavidis psavidis assigned yanavasileva and unassigned psavidis Mar 15, 2024
psavidis added a commit that referenced this issue Mar 22, 2024
@psavidis
feat(engine): Support Exclusive Jobs across Process Hierarchies
94024a0 
Context: Exclusive jobs that originate from process hierarchies (processes which contain multi-instance subprocesses) can now be executed exclusively.

Why: The exclusive execution would only be applied on tasks that originate at a root level. Any subprocess spawned by root processes would not be correlated and considered for exclusive execution.

Changes: See below the changes of this feature

- New column `ROOT_PROC_INST_ID_ ` is introduced to correlated a process instance with its root parent
    - The column is added to all supported databases & `7.20_to_7.21` migration scripts.
- The query `selectNextJobsToExecute` is enriched to consider the root process instance id when the feature is enabled.
- The feature flag `jobExecutorAcquireExclusiveOverProcessHierarchies` enables / disables the feature.
- The feature is disabled by default for backwards compatibility with the legacy behaviour.

Tests:

- Unit tests under `ExclusiveJobAcquisitionTest` which cover:
    - Legacy behaviour with the feature disabled
    - Feature behaviour with the feature enabled & a 1-level deep process hierarchy (process which spawns a subprocess)
    - Feature behaviour with the feature enabled & a 2-level deep process hierarchy (process which spawns a subprocess - which spawns another subprocess)

- Migration tests under `ExclusiveOverProcessHierarchiesTest` which cover:
    - How the job acquisition and execution (see `AcquireJobsCmd`) behave with existing 7.20 process instances when the feature is disabled (legacy behaviour and backwards compatibility)
    - How the job acquisition and execution (see `AcquireJobsCmd`) behave with existing 7.20 process instances when the feature is enabled (feature behaviour)
    - The tests need to select all the jobs and lock them to perform their logic. After the test execution the jobs are unlocked.

Other Notable Changes: 

- Fixed flakiness of `DecisionDefinitionTest`
    - The Test class was susceptible to timezone changes due its test data using the current date; as a result, depending on the date of execution, changing to DST could fail the tests.
    - This behaviour is fixed by adjusting the test data to use a fixed past date, rendering the test executions immune to date of execution.
- The waiting behaviour on jobs for the tests to rely on the scheduler, shared by `SequentialJobAcquisitionTest`, `ExclusiveJobAcquisitionTest` is extracted into class `JobExecutorWaitUtils`

Co-authored-by: daniel.kelemen
Co-authored-by: yanavasileva
Co-authored-by: petros.savvidis

Related-to: #4004 , #4003
psavidis added a commit to camunda/camunda-docs-manual that referenced this issue Mar 22, 2024
@psavidis
docs(content): Add Exclusive Jobs of Process Hierarchies
18c4aca 
Context: Documentation on the following:

- How exclusive jobs of process hierarchies behave by default
- The feature that enables exclusive execution over hierarchies
- New feature flag configuration

The commit adds an image(png) that demonstrates the limited application of `exclusive` on a hierarchical process setup.

Co-authored-by: yanavasileva
Co-authored-by: petros.savvidis

Related-to:
- camunda/camunda-bpm-platform#4003
- camunda/camunda-bpm-platform#4005
@psavidis
Copy link
Contributor

Merged code & documentation to master. Closing the issue.

@psavidis
Copy link
Contributor

psavidis commented Apr 2, 2024

Notes for QA Testing

  • Process which contains hierarchy using a call stack and a subprocess exclusive over process hierarchies.zip
  • jobExecutorAcquireExclusiveOverProcessHierarchies feature flag enables / disables the feature (by default disabled for backwards compatibility sake of the legacy behaviour)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@psavidis psavidis

Labels
type:feature Issues that add a new user feature to the project. version:7.21.0-alpha5 version:7.21.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tasso94 @danielkelemen @yanavasileva @psavidis