Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Soft pause information is persisted across all replicas. #16869

Merged
merged 16 commits into from
Apr 3, 2024
Merged

Soft pause information is persisted across all replicas. #16869

merged 16 commits into from
Apr 3, 2024

Conversation

rodrigo-lourenco-lopes
Copy link
Contributor

@rodrigo-lourenco-lopes rodrigo-lourenco-lopes commented Mar 14, 2024 •
edited

Description

This feature enables the soft pause state to persist in all the partitions replicas even when these are restarted.

Related issues

closes #16642

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

  • The changes are backwards compatibility with previous versions
  • If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

  • There are unit/integration tests that verify all acceptance criterias of the issue
  • New tests are written to ensure backwards compatibility with further versions
  • The behavior is tested manually
  • The change has been verified by a QA run
  • The impact of the changes is verified by a benchmark

Documentation:

  • The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
  • If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

Please refer to our review guidelines.

@github-actions github-actions bot added the component/zeebe Related to the Zeebe component/team label Mar 14, 2024
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes marked this pull request as ready for review March 14, 2024 10:29
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes force-pushed the rl-persist-soft-pause-info branch 2 times, most recently from 08c35bc to 5067068 Compare March 14, 2024 12:54
@rodrigo-lourenco-lopes
test: add test to assert that paused state is persisted after broker …
9b09ee8 
…shutdown

The same test for soft pause will be added once we expose the
method on the gateway.
deepthidevaki
deepthidevaki requested changes Mar 15, 2024
View reviewed changes
Copy link
Contributor

@deepthidevaki deepthidevaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Rodrigo. 🚀

❌ This PR still not address my comment here

The exporter director should be able to start with phase soft-paused if it was soft-paused before restart.

Writing the state to the file is not an ideal solution. But I want to revisit this approach in general in a different issue anyway. So let's keep your solution. But please address my comments.

❌ Now that it has more logic, a unit test for PartitionProcessingState would be good. Could you please add it?

...r/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingState.java Outdated Show resolved Hide resolved
...r/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingState.java Outdated Show resolved Hide resolved
...r/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingState.java Outdated Show resolved Hide resolved
...r/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingState.java Outdated Show resolved Hide resolved
...e/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java Outdated Show resolved Hide resolved
deepthidevaki
deepthidevaki requested changes Apr 2, 2024
View reviewed changes
Copy link
Contributor

@deepthidevaki deepthidevaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for adding the tests. Looks good.

❌ You should use a temp directory in the tests for the pause state files.

...r/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingState.java Outdated Show resolved Hide resolved
...c/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java Outdated Show resolved Hide resolved
...c/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java Outdated Show resolved Hide resolved
...c/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java Outdated Show resolved Hide resolved
...c/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java Show resolved Hide resolved
...c/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java Outdated Show resolved Hide resolved
...e/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java Show resolved Hide resolved
...e/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java Show resolved Hide resolved
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes added this pull request to the merge queue Apr 3, 2024
Merged via the queue into main with commit fa8db29 Apr 3, 2024
31 checks passed
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes deleted the rl-persist-soft-pause-info branch April 3, 2024 10:35
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes added backport stable/8.2 Backport a pull request to 8.2.x backport stable/8.3 Backport a pull request to 8.3.x backport stable/8.4 Backport a pull request to 8.4.x backport stable/8.5 Backport a pull request to stable/8.5 labels Apr 25, 2024
@rodrigo-lourenco-lopes
Copy link
Contributor Author

/backport

@backport-action
Copy link
Collaborator

Backport failed for stable/8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.2
git worktree add -d .worktree/backport-16869-to-stable/8.2 origin/stable/8.2
cd .worktree/backport-16869-to-stable/8.2
git switch --create backport-16869-to-stable/8.2
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

@backport-action
Copy link
Collaborator

Backport failed for stable/8.3, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.3
git worktree add -d .worktree/backport-16869-to-stable/8.3 origin/stable/8.3
cd .worktree/backport-16869-to-stable/8.3
git switch --create backport-16869-to-stable/8.3
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

@backport-action
Copy link
Collaborator

Backport failed for stable/8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.4
git worktree add -d .worktree/backport-16869-to-stable/8.4 origin/stable/8.4
cd .worktree/backport-16869-to-stable/8.4
git switch --create backport-16869-to-stable/8.4
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

@backport-action backport-action mentioned this pull request Apr 25, 2024
Merged
@backport-action
Copy link
Collaborator

Successfully created backport PR for stable/8.5:

github-merge-queue bot pushed a commit that referenced this pull request Apr 25, 2024
@camundait
[Backport stable/8.5] Soft pause information is persisted across all …
0a22722 
…replicas. (#17814)

# Description
Backport of #16869 to `stable/8.5`.

relates to #16642
original author: @rodrigo-lourenco-lopes
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes removed the backport stable/8.5 Backport a pull request to stable/8.5 label Apr 26, 2024
@rodrigo-lourenco-lopes
Copy link
Contributor Author

/backport

@backport-action
Copy link
Collaborator

Backport failed for stable/8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.2
git worktree add -d .worktree/backport-16869-to-stable/8.2 origin/stable/8.2
cd .worktree/backport-16869-to-stable/8.2
git switch --create backport-16869-to-stable/8.2
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

@backport-action
Copy link
Collaborator

Backport failed for stable/8.3, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.3
git worktree add -d .worktree/backport-16869-to-stable/8.3 origin/stable/8.3
cd .worktree/backport-16869-to-stable/8.3
git switch --create backport-16869-to-stable/8.3
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

@backport-action
Copy link
Collaborator

Backport failed for stable/8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.4
git worktree add -d .worktree/backport-16869-to-stable/8.4 origin/stable/8.4
cd .worktree/backport-16869-to-stable/8.4
git switch --create backport-16869-to-stable/8.4
git cherry-pick -x a9c59270030f732301fe63043fc649844caaff7b 9b09ee83e2a60172b3e912d6d111839e24ccb1c5 da57c7ec5fa8ba83a60e12b8fbce15e5a7d3a58e 93eb1d9fb1eda64d595ed589f791773c855553cb ab027d4ae0dbfcce11166703e68649df27229711 6f40b0b966e50a37ce41d598c473c82fdfae7aad 7be1e7721eac484cec82c9d4e9bd2673dd1100b4 0082f9728163a75e2edf40cb1dcef39f3c0ccec1 074170a773b97bcdc3cc75dce99d497553813c48 e34119606ed4873806b9c9d3bfa1828e401eed72 2b023def66d54c6133e58ac6252e59cc78194236 3442d2ae036686d9518aa7b9f5c174667fddd6b6 5c2cad67a8338fa49529909d27c7584fa6a739e7 48db4732b39f0cde9f02d0a53c4a2534ae0d4017 8f3105a42b3cd7bd2afd2a2e2d99230e9dc416c0 6a514ea86e4cef219c046c8852f598d2caf44eda

This was referenced Apr 29, 2024
Merged
Merged
Merged
@rodrigo-lourenco-lopes rodrigo-lourenco-lopes added the backport stable/8.5 Backport a pull request to stable/8.5 label Apr 29, 2024
github-merge-queue bot pushed a commit that referenced this pull request Apr 29, 2024
@rodrigo-lourenco-lopes
[Backport stable/8.4] Soft pause information is persisted across all …
4ec9073 
…replicas. (#18040)

Backport of #16869 to stable/8.4.

relates to #16642

Original merge conflict:

CONFLICT (file location):
`zeebe/broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java
added in da57c7e (test: add PartitionProcessingStateTest) inside a
directory that was renamed in HEAD, suggesting it should perhaps be
moved to
broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java.`

The directory where PartitionProcessingStateTest was added had been
renamed. Just moved to the new place.
github-merge-queue bot pushed a commit that referenced this pull request Apr 29, 2024
@rodrigo-lourenco-lopes
[Backport stable/8.3] Soft pause information is persisted across all …
8091a21 
…replicas. (#18041)

Backport of #16869 to stable/8.3.

relates to #16642

`CONFLICT (modify/delete):
zeebe/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java
deleted in HEAD and modified in 9b09ee8 (test: add test to assert
that paused state is persisted after broker shutdown). Version
9b09ee8 (test: add test to assert that paused state is persisted
after broker shutdown) of
zeebe/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java
left in tree.`

The file had been deleted, so I just added it in full. 


`
CONFLICT (file location):
zeebe/broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java
added in da57c7e (test: add PartitionProcessingStateTest) inside a
directory that was renamed in HEAD, suggesting it should perhaps be
moved to
broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java.`

The directory where PartitionProcessingStateTest was added had been
renamed. Just moved to the new place.
github-merge-queue bot pushed a commit that referenced this pull request Apr 30, 2024
@rodrigo-lourenco-lopes
[Backport stable/8.2] Soft pause information is persisted across all …
ae9ead8 
…replicas. (#18044)

Backport of #16869 to stable/8.2.

relates to #16642

`CONFLICT (modify/delete):
zeebe/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java
deleted in HEAD and modified in 9b09ee8 (test: add test to assert
that paused state is persisted after broker shutdown). Version
9b09ee8 (test: add test to assert that paused state is persisted
after broker shutdown) of
zeebe/qa/integration-tests/src/test/java/io/camunda/zeebe/it/management/ExportingEndpointIT.java
left in tree.`

The file had been deleted, so I just added it in full. 



`
CONFLICT (file location):

zeebe/broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java
added in da57c7e (test: add PartitionProcessingStateTest) inside a
directory that was renamed in HEAD, suggesting it should perhaps be
moved to
broker/src/test/java/io/camunda/zeebe/broker/system/partitions/impl/PartitionProcessingStateTest.java.
`

The directory where PartitionProcessingStateTest was added had been
renamed. Just moved to the new place.

`
Auto-merging
broker/src/main/java/io/camunda/zeebe/broker/system/partitions/PartitionStartupAndTransitionContextImpl.java
CONFLICT (content): Merge conflict in
broker/src/main/java/io/camunda/zeebe/broker/system/partitions/PartitionStartupAndTransitionContextImpl.java
error: could not apply 2b023de... refactor: refactor get of exporter
state.
`

Merge conflic in PartitionStartupAndTransitionContextImpl.java for the
import of
`io.camunda.zeebe.broker.transport.adminapi.AdminApiRequestHandler`
@Zelldon Zelldon added the version:8.5.1 Marks an issue as being completely or in parts released in 8.5.1 label May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deepthidevaki deepthidevaki deepthidevaki approved these changes

Assignees

@rodrigo-lourenco-lopes rodrigo-lourenco-lopes

Labels
backport stable/8.2 Backport a pull request to 8.2.x backport stable/8.3 Backport a pull request to 8.3.x backport stable/8.4 Backport a pull request to 8.4.x backport stable/8.5 Backport a pull request to stable/8.5 component/zeebe Related to the Zeebe component/team version:8.5.1 Marks an issue as being completely or in parts released in 8.5.1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The Soft Pause information is Persisted in all Partions Replicas
4 participants
@rodrigo-lourenco-lopes @backport-action @deepthidevaki @Zelldon