Add a standalone app that can be used to restore a broker from a backup #10412

deepthidevaki · 2022-09-20T11:48:06Z

Description

Added a new module for restore.
Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given BrokerCfg. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and RaftPartitionGroupFactory, there is no other way. There is scope of splitting broker to take these shared configs out of it.
Added a RestoreApp in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components BrokerConfiguration and WorkerDirectory to restore module.
The backup id to restore from is passed as a command line argument --backupId=x. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb.

A basic test to verify a broker can be restored is added. More scenarios should be added later.

PS:- App is not added to the distribution yet.

Related issues

related #10263

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

The changes are backwards compatibility with previous versions
If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

There are unit/integration tests that verify all acceptance criterias of the issue
New tests are written to ensure backwards compatibility with further versions
The behavior is tested manually
The change has been verified by a QA run
The impact of the changes is verified by a benchmark

Documentation:

The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
New content is added to the release announcement
If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

qa/integration-tests/src/test/java/io/camunda/zeebe/it/backup/BackupIT.java

github-actions · 2022-09-20T12:04:50Z

Test Results

  925 files ±  0   925 suites ±0 2h 10m 47s ⏱️ + 10m 38s
7 321 tests - 54 7 311 ✔️ - 54 10 💤 ±0 0 ❌ ±0
7 509 runs - 54 7 499 ✔️ - 54 10 💤 ±0 0 ❌ ±0

Results for commit f80cb8d. ± Comparison against base commit c7baaf4.

♻️ This comment has been updated with latest results.

npepinpe

👍

No blockers, but maybe the integration tests need to be updated.

npepinpe · 2022-09-21T14:29:04Z

dist/src/main/java/io/camunda/zeebe/restore/BackupStoreComponent.java

+      throw new IllegalArgumentException("No backup store configured, cannot restore from backup.");
+    }
+
+    if (store == BackupStoreType.S3) {


💭 It sounds to me like the BackupStore will become another one of these things which should most likely be configured outside of the broker, and simply injected there. That said, we can leave it like this for now.

npepinpe · 2022-09-21T14:29:51Z

dist/src/main/java/io/camunda/zeebe/restore/RestoreApp.java

+  private final BackupStore backupStore;
+
+  @Value("${backupId}")
+  // Parsed from commandline Eg:-`--backupId=100`


❓ I guess it can come from anywhere, e.g. env vars, properties, etc., right? Or just command line?

I think not. It is a command line argument, not a configuration property.

dist/src/main/java/io/camunda/zeebe/restore/RestoreApp.java

qa/integration-tests/src/test/java/io/camunda/zeebe/it/backup/BackupTest.java

restore/src/main/java/io/camunda/zeebe/restore/RestoreManager.java

deepthidevaki · 2022-09-22T10:57:48Z

bors merge

10391: deps(maven): bump log4j-bom from 2.18.0 to 2.19.0 r=npepinpe a=dependabot[bot] Bumps log4j-bom from 2.18.0 to 2.19.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.logging.log4j:log4j-bom&package-manager=maven&previous-version=2.18.0&new-version=2.19.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> 10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10443: Do not take a backup if it already exists r=deepthidevaki a=deepthidevaki ## Description After restore, the log is truncated to the checkpoint position. So the checkpoint record is processed again and will trigger a new backup with the same Id of the backup it restored from. With this PR, `BackupService` handles this case gracefully. In addition, we also do not take a new backup if existing backup is failed or in progress. Alternatively, we can delete this backup and take a new one. But chances of it happening (i.e triggering a new backup when one already is in progress/failed) is very low. So we can keep this simple. ## Related issues closes #10430 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-22T12:26:39Z

This PR was included in a batch that timed out, it will be automatically retried

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10443: Do not take a backup if it already exists r=deepthidevaki a=deepthidevaki ## Description After restore, the log is truncated to the checkpoint position. So the checkpoint record is processed again and will trigger a new backup with the same Id of the backup it restored from. With this PR, `BackupService` handles this case gracefully. In addition, we also do not take a new backup if existing backup is failed or in progress. Alternatively, we can delete this backup and take a new one. But chances of it happening (i.e triggering a new backup when one already is in progress/failed) is very low. So we can keep this simple. ## Related issues closes #10430 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-22T14:09:30Z

Build failed (retrying...):

Test summary

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-22T15:29:11Z

Build failed:

Test summary

lenaschoenburg · 2022-09-22T15:29:18Z

bors r+

10412: Add a standalone app that can be used to restore a broker from a backup r=oleschoenburg a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10418: Add monitor backup API endpoint r=oleschoenburg a=npepinpe ## Description This PR extends the backup management endpoint to add a monitoring API under `GET /{id}` (e.g. if the actuator is at `http://localhost:9600/actuator/backups`, then `GET http://localhost:9600/actuator/backups/{id}`). At the moment we use the internal types provided by the broker client as the response types, but in the future this will be replaced by the OpenAPI generated types. As such, I didn't think it was necessary to create user facing types and write some mapping code, since we will scrape it and replace it. However, I understand that we don't know when that will done, so I'm open to doing it anyway for safety. The acceptance tests were updated (and renamed to better reflect what they do), and as they overlap with the old `BackupIT` tests, those were deleted. ## Related issues closes #9902 10443: Do not take a backup if it already exists r=oleschoenburg a=deepthidevaki ## Description After restore, the log is truncated to the checkpoint position. So the checkpoint record is processed again and will trigger a new backup with the same Id of the backup it restored from. With this PR, `BackupService` handles this case gracefully. In addition, we also do not take a new backup if existing backup is failed or in progress. Alternatively, we can delete this backup and take a new one. But chances of it happening (i.e triggering a new backup when one already is in progress/failed) is very low. So we can keep this simple. ## Related issues closes #10430 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>

zeebe-bors-camunda · 2022-09-22T16:39:06Z

Build failed (retrying...):

Test summary

10412: Add a standalone app that can be used to restore a broker from a backup r=oleschoenburg a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-22T18:23:55Z

Build failed:

Test summary

deepthidevaki · 2022-09-23T06:18:08Z

bors merge

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10461: deps(maven): bump aws-java-sdk-core from 1.12.308 to 1.12.309 r=github-actions[bot] a=dependabot[bot] Bumps [aws-java-sdk-core](https://github.com/aws/aws-sdk-java) from 1.12.308 to 1.12.309. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">aws-java-sdk-core's changelog</a>.</em></p> <blockquote> <h1><strong>1.12.309</strong> <strong>2022-09-22</strong></h1> <h2><strong>AWS Backup Gateway</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Changes include: new GetVirtualMachineApi to fetch a single user's VM, improving ListVirtualMachines to fetch filtered VMs as well as all VMs, and improving GetGatewayApi to now also return the gateway's MaintenanceStartTime.</li> </ul> </li> </ul> <h2><strong>AWS Device Farm</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds the support for VPC-ENI based connectivity for private devices on AWS Device Farm.</li> </ul> </li> </ul> <h2><strong>AWS Glue</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Added support for S3 Event Notifications for Catalog Target Crawlers.</li> </ul> </li> </ul> <h2><strong>AWS SSO Identity Store</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Documentation updates for the Identity Store CLI Reference.</li> </ul> </li> </ul> <h2><strong>Amazon Elastic Compute Cloud</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Documentation updates for Amazon EC2.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java/commit/dab3ce334bba719453a4587cea633a6462c4121f"><code>dab3ce3</code></a> AWS SDK for Java 1.12.309</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/765446355993783014b67b1d1f4456ec6c4f5215"><code>7654463</code></a> Update GitHub version number to 1.12.309-SNAPSHOT</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-java/compare/1.12.308...1.12.309">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.amazonaws:aws-java-sdk-core&package-manager=maven&previous-version=1.12.308&new-version=1.12.309)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> 10462: deps(maven): bump software.amazon.awssdk:bom from 2.17.278 to 2.17.280 r=github-actions[bot] a=dependabot[bot] Bumps [software.amazon.awssdk:bom](https://github.com/aws/aws-sdk-java-v2) from 2.17.278 to 2.17.280. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md">software.amazon.awssdk:bom's changelog</a>.</em></p> <blockquote> <h1><strong>2.17.280</strong> <strong>2022-09-22</strong></h1> <h2><strong>AWS Backup Gateway</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Changes include: new GetVirtualMachineApi to fetch a single user's VM, improving ListVirtualMachines to fetch filtered VMs as well as all VMs, and improving GetGatewayApi to now also return the gateway's MaintenanceStartTime.</li> </ul> </li> </ul> <h2><strong>AWS Device Farm</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds the support for VPC-ENI based connectivity for private devices on AWS Device Farm.</li> </ul> </li> </ul> <h2><strong>AWS Glue</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Added support for S3 Event Notifications for Catalog Target Crawlers.</li> </ul> </li> </ul> <h2><strong>AWS SDK for Java v2</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Updated service endpoint metadata.</li> </ul> </li> </ul> <h2><strong>AWS SSO Identity Store</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Documentation updates for the Identity Store CLI Reference.</li> </ul> </li> </ul> <h2><strong>Amazon Elastic Compute Cloud</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Documentation updates for Amazon EC2.</li> </ul> </li> </ul> <h1><strong>2.17.279</strong> <strong>2022-09-21</strong></h1> <h2><strong>AWS S3 Control</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>S3 on Outposts launches support for object versioning for Outposts buckets. With S3 Versioning, you can preserve, retrieve, and restore every version of every object stored in your buckets. You can recover from both unintended user actions and application failures.</li> </ul> </li> </ul> <h2><strong>AWS SDK for Java v2</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Updated service endpoint metadata.</li> </ul> </li> <li> <h3>Bugfixes</h3> <ul> <li>Fixed issue where errors were being wrapped by SdkClientException</li> </ul> </li> </ul> <h2><strong>Amazon Comprehend</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Amazon Comprehend now supports synchronous mode for targeted sentiment API operations.</li> </ul> </li> </ul> <h2><strong>Amazon SageMaker Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>SageMaker now allows customization on Canvas Application settings, including enabling/disabling time-series forecasting and specifying an Amazon Forecast execution role at both the Domain and UserProfile levels.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/ea3d8838bd54c96b559f25c4d3238b63dfd92c84"><code>ea3d883</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-sdk-java-v2/issues/2151">#2151</a> from aws/staging/40fbf60b-c3e9-458a-b106-4c80e497adf1</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/697221433153dbf228deeed020e2ecf0fbc3163a"><code>6972214</code></a> Release 2.17.280. Updated CHANGELOG.md, README.md and all pom.xml.</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/503ce29845146fbe71254fc273d22f8d1149a30c"><code>503ce29</code></a> Updated endpoints.json.</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/e1850bb772b5109fb188f97815fc34b7c2badf84"><code>e1850bb</code></a> AWS Device Farm Update: This release adds the support for VPC-ENI based conne...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/e6b79ce62d7ba9ada2f29c0d85de7a587d4c1e67"><code>e6b79ce</code></a> AWS Backup Gateway Update: Changes include: new GetVirtualMachineApi to fetch...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/da00d51297e437aa509b7d3a73b74e4e0f5c1f81"><code>da00d51</code></a> AWS SSO Identity Store Update: Documentation updates for the Identity Store C...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/fe6efc050628fe91e0ec7a2bfcc34fb206fa714d"><code>fe6efc0</code></a> AWS Glue Update: Added support for S3 Event Notifications for Catalog Target ...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/c8dfc89b8bb345bfa513834b182fb55659a37da3"><code>c8dfc89</code></a> Amazon Elastic Compute Cloud Update: Documentation updates for Amazon EC2.</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/98e21b1d8a8a6cde598c8202979b3ba6c08775d1"><code>98e21b1</code></a> Update to next snapshot version: 2.17.280-SNAPSHOT</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/f3e99c64eb01477737b9b6b470ed2f1cd52e0b85"><code>f3e99c6</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-sdk-java-v2/issues/2035">#2035</a> from aws/salande/object-versioning</li> <li>Additional commits viewable in <a href="https://github.com/aws/aws-sdk-java-v2/compare/2.17.278...2.17.280">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=software.amazon.awssdk:bom&package-manager=maven&previous-version=2.17.278&new-version=2.17.280)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

zeebe-bors-camunda · 2022-09-23T07:38:10Z

Build failed (retrying...):

Test summary

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10431: Add backpressure technical documentation r=saig0 a=remcowesterhoud ## Description  Explains how backpressure is implemented in Zeebe ## Related issues  N/A Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: Remco Westerhoud <remco@westerhoud.nl>

zeebe-bors-camunda · 2022-09-23T12:42:53Z

Build failed (retrying...):

Test summary

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-23T14:02:08Z

Build failed:

Test summary

deepthidevaki · 2022-09-23T15:52:52Z

bors merge

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

zeebe-bors-camunda · 2022-09-23T16:12:54Z

Build failed:

Test summary

deepthidevaki · 2022-09-26T07:17:30Z

bors merge

deepthidevaki · 2022-09-26T07:40:03Z

bors merge

zeebe-bors-camunda · 2022-09-26T07:40:05Z

Already running a review

10412: Add a standalone app that can be used to restore a broker from a backup r=deepthidevaki a=deepthidevaki ## Description * Added a new module for restore. * Adds a RestoreManager that coordinates restoring of all partitions. It first determines which partitions to restore from based on the given `BrokerCfg`. This configuration must be the same one used by the Broker when it is started after the restore. Because of this, restore module depens on broker module. But since it needs access to the configuration and `RaftPartitionGroupFactory`, there is no other way. There is scope of splitting broker to take these shared configs out of it. * Added a `RestoreApp` in dist. My plan was to add it in restore module. But it needs access to BrokerConfig, which is built in dist. So the app is added to dist module. Alternative is to copy components `BrokerConfiguration` and `WorkerDirectory` to restore module. * The backup id to restore from is passed as a command line argument `--backupId=x`. This is parsed by spring boot. I decided to keep it simple for now, since we are planning to release it as experimental feature anyway. If we decide to support this app long term, then I suggest to switch to something like picocli that we use in zdb. A basic test to verify a broker can be restored is added. More scenarios should be added later. PS:- App is not added to the distribution yet. ## Related issues related #10263 10440: Extract KeyGenerator from ZeebeState r=oleschoenburg a=Zelldon ## Description As preparation for the StreamProcessor - Engine module split we have to move out the Keygenerator out from the zeebeState, since this part of the stream processing (key is part of RecordMetadata). We have to reset the keys during replay and for that we need access to the key generator. The Key generator is now created in the StreamProcessor and passed into the ZeebeState (as dependency), this allowed to make minimal changes. At the start I removed completely the KEygenerator from the ZeebeState, but then we would need to adjust a LOT of setup code, which doesn't felt necessary tbh since it just about where this generator is created. The change also allowed us to remove the ZeebeState from the StreamProcessorContext 💪 `@npepinpe` `@saig0` who ever has time to review this. I think `@npepinpe` is currently a bit busy with other stuff related to backup and reviews, maybe you have time to review it `@saig0`  ## Related issues  related to #10130 related to #9727 10483: deps(maven): bump checkstyle from 10.3.3 to 10.3.4 r=github-actions[bot] a=dependabot[bot] Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 10.3.3 to 10.3.4. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/checkstyle/checkstyle/releases">checkstyle's releases</a>.</em></p> <blockquote> <h2>checkstyle-10.3.4</h2> <p><a href="https://checkstyle.org/releasenotes.html#Release_10.3.4">https://checkstyle.org/releasenotes.html#Release_10.3.4</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/checkstyle/checkstyle/commit/6de3b9f1feb5c80450af2ad646f82ba34dc250bb"><code>6de3b9f</code></a> [maven-release-plugin] prepare release checkstyle-10.3.4</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/a4497de578d5a72b386e7ca88676da190584eece"><code>a4497de</code></a> doc: release notes 10.3.4</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/32e8d374e67e5ef5c19a688064bb2a77b5947651"><code>32e8d37</code></a> Issue <a href="https://github-redirect.dependabot.com/checkstyle/checkstyle/issues/12145">#12145</a>: corrected tokens so all are required</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/96e3e05a526bca3f7a157ca929666530eb0b089f"><code>96e3e05</code></a> dependency: bump pitest-accelerator-junit5 from 1.0.1 to 1.0.2</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/37842d2f291b138b2e9d44b33350add3f9562f34"><code>37842d2</code></a> Issue <a href="https://github-redirect.dependabot.com/checkstyle/checkstyle/issues/3955">#3955</a>: corrected tokens so all are required</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/38ee347a49376e80858c969f742e6461dfe7d731"><code>38ee347</code></a> minor: remove unnecessary checkstyle versions to diff.groovy</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/318770dc1343e9be90b8304c275d9adaf4ad8d45"><code>318770d</code></a> dependency: bump slf4j-simple from 2.0.1 to 2.0.2</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/119453600ff9f6accc247f748df8c0dd6c65a2af"><code>1194536</code></a> dependency: bump junit.version from 5.9.0 to 5.9.1</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/5a56c16882b68e0aaedf3a2a29737f412bbd66be"><code>5a56c16</code></a> Issue <a href="https://github-redirect.dependabot.com/checkstyle/checkstyle/issues/12132">#12132</a>: Fix ArrayIndexOutOfBoundsException in pitest-survival-check-xml...</li> <li><a href="https://github.com/checkstyle/checkstyle/commit/701bd65dec286ab75fed35b99a580fc77c8ea03f"><code>701bd65</code></a> Issue <a href="https://github-redirect.dependabot.com/checkstyle/checkstyle/issues/12210">#12210</a>: Add method to ignore unstable checker framework violations</li> <li>Additional commits viewable in <a href="https://github.com/checkstyle/checkstyle/compare/checkstyle-10.3.3...checkstyle-10.3.4">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.puppycrawl.tools:checkstyle&package-manager=maven&previous-version=10.3.3&new-version=10.3.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: Christopher Zell <zelldon91@googlemail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…tion group Previously it was calculated when starting the partition. But all necessary info to determin the partition members are already available when building the group. For restore app, we have to determine the partition members, but there is no need to start the partitions. So it is moved from `join` to the constructor.

Restore app needs access to it to determine partition members.

RestoreApp has to read the same broker configuration. We cannot move this to io.camunda.zeebe.shared because it is used by gateway as well.

…n backupId

zeebe-bors-camunda · 2022-09-26T08:08:40Z

Canceled.

deepthidevaki · 2022-09-26T08:09:45Z

bors merge

zeebe-bors-camunda · 2022-09-26T09:33:23Z

Build succeeded:

10418: Add monitor backup API endpoint r=deepthidevaki a=npepinpe ## Description This PR extends the backup management endpoint to add a monitoring API under `GET /{id}` (e.g. if the actuator is at `http://localhost:9600/actuator/backups`, then `GET http://localhost:9600/actuator/backups/{id}`). At the moment we use the internal types provided by the broker client as the response types, but in the future this will be replaced by the OpenAPI generated types. As such, I didn't think it was necessary to create user facing types and write some mapping code, since we will scrape it and replace it. However, I understand that we don't know when that will done, so I'm open to doing it anyway for safety. The acceptance tests were updated (and renamed to better reflect what they do), and as they overlap with the old `BackupIT` tests, those were deleted. ## Related issues closes #9902 10442: feat: support terminate end events r=saig0 a=saig0 ## Description Add support for BPMN terminate end events. See #8789 (comment) on how the BPMN element should work. The implementation doesn't follow the BPMN spec in one point: the flow scope that contains the terminate end event is not terminated but completed. Reasoning: - The state of the flow scope is a detail that doesn't influence the core behavior. In both cases, the process instance should continue, for example, by taking the outgoing sequence flow. The difference is not visible to process participants but only when monitoring the process instance, for example, in Operate. - It fits better with the existing implementation. It would be a bigger effort to continue the process instance (e.g. taking the outgoing sequence flow) when the flow scope is terminated. As a result, we would end up in more complex code. - It aligns with the behavior of Camunda Platform 7. Side note: I implemented the major parts during a [Live Hacking session](https://www.twitch.tv/videos/1584245006). 🎥 ## Related issues closes #8789 10464: Add restore app to the distribution r=deepthidevaki a=deepthidevaki ## Description * Create a binary `bin/restore` when building the project. * Configure docker image to run restore, when a specific env variable is set. This is done similarly to chose between broker and gateway. * In addition, extended the restore app to delete data directory if restore failed. Depends on #10412 ## Related issues closes #10263 Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com> Co-authored-by: Philipp Ossler <philipp.ossler@gmail.com> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>

deepthidevaki mentioned this pull request Sep 20, 2022

Add support for restoring a partition from a backup #10368

Merged

10 tasks

github-advanced-security bot found potential problems Sep 20, 2022

View reviewed changes

qa/integration-tests/src/test/java/io/camunda/zeebe/it/backup/BackupIT.java Fixed Show fixed Hide fixed

deepthidevaki mentioned this pull request Sep 21, 2022

After restore, a completed backup is marked as failed #10430

Closed

deepthidevaki force-pushed the dd-10263-coordinate-restore branch 3 times, most recently from e5eff50 to a40bcbb Compare September 21, 2022 13:52

deepthidevaki changed the title ~~[Draft] Add restore app~~ Add a standalone app that can be used to restore a broker from a backup Sep 21, 2022

deepthidevaki marked this pull request as ready for review September 21, 2022 14:07

deepthidevaki requested a review from npepinpe September 21, 2022 14:23

npepinpe approved these changes Sep 22, 2022

View reviewed changes

deepthidevaki force-pushed the dd-10263-coordinate-restore branch from a40bcbb to 2308203 Compare September 22, 2022 10:26

deepthidevaki mentioned this pull request Sep 23, 2022

Add restore app to the distribution #10464

Merged

10 tasks

deepthidevaki added 10 commits September 26, 2022 10:08

feat(pom): add new module for backup restore

96e832a

refactor(restore): move all restore classes to a new module

89f2024

refactor(broker): make RaftPartitionGroupFactory public

950a2a2

Restore app needs access to it to determine partition members.

feat(restore): coordinate restore of all partitions

61b6229

fix(restore): fix check if directory is empty

0ce8eb4

refactor(dist): move configuration needed by restore app to shared

5bb6bc2

RestoreApp has to read the same broker configuration. We cannot move this to io.camunda.zeebe.shared because it is used by gateway as well.

feat(dist): add restore app that can restore the broker from the give…

649c510

…n backupId

refactor(restore): cleanup composing futures

c663d91

refactor(restore): remove broker banner

f80cb8d

deepthidevaki force-pushed the dd-10263-coordinate-restore branch from 2308203 to f80cb8d Compare September 26, 2022 08:08

zeebe-bors-camunda bot merged commit db48127 into main Sep 26, 2022

zeebe-bors-camunda bot deleted the dd-10263-coordinate-restore branch September 26, 2022 09:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a standalone app that can be used to restore a broker from a backup #10412

Add a standalone app that can be used to restore a broker from a backup #10412

deepthidevaki commented Sep 20, 2022 •

edited

github-actions bot commented Sep 20, 2022 •

edited

npepinpe left a comment

npepinpe Sep 21, 2022

npepinpe Sep 21, 2022

deepthidevaki Sep 22, 2022

deepthidevaki commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

lenaschoenburg commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

deepthidevaki commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

deepthidevaki commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

deepthidevaki commented Sep 26, 2022

deepthidevaki commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

deepthidevaki commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

Add a standalone app that can be used to restore a broker from a backup #10412

Add a standalone app that can be used to restore a broker from a backup #10412

Conversation

deepthidevaki commented Sep 20, 2022 • edited

Description

Related issues

Definition of Done

github-actions bot commented Sep 20, 2022 • edited

Test Results

npepinpe left a comment

Choose a reason for hiding this comment

npepinpe Sep 21, 2022

Choose a reason for hiding this comment

npepinpe Sep 21, 2022

Choose a reason for hiding this comment

deepthidevaki Sep 22, 2022

Choose a reason for hiding this comment

deepthidevaki commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

lenaschoenburg commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

zeebe-bors-camunda bot commented Sep 22, 2022

deepthidevaki commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

deepthidevaki commented Sep 23, 2022

zeebe-bors-camunda bot commented Sep 23, 2022

deepthidevaki commented Sep 26, 2022

deepthidevaki commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

deepthidevaki commented Sep 26, 2022

zeebe-bors-camunda bot commented Sep 26, 2022

deepthidevaki commented Sep 20, 2022 •

edited

github-actions bot commented Sep 20, 2022 •

edited