Restore the key generator on replay

[saig0]

Description

• reset the key generator after the replay to the highest key that was written on the stream
• update the key generator after skipping a command on replay
  • the key generator is used by not yet migrated processors
  • this part can be removed after all processors are migrated
• extract an interface from the ZeebeState to avoid that internal controls (key generator, last processed state) are exposed

Hints for the reviewer:

• I recommend to review the commits separately, the first includes the actual logic, the second is just the refactoring of the state
• the test StreamProcessorReplayTest#shouldRestoreKeyGeneratorAfterSkippingCommand() will be removed with the migration logic after all processors are migrated

Related issues

closes #6164

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/0.25) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement

[saig0]

Update: I added a third commit to ignore records from other partitions when restoring the key generator on replay. This happens when distributing a deployment to other partitions.

[korthout]

Nice and simple fix in the third commit. I generally like the changes, and everything seems correct. However, it's relative complex wording in an already complex context, so I've made some suggestions around understandability.

[saig0]

@korthout I applied your hints. Please have a second look :)

[korthout]

Thanks for the changes. LGTM 🧸

[saig0]

bors r+

[zeebe-bors]

Build succeeded:

• continuous-integration/jenkins/branch

[npepinpe]

It looks like this caused a performance drop of ~25 WI/s - do we have any ideas why?

[saig0]

No 🤔
The changes should affect only the reprocessing/replay. But nothing has changed for the processing phase.

[npepinpe]

Forgot to update, this PR didn't cause the changes - we found that the changes which caused a dip occurred in the following:

Quoting from Slack:

so i run several benchmarks, and what i see is slight dips over time it seems.

given that we have these merge commits in chronological order: 359aec0, 2ac8a25, f0f91fd, d73ae31

359aec0 does ~116 WI/s
2ac8a25 does ~116 WI/s
f0f91fd does ~105 WI/s
d73ae31 does ~95 WI/s

Between 2ac8a25 and f0f91fd we have:

* | | |   f0f91fd6d  Merge #6319 zeebe-bors[bot] Fri Feb 12 13:29:42 2021 +0000 
|\ \ \ \  
| |_|/ /  
|/| | |   
| * | | 27de076fb  chore(qa/it): fix flaky test Christopher Zell Thu Feb 11 13:28:43 2021 +0100 
* | | |   875dcf072  Merge #6289 zeebe-bors[bot] Fri Feb 12 12:04:57 2021 +0000 
|\ \ \ \  
| * | | | 4df43422d  chore(engine): add WorkflowRecord related tests Christopher Zell Thu Feb 11 08:54:46 2021 +0100 
| * | | | c622a9a94  chore(engine): write WorkflowRecord Christopher Zell Wed Feb 10 07:07:47 2021 +0100 
| * | | | 441b6b6f1  chore(protocol-impl): rename Workflow to WorkflowRecord Christopher Zell Wed Feb 10 06:38:52 2021 +0100 
| |/ / /  
* | | |   2ac8a2570  Merge #6328 zeebe-bors[bot] Fri Feb 12 10:09:02 2021 +0000

Between f0f91fd and d73ae31 we have:

* | | | |   d73ae31ea  Merge #6282 zeebe-bors[bot] Fri Feb 12 18:45:06 2021 +0000 
|\ \ \ \ \  
| |_|/ / /  
|/| | | |   
| * | | | 00ea30be7  chore(engine): apply review hints Nico Korthout Fri Feb 12 19:44:25 2021 +0100 
| * | | | f51a15f8d  refactor(engine): extract writers from processingcontext Nico Korthout Fri Feb 12 19:44:08 2021 +0100 
| * | | | 6a0f948a4  refactor(engine): remove appendNewEvent method Nico Korthout Fri Feb 12 19:42:53 2021 +0100 
| * | | | 96b1b060f  chore(engine): add StateWriter to processor init Nico Korthout Fri Feb 12 19:42:19 2021 +0100 
|/ / / /  
* | | |   f0f91fd6d  Merge #6319 zeebe-bors[bot] Fri Feb 12 13:29:42 2021 +0000

[npepinpe]

As of now I don't think it's dramatic enough to look into - we expect that in our current intermediate state things may be a little slower - but we should definitely keep an eye out when we run our weekly load tests, and probably take a bit of time at the end to profile/optimize.