POC: Enhance CF prefix check

[Zelldon]

Description

Idea:

We know that the prefixes normally look like this: [0, 0, 0, 0, 0, 0, 0, 21] length = 8
the keys we check them against look mostly like this: [0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 12] length = 12

Right now we check the prefixes via iterating from the beginning, which means we have to iterate over several zero values which could be improved if we start from the back. Right now this will only improve the case when the prefix is different. This was what I first tried.

Later I thought in order to improve this for all cases, I could check the column family byte directly (currently we have less than 128 column families), this means we can just check the last byte of the long (we write them in ByteOrder.BIG_ENDIAN). If this is equal the entry is a valid entry for the CF otherwise not.

Furthermore, sometimes we don't provide any prefix this means we can check whether the length of the prefix is zero so we don't check the key further. If the prefix key is larger then zero we can check the key (starting after the CF prefix).

The JMH benchmark results were interesting since they show a much lower error rate (variance) than the other test runs.

Result "io.camunda.zeebe.engine.perf.EnginePerformanceTest.measureProcessExecutionTime":
  223.584 ±(99.9%) 2.052 ops/s [Average]
  (min, avg, max) = (196.934, 223.584, 238.563), stdev = 8.686
  CI (99.9%): [221.533, 225.636] (assumes normal distribution)

Benchmark                                           Mode  Cnt    Score   Error  Units
EnginePerformanceTest.measureProcessExecutionTime  thrpt  200  223.584 ± 2.052  ops/s

I want to start a benchmark by maxing out performance to see whether we see any difference.

Related issues

related #12241

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

• Operate
• Tasklist
• Web Modeler
• Desktop Modeler
• Optimize

Please refer to our review guidelines.

[github-actions]

Setup

Deployed to measurement-4872866750

camunda-platform:
  zeebe:
    image:
      repository: gcr.io/zeebe-io/zeebe
      tag: zell-improve-prefix-check-benchmark-bccd00e
  zeebe-gateway:
    image:
      repository: gcr.io/zeebe-io/zeebe
      tag: zell-improve-prefix-check-benchmark-bccd00e
global:
  image:
    tag: zell-improve-prefix-check-benchmark-bccd00e

Measurement before

Process Instance Execution Time: p99=0.915 p90=0.337 p50=0.098
Throughput: 149.971 PI/s
Grafana

Chaos injection

Deployed chaos network-latency-5

Measurement after

Process Instance Execution Time: p99=4.183 p90=2.250 p50=0.941
Throughput: 74.911 PI/s
Grafana

Details

See https://github.com/camunda/zeebe/actions/runs/4872866750

[Zelldon]

Closing for now, branch will stay to apply if we need it.