Prefer replicating events instead of snapshots #7957

lenaschoenburg · 2021-10-08T10:31:21Z

Description

This adds a threshold on number of events a follower can lag behind a leader to decide wether we should replicate events or a snapshot. This should save resources because replicating a low number of events should be more efficient than replicating a snapshot.
We can't always replicate events even if the lag is below the threshold because some events might not be available anymore due to log compaction.

Related issues

closes #7784

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

The changes are backwards compatibility with previous versions
If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/0.25) to the PR, in case that fails you need to create backports manually.

Testing:

There are unit/integration tests that verify all acceptance criterias of the issue
New tests are written to ensure backwards compatibility with further versions
The behavior is tested manually
The change has been verified by a QA run
The impact of the changes is verified by a benchmark

Documentation:

The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
New content is added to the release announcement

atomix/cluster/src/test/java/io/atomix/raft/RaftReplicationTest.java

lenaschoenburg · 2021-10-08T15:34:20Z

There are a couple of tests that I'll need to adjust because they are assuming that we'd replicate snapshots in most cases.

lenaschoenburg · 2021-10-11T14:05:13Z

@deepthidevaki Seems like there are still some unresolved test failures that I have to work on, sorry for the early request for review.

deepthidevaki · 2021-10-11T14:12:35Z

@oleschoenburg Let me know if you need help with the failing tests.

lenaschoenburg · 2021-10-11T15:27:43Z

Tests seem to run through now (except a temporary issue for LGTM, not sure how to restart that). At least on my machine the relevant tests seem unusually flaky. One issue seems to be that during tests, filling a segment takes too long. I'd like to add a large payload to every message that is used to fill the segment to hopefully speed up this process.

deepthidevaki

🚀
Please see my comment on the new "condition" and the test to reproduce the error. Othwerise it looks good, just have some optional comments.

atomix/cluster/src/main/java/io/atomix/raft/roles/LeaderAppender.java

atomix/cluster/src/test/java/io/atomix/raft/RaftReplicationTest.java

qa/integration-tests/src/test/java/io/camunda/zeebe/it/clustering/ClusteredSnapshotTest.java

qa/integration-tests/src/test/java/io/camunda/zeebe/it/clustering/RestoreTest.java

atomix/cluster/src/test/java/io/atomix/raft/RaftReplicationTest.java

atomix/cluster/src/main/java/io/atomix/raft/impl/RaftContext.java

atomix/cluster/src/test/java/io/atomix/raft/RaftReplicationTest.java

This enables us to perform snapshots on members without triggering log compaction. To achieve this we remove the RaftSnapshotListener and (optionally) run compaction synchronously.

… be preferred Adds a config value that indicates how much a follower must lag behind the leader before we prefer replicating a snapshot instead of events. The default value of 100 was chosen arbitrarily and might need readjustment.

…and snapshots

…r snapshot

deepthidevaki

Great! 🎉

Please see my comment before merging.

qa/integration-tests/src/test/java/io/camunda/zeebe/it/clustering/ClusteredSnapshotTest.java

Various tests assumed that we always replicate snapshots. Here we either remove tests that are no longer necessary or adjust tests such that snapshots are replicated, for example by triggering log compaction

lenaschoenburg · 2021-10-14T14:25:13Z

I'll run some benchmarks before merging to confirm that the current threshold of 100 is a sensible value.

lenaschoenburg · 2021-10-15T09:02:21Z

Looking at the benchmarks it appears that this PR has no performance impact. With the current benchmark setup, some followers are lagging behind by >10k events which means that the current threshold of 100 is too low to prevent replicating snapshots.

We can still find a better threshold or a different method to choose between replicating snapshots or events later.

lenaschoenburg · 2021-10-15T09:04:45Z

bors merge

deepthidevaki · 2021-10-15T09:30:09Z

Thanks @oleschoenburg for running the benchmark.
If the follower is lagging behind be ~10K events, then it is better to send the snapshot instead of the events. As you said it make sense to evaluate what would be a good threshold for it. 100 might be low, but 10K is too big.

ghost · 2021-10-15T09:32:02Z

Build succeeded:

continuous-integration/jenkins/branch

Zelldon · 2021-10-22T12:43:50Z

@npepinpe are we planning to backport this?

npepinpe · 2021-10-22T12:45:19Z

Would this fix or help fix any bugs?

Zelldon · 2021-10-22T12:53:11Z

My current assumption is that it fixes #7955 but I'm still investigating. I started a new benchmark yesterday and will keep it running for a while.

npepinpe · 2021-10-22T12:54:29Z

Then we can do that 👍 @oleschoenburg @deepthidevaki - how do you see these changes? How is the risk/value ratio of backporting this? It doesn't seem like too much at a quick glance so I'd be fine with backporting, but you probably have a better idea.

If you think it's fine and worth it, then can one of you please just walk Ole through how to backport things? Thanks!

deepthidevaki · 2021-10-22T13:08:33Z

I don't see any risk backporting this. If there is a chance this fixes the bug, we can backport it.
In @oleschoenburg 's benchmark, there were still many "InstallRequests" because the follower was lagging behind by 1000s of events.

lenaschoenburg · 2021-10-22T13:08:47Z

I'm not sure about the risk, I'll defer to @deepthidevaki for that. As for "is it worth": I'd be positively surprised if this solves #7955 because in our (admittedly limited) benchmarks we weren't able to see any impact.

Zelldon · 2021-10-22T13:09:46Z

Lets wait until next week, then I will check my benchmark again. :)

Zelldon · 2021-10-25T10:18:46Z

We can scratch that from our list. It is still failing with one partition and this fix. http://34.77.165.228/d/I4lo7_EZk/zeebe?orgId=1&from=1634801065067&to=1634854108575&var-DS_PROMETHEUS=Prometheus&var-namespace=zell-chaos-cw42&var-pod=All&var-partition=All

We can see again lot of install requests 🤷

I think the breaking of performance is also highly depended on the state size

lenaschoenburg commented Oct 8, 2021

View reviewed changes

atomix/cluster/src/test/java/io/atomix/raft/RaftReplicationTest.java Show resolved Hide resolved

lenaschoenburg requested a review from deepthidevaki October 11, 2021 12:58

lenaschoenburg mentioned this pull request Oct 11, 2021

Configurable threshold for deciding between replicating events or snapshots #7968

Closed

deepthidevaki requested changes Oct 12, 2021

View reviewed changes

Zelldon mentioned this pull request Oct 13, 2021

Unexpected drop of performance with 1.2 #7955

Closed

lenaschoenburg requested a review from deepthidevaki October 13, 2021 14:42

lenaschoenburg added 4 commits October 14, 2021 14:56

feat(atomix): enable taking snapshots in tests without log compaction

64d9252

This enables us to perform snapshots on members without triggering log compaction. To achieve this we remove the RaftSnapshotListener and (optionally) run compaction synchronously.

test(atomix): test heuristic for deciding between replication events …

5dc82df

…and snapshots

feat(atomix): Use threshold to decide if we should replicate events o…

6ffc0cb

…r snapshot

lenaschoenburg force-pushed the 7784-event-replication-heuristic branch from 4dc1c4d to b05dcb0 Compare October 14, 2021 13:05

deepthidevaki approved these changes Oct 14, 2021

View reviewed changes

qa/integration-tests/src/test/java/io/camunda/zeebe/it/clustering/ClusteredSnapshotTest.java Outdated Show resolved Hide resolved

test(qa): Adjust tests that expect snapshots replication

6af180e

Various tests assumed that we always replicate snapshots. Here we either remove tests that are no longer necessary or adjust tests such that snapshots are replicated, for example by triggering log compaction

lenaschoenburg force-pushed the 7784-event-replication-heuristic branch from b05dcb0 to 6af180e Compare October 14, 2021 14:18

ghost merged commit 8a4adb8 into develop Oct 15, 2021

ghost deleted the 7784-event-replication-heuristic branch October 15, 2021 09:32

menski added the Release: 1.3.0-alpha1 label Nov 8, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefer replicating events instead of snapshots #7957

Prefer replicating events instead of snapshots #7957

lenaschoenburg commented Oct 8, 2021

lenaschoenburg commented Oct 8, 2021

lenaschoenburg commented Oct 11, 2021

deepthidevaki commented Oct 11, 2021

lenaschoenburg commented Oct 11, 2021

deepthidevaki left a comment

deepthidevaki left a comment

lenaschoenburg commented Oct 14, 2021

lenaschoenburg commented Oct 15, 2021

lenaschoenburg commented Oct 15, 2021

deepthidevaki commented Oct 15, 2021

ghost commented Oct 15, 2021

Zelldon commented Oct 22, 2021

npepinpe commented Oct 22, 2021

Zelldon commented Oct 22, 2021

npepinpe commented Oct 22, 2021 •

edited

Loading

deepthidevaki commented Oct 22, 2021

lenaschoenburg commented Oct 22, 2021

Zelldon commented Oct 22, 2021

Zelldon commented Oct 25, 2021 •

edited

Loading

Prefer replicating events instead of snapshots #7957

Prefer replicating events instead of snapshots #7957

Conversation

lenaschoenburg commented Oct 8, 2021

Description

Related issues

Definition of Done

lenaschoenburg commented Oct 8, 2021

lenaschoenburg commented Oct 11, 2021

deepthidevaki commented Oct 11, 2021

lenaschoenburg commented Oct 11, 2021

deepthidevaki left a comment

Choose a reason for hiding this comment

deepthidevaki left a comment

Choose a reason for hiding this comment

lenaschoenburg commented Oct 14, 2021

lenaschoenburg commented Oct 15, 2021

lenaschoenburg commented Oct 15, 2021

deepthidevaki commented Oct 15, 2021

ghost commented Oct 15, 2021

Zelldon commented Oct 22, 2021

npepinpe commented Oct 22, 2021

Zelldon commented Oct 22, 2021

npepinpe commented Oct 22, 2021 • edited Loading

deepthidevaki commented Oct 22, 2021

lenaschoenburg commented Oct 22, 2021

Zelldon commented Oct 22, 2021

Zelldon commented Oct 25, 2021 • edited Loading

npepinpe commented Oct 22, 2021 •

edited

Loading

Zelldon commented Oct 25, 2021 •

edited

Loading