Journal reset resulting in an intermediate state which is detected as corruption

[deepthidevaki]

Describe the bug

Related to #12374

A customer observed another case where the startup failed with the error:

"Expected to find a snapshot at index >= log's first index X, but found snapshot Y. A previous snapshot is most likely corrupted."

I could not verify it, but our assumption currently is that this is a false positive. The following might have happened:

1. Follower received snapshot
2. Before committing the snapshot, it reset segments
3. While deleting the segments, the node was shutdown.
4. After restart, it has the old snapshot plus partially deleted segments.

I think this is plausible as the reset/deleting segments is not and atomic operation. We might not be handling it correctly.

Expected behavior

The reset or snapshot commit process should not result in an invalid intermediate state which is detected as corruption.

Environment:

• Zeebe Version: 8.1.*

Related to support

[deepthidevaki]

A potential solution might be to delete the segments in the reverse order, combined with the PR that fixed #12374.

In a related issue, I discussed with @npepinpe a different approach. I was planning to create a separate issue for it, but I'm just dumping my thoughts here, as I think it is also relevant for this bug.

---

Terminologies

In-Sync replicas - The followers that are in sync with leaders and participating in the commit. In-sync replicas forms the quorum.

Out-of-sync follower - The follower which is receiving the event potentially after it is already committed.

In a setup with replicationFactor 3, there will be 2 in-sync replicas, out of which one will be the leader. The third one can be an out-of-sync follower.

Raft guarantees that any committed event is available in all in-sync replicas. The events are replicated to the out-of-sync follower eventually. But there is no guarantee when they are replicated. Ideally, they are replicated immediately. But it is possible the follower is slow, or the network is slow, resulting the out-of-sync follower to lag behind the leader by a large amount.

At some point, the leader sends a snapshot instead of the events, because the follower is lagging behind. At this point, the current behavior is as follows:

Follower resets the log (delete all segments)
Receives and commits the new snapshot
Delete the old snapshot
Continue receiving the events after the snapshot index

The above process can lead to the following scenarios:

Step 1 and 2 are not atomic. If the replica crashes in between, then it restarts with an empty state.
If it restarts before step 4, then it starts with a snapshot that is not use-able. Snapshot is valid only if the event corresponding to lastFollowUpEventPosition is in the log.

In a normal scenario, the above situation is not problematic because the in-sync replicas are healthy and it is guaranteed that only one of the in-sync replica becomes the leader.

However, in a disaster scenario where all in-sync replicas are gone, it might be acceptable to continue functioning with whatever data is available in the out-of-sync replica. It might not have up-to-date state, but for some use-cases it is ok to lose last X amount of data. In such cases, we want to enable users to recover the cluster from the state of the out-of-sync replica. However, in the above scenario the out-of-sync replica might be in a state where it's state is empty or the snapshot is not useable.

So, it would be good if we could ensure that the state in the out-of-sync follower is always in a valid state. To acheive that we can

Do not delete old snapshot and reset log immediately when it receives a snapshot.
Instead keep the old snapshot and the logs until the new snapshot is committed and its follow up event is received.

Solution 1:

We keep the old snapshot + logs in a different folder, the new snapshot and new logs in a different folder. On restarts, it attempts to use the new snapshot + log and fall back to the old state if necessary.

Solution 2:

Do not reset the logs, when a new snapshot is received. Instead, add a marker record in the journal to indicate that there is a snapshot at the position. The readers must know how to handle these “gaps”. For example, if stream processor reader hits this marker record, it has to throw away its state and replace with the snapshot. If the raft leader reads this record, it should replicate the snapshot instead of the event etc.

The compaction logic should take this into account, and compact only if there is one valid snapshot with its followup event in the log. So the follower can rollback to an old valid state if necessary.

This would also help in some issue that we observed recently where restarting the node during resetting the log can incorrectly lead to detecting it as corrupted log.

[npepinpe]

With #12868, we implemented a quick fix to prevent the specific case we're aware of. By deleting the segments in reverse order during reset, we can ensure that there are no gaps in the log/snapshot, and thus no corruption. Data loss is acceptable here as it was the desired outcome of the reset operation.

However, we agreed not to close the issue - this quick fix only solves a specific thing, and not the root cause, which is that the install operation on the follower is not atomic, specifically persisting the snapshot/clearing the log/updating the meta store/getting the first record for the snapshot/etc. are all independent operations and the node may stop at any point in between, leaving us in a weird state.

[megglos]

ZDP-Planning:

• while we are at it, it would be worth to assess potential solutions to get an understanding of how much effort it is to change
• needs prototyping
• back to backlog as upcoming for now to be picked up going forward
• relates to upcoming resilience epic