[v23.1.x] Fix aborted txs being visible #11309
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we treat last_visible_index as LSO there is a chance of it being ahead of last applied offset so we can't conclude that the stm doesn't have ongoing txs fixes redpanda-data#10097 (cherry picked from commit 166119f)
Transactions in kafka protocol are stateful: the processing of the requests depends on the previous commands executed by the same or even different producer. It makes the situations when the replica- tion fails with the indecisive errors such as timeout dangerous because the true state is unknown. At the same time we have a fundamental invariant enforcing that a new leader start processing the request only when it has seen all the messages written by the previous leader. Switching rm_stm to use this invariant to handle the replication uncertainties by doing a forced step down or errors. (cherry picked from commit 996c4c7)
In general case LSO can't go beyond applied offset but it reduces acks=1 e2e scenarios because a record only becomes visible when its offset is behind LSO but LSO depends on applied offset which depends on a record being fully replicated so reading acks=1 is as slow as reading acks=all. A leader knows about the all inflight replication requests so it may return LSO beyond applied offset when it knows that it can't be affected by the inflight requests. (cherry picked from commit 3d1ef4e)
mmaslankaprv
approved these changes
Jun 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #10671
Fixes #10819
Backports Required
Release Notes