fix(kds): retry sync updates on write conflicts (backport of #17738) - #17841
Merged
Conversation
On a Zone, KDS sync competes with zone-local writers for the same resources: the VIP allocator owns `Status.VIPs` and the hostname generators own `Status.Addresses`. Stores version the whole resource rather than its halves, so any of those writes invalidates the version KDS read at `List` time and the downstream `Update` fails with a conflict. On Kubernetes the store reads through an informer cache, which can hand back an already stale version even without a concurrent writer. The conflict propagates out of `Sync` and tears the stream down. Because delta xDS marks a resource delivered when the response is sent, the upstream keeps considering the zone current while it silently drifts until the next full resync. - Updates that lose a write conflict are collected instead of aborting the transaction, and reapplied by `retryConflictedUpdates` after it commits, so the backoff holds no connection or row locks. - Each retry re-reads the stored copy and rebases the pending upstream change onto it, taking `Status` from the fresh copy when `IgnoreStatusChange` is set, matching what `Sync` already does when it first decides to update. - Up to 3 attempts with a jittered pause of at most 100ms in between, shared by the whole batch: the first retry runs immediately, which is all a store reading its own writes needs, and the later ones wait so a cached store (Kubernetes informer) can observe the write that won. - Exhausted retries return the conflict to the caller, keeping the existing tear-down-and-resync behaviour rather than silently skipping the resource, which would leave the zone drifting. - Unit tests cover both the recovering case and the give-up case. N/A --------- Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
slonka
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automatic cherry-pick of #17738 for branch release-2.14
Generated by action
cherry-picked commit d6edc9c