Repository navigation
Replies: 1 comment
|
The retry case I’d want covered is a deletion on the remote branch. Say automation A bumps an image, but before it pushes, someone removes that workload. After the rejected push, fetching the new tip and rerunning the image update should leave the workload deleted. If there’s nothing left to update, that’s a successful no-op. I’d test that alongside the common case where A and B update different services and both changes should survive. For the month you’ve been running the patch, do you know how often the first retry succeeds and how often it hits the limit? That would help size the retry budget for a busy shared branch. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Problem
When many independent
ImageUpdateAutomationobjects push to the same branch of one GitOps repo (e.g. one IUA per service/region, all on independent reconcile timers), a multi-service release can cause many of them to try pushing around the same time. A concrete example observed in production: five regions all resolved a new image tag within 1-2 minutes of each other, but the resulting git commits landed 13-29 minutes apart, because several IUAs lost a non-fast-forward push race against others pushing to the same branch:There is no in-process retry today — the next attempt only happens on the next scheduled reconcile, or via controller-runtime's per-item exponential-backoff requeue (750ms doubling, capped at 15min). Under heavy contention, a single IUA can lose several races in a row, each costing a full backoff cycle, which is how a sub-minute git operation turns into a delay up to 1h. This reproduces on every release with enough concurrent writers to one branch, and gets worse as more services onboard per-instance IUAs.
Proposed solution
Add an in-process retry-with-backoff loop that reuses the existing local clone (fetch + hard-reset instead of a full re-clone) to resolve a lost push race in seconds instead of minutes, following the same shape as a standard
retry_with_backoff(5, 2, ...)wrapper around agit pull --rebase && git pushcycle. Gated behind a new, disabled-by-default feature gate (GitPushRetryOnFailure) so this ships as an explicit opt-in rather than a behavior change for existing installs.This requires a small, additive change to
fluxcd/pkg/git/gogitfirst, sincegogit.Clientonly exposesClone, not an incremental fetch.Linked PRs
(*gogit.Client).FetchAndReset, the primitive the retry loop needs to catch a working directory up to the new remote tip without a full re-clone.commitAndPushWithRetryand theGitPushRetryOnFailurefeature gate, built on top of the above.Linked Slack message for reference:
We've been running our custom build for over a month now and it has reduced image update delays significantly.
Would be nice if I could get some feedback on whether this can be added or not?
All reactions