🐛 fix: allow reconciliation of deadline-exceeded ClusterObjectSets#2643
Open
joelanford wants to merge 1 commit intooperator-framework:mainfrom
Open
🐛 fix: allow reconciliation of deadline-exceeded ClusterObjectSets#2643joelanford wants to merge 1 commit intooperator-framework:mainfrom
joelanford wants to merge 1 commit intooperator-framework:mainfrom
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
joelanford
force-pushed
the
fix/cos-deadline-exceeded-archival
branch
from
9dece97 to
a586063
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a reconciliation dead-end where ClusterObjectSet (COS) updates were being dropped once a revision hit ProgressDeadlineExceeded, preventing stuck revisions from being archived and cleaned up. It removes the update-blocking predicate, makes progress-deadline handling “sticky” in status updates, and adds a deadline-aware rate limiter plus an E2E scenario to validate archival cleanup.
Changes:
- Remove the
ProgressDeadlineExceeded-skipping watch predicate and introduce a controllerRateLimiterthat caps exponential backoff at the progress deadline. - Refactor progress-deadline computation into a shared
durationUntilDeadlinehelper and adjust progressing/retrying status updates to preferProgressDeadlineExceededonce exceeded. - Add an E2E scenario (and step helper) that archives a deadline-exceeded COS and verifies resource cleanup.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
internal/operator-controller/controllers/clusterobjectset_controller.go |
Removes the skip predicate, adds deadline-aware rate limiting, refactors deadline computation, and changes progressing/retrying condition behavior when the deadline is exceeded. |
test/e2e/steps/steps.go |
Adds a new Godog step to patch a COS lifecycle state to Archived. |
test/e2e/features/revision.feature |
Adds an E2E scenario that forces ProgressDeadlineExceeded, archives the COS, and asserts resources are removed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| drift := 2 * time.Second | ||
| requeueAfter := (remaining + drift).Round(time.Second) | ||
| l.Info(fmt.Sprintf("ProgressDeadline not exceeded, requeue after ~%v to check again.", requeueAfter)) | ||
| res = ctrl.Result{RequeueAfter: requeueAfter} |
Comment on lines
24
to
35
| "k8s.io/utils/clock" | ||
| "pkg.package-operator.run/boxcutter" | ||
| "pkg.package-operator.run/boxcutter/machinery" | ||
| machinerytypes "pkg.package-operator.run/boxcutter/machinery/types" | ||
| "pkg.package-operator.run/boxcutter/ownerhandling" | ||
| "pkg.package-operator.run/boxcutter/probing" | ||
| "k8s.io/client-go/util/workqueue" | ||
| ctrl "sigs.k8s.io/controller-runtime" | ||
| "sigs.k8s.io/controller-runtime/pkg/builder" | ||
| "sigs.k8s.io/controller-runtime/pkg/client" | ||
| "sigs.k8s.io/controller-runtime/pkg/controller" | ||
| "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" |
| backoff := r.delegate.When(item) | ||
|
|
||
| cos := &ocv1.ClusterObjectSet{} | ||
| if err := r.client.Get(context.Background(), item.NamespacedName, cos); err != nil { |
| When ClusterObjectSet "${COS_NAME}" lifecycle is set to "Archived" | ||
| Then ClusterObjectSet "${COS_NAME}" is archived | ||
| And resource "configmap/test-configmap" is eventually not found | ||
| And resource "deployment/test-deployment" is eventually not found |
Contributor
There was a problem hiding this comment.
It does not seems to be testing the same scenario @joelanford
Could we ensure the same scenario here?
Contributor
There was a problem hiding this comment.
- User installs a ClusterExtension. The CE controller creates COS-rev-1.
- COS-rev-1 gets stuck (e.g. a Deployment never becomes ready). After ProgressDeadlineMinutes, the reconciler sets Progressing=False/ProgressDeadlineExceeded.
- User updates the ClusterExtension. The CE controller creates COS-rev-2.
- COS-rev-2 rolls out successfully. It patches COS-rev-1 with lifecycleState: Archived so the old revision gets cleaned up.
- The watch predicate sees COS-rev-1 has ProgressDeadlineExceeded and drops the event.
- COS-rev-1 never reconciles, never processes the archival, and stays stuck forever.
Remove the skipProgressDeadlineExceededPredicate that blocked all update events for COS objects with ProgressDeadlineExceeded. This predicate prevented archival of stuck revisions because the lifecycle state patch was dropped as an update event. To prevent the reconcile loop that the predicate was masking, markAsProgressing now sets ProgressDeadlineExceeded instead of RollingOut/Retrying when the deadline has been exceeded. Terminal reasons (Succeeded) always apply. Unregistered reasons panic. Continue reconciling after ProgressDeadlineExceeded rather than clearing the error and stopping requeue. This allows revisions to recover if a transient error resolves itself, even after the deadline was exceeded. Extract durationUntilDeadline as a shared helper for deadline computation. Add a deadlineAwareRateLimiter that caps exponential backoff at the deadline so ProgressDeadlineExceeded is set promptly even during error retries. Add an e2e test that creates a COS with a never-ready deployment, waits for ProgressDeadlineExceeded, archives the COS, and verifies cleanup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
joelanford
force-pushed
the
fix/cos-deadline-exceeded-archival
branch
from
a586063 to
46dcf54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
skipProgressDeadlineExceededPredicateblocked all update events for COS objectswith
ProgressDeadlineExceeded, which prevented archival of stuck revisions — thelifecycle state patch was silently dropped.
This PR:
markAsProgressingto setProgressDeadlineExceededinstead ofRollingOut/Retryingwhen the deadline has been exceeded, preventing the reconcileloop the predicate was masking.
Succeededalways applies; unregistered reasons panicProgressDeadlineExceededrather than clearing theerror and stopping requeue. This allows revisions to recover if a transient error
resolves itself, even after the deadline was exceeded
durationUntilDeadlineas a shared helper for deadline computationdeadlineAwareRateLimiterthat caps exponential backoff at the deadline soProgressDeadlineExceededis set promptly even during error retriesProgressDeadlineExceeded, archives the COS, and verifies resource cleanupAddresses feedback from:
#2610 (comment)
Reviewer Checklist