🐛 Fix Boxcutter manifest ordering inconsistency #2329

tmshort · 2025-11-12T20:46:51Z

Issue:
Manifest ordering inconsistency: CRDs from Helm release manifest and
bundle manifest appeared in different orders, causing PhaseSort to
produce different phase structures even though they contained the same
objects.

Solution:
Added deterministic sorting in PhaseSort (phase.go):

Sort objects within each phase by Group, Version, Kind, Namespace, Name
Ensures consistent phase structure regardless of input order
Critical for comparing revisions from different sources

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Description

Reviewer Checklist

API Go Documentation
Tests: Unit Tests (and E2E Tests, if appropriate)
Comprehensive Commit Messages
Links to related GitHub Issue(s)

netlify · 2025-11-12T20:46:57Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`48e05e1`
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/691cc02223cecb00080a1c66
😎 Deploy Preview	https://deploy-preview-2329--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2025-11-12T22:27:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.52%. Comparing base (6ef62de) to head (48e05e1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2329      +/-   ##
==========================================
- Coverage   74.19%   70.52%   -3.68%     
==========================================
  Files          91       91              
  Lines        7228     7239      +11     
==========================================
- Hits         5363     5105     -258     
- Misses       1433     1699     +266     
- Partials      432      435       +3

Flag	Coverage Δ
e2e	`44.32% <0.00%> (-0.08%)`	⬇️
experimental-e2e	`14.08% <0.00%> (-34.31%)`	⬇️
unit	`58.54% <100.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joelanford · 2025-11-12T22:41:01Z

CollisionProtection mismatch: The migrated revision had collisionProtection=None (needed to adopt Helm-managed resources) while the bundle-generated revision had collisionProtection=Prevent (the default value).

I think this is intentional. We need None in order to adopt the existing objects. But then we want Prevent to ensure that we stop automatically adopting those objects in the future. I'm pretty sure that @thetechnick intended the adoption mechanics of None to be a one-time thing.

perdasilva · 2025-11-13T07:50:20Z

CollisionProtection mismatch: The migrated revision had collisionProtection=None (needed to adopt Helm-managed resources) while the bundle-generated revision had collisionProtection=Prevent (the default value).

I think this is intentional. We need None in order to adopt the existing objects. But then we want Prevent to ensure that we stop automatically adopting those objects in the future. I'm pretty sure that @thetechnick intended the adoption mechanics of None to be a one-time thing.

I think the original behavior makes sense. We shouldn't keep None beyond the initial migration to assume a conservative posture by default going forward. Leaving it as None opens the cluster to possible unforeseen pain in the future. I'd also say it's fine to generate a new revision with the new collisionProtection setting to keep an audit trail.

perdasilva · 2025-11-13T07:51:32Z

internal/operator-controller/applier/phase.go

+// to ensure consistent ordering regardless of input order. This is critical for
+// Helm-to-Boxcutter migration where the same resources may come from different sources
+// (Helm release manifest vs bundle manifest) and need to produce identical phases.
+func compareClusterExtensionRevision(a, b ocv1.ClusterExtensionRevisionObject) int {


Suggested change

func compareClusterExtensionRevision(a, b ocv1.ClusterExtensionRevisionObject) int {

func compareClusterExtensionRevisionObject(a, b ocv1.ClusterExtensionRevisionObject) int {

or even just compareRevisionObject ?

I have a rename for this, but I'm waiting on it, in case the decision is that we don't want the CollisionProtection change.

tmshort · 2025-11-13T14:26:24Z

CollisionProtection mismatch: The migrated revision had collisionProtection=None (needed to adopt Helm-managed resources) while the bundle-generated revision had collisionProtection=Prevent (the default value).

I think this is intentional. We need None in order to adopt the existing objects. But then we want Prevent to ensure that we stop automatically adopting those objects in the future. I'm pretty sure that @thetechnick intended the adoption mechanics of None to be a one-time thing.

I think the original behavior makes sense. We shouldn't keep None beyond the initial migration to assume a conservative posture by default going forward. Leaving it as None opens the cluster to possible unforeseen pain in the future. I'd also say it's fine to generate a new revision with the new collisionProtection setting to keep an audit trail.

So, removing either of the changes will cause the bug (two CERs created) to assert itself. If we are OK with this, then we don't necessarily need this fix, although sorting the objects during the phase is probably still a good idea. @dtfranz and I found the same solution to the problem:

On its own, the manifest sanitization does not completely fix the other issue, but it does make it quite a bit better. Following the repro steps in OCPBUGS-62943 on main, I get infinite additional CERs. When I add my fix for OPRUN-4238, I only get one additional CER. This extra CER is being created due to the diff between revisions created by GenerateRevisionFromHelmRelease and GenerateRevision; the former sets the CollisionProtection flag, and the latter does not. The objects within the phases may also need to be sorted. If you resolve those differences, the extra CER does not get generated. I'm not 100% sure of the fix method (or, as @perdasilva mentioned, if it's even an issue that needs fixing), but I'm pretty sure we need to disregard the CollisionProtection flag when comparing observed vs desired CERs. If we do that then the controller should understand that no new revision is required.

If we're good with having two initial CERs upon transition from the helm applier to the boxcutter applier, then I can reduce this to be just the sorting changes, and remove the CollisionProtection change. I would then close the bug as intentional.

tmshort · 2025-11-14T22:11:52Z

@joelanford @dtfranz giving me 👍 doesn't really tell me what you want! Do we want to fix the two CER issue or just fix the sorting?

tmshort · 2025-11-17T19:27:57Z

@joelanford @dtfranz now only the sorting remains.

dtfranz · 2025-11-18T07:29:53Z

/lgtm

camilamacedo86 · 2025-11-18T08:06:35Z

/lgtm

perdasilva · 2025-11-18T13:13:28Z

internal/operator-controller/applier/phase.go

+// to ensure consistent ordering regardless of input order. This is critical for
+// Helm-to-Boxcutter migration where the same resources may come from different sources
+// (Helm release manifest vs bundle manifest) and need to produce identical phases.
+func compareClusterExtensionRevisionObjects(a, b ocv1.ClusterExtensionRevisionObject) int {


would using cmp.Compare be a nice alternative here? e.g.

cmp.Compare(cmp.Or(aGVK.Group, bGVK.Group), cmp.Or(...), ...)

return cmp.Compare( cmp.Or(a.Object.GroupVersionKind().Group, b.Object.GroupVersionKind().Group), cmp.Or(a.Object.GroupVersionKind().Version, b.Object.GroupVersionKind().Version), cmp.Or(a.Object.GroupVersionKind().Kind, b.Object.GroupVersionKind().Kind), cmp.Or(a.Object.GetNamespace(), b.Object.GetNamespace()), cmp.Or(a.Object.GetName(), b.Object.GetName()), )

only a suggestion - if it doesn't spark joy, I'm happy to approve. Let me know.

I'm not familiar with that API, right now, it's not sparking joy (😉), but what's here works... so let's leave as-is.

np - it's the cool kid in town: https://pkg.go.dev/cmp basically obviates a lot the branching here

openshift-ci · 2025-11-18T17:12:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/operator-controller/OWNERS~~ [perdasilva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

perdasilva · 2025-11-18T17:24:46Z

internal/operator-controller/applier/phase.go

+	}
+
+	// Compare Version
+	if aGVK.Version < bGVK.Version {


Add another couple of resources to the test that exercise the version comparison (or just change one of the current resources to use the apps/v2 api group or something) - I think that will clear the codecov problem

If I do this, then I might as well look at cmp

And it's still less than threshold... 😢

Issue: Manifest ordering inconsistency: CRDs from Helm release manifest and bundle manifest appeared in different orders, causing PhaseSort to produce different phase structures even though they contained the same objects. Solution: Added deterministic sorting in PhaseSort (phase.go): - Sort objects within each phase by Group, Version, Kind, Namespace, Name - Ensures consistent phase structure regardless of input order - Critical for comparing revisions from different sources 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Todd Short <tshort@redhat.com>

perdasilva · 2025-11-18T18:56:41Z

/lgtm

tmshort · 2025-11-18T19:22:28Z

/override project

openshift-ci · 2025-11-18T19:22:52Z

@tmshort: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

project

Only the following failed contexts/checkruns were expected:

Verify PR title
codecov/project
crd-diff
e2e-kind
extension-developer-e2e
go-apidiff
go-verdiff
goreleaser
lint
netlify/olmv1/deploy-preview
tide
unit-test-basic
upgrade-e2e
verify

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

In response to this:

/override project

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tmshort · 2025-11-18T19:23:22Z

/override codecov/project

openshift-ci · 2025-11-18T19:23:26Z

@tmshort: Overrode contexts on behalf of tmshort: codecov/project

In response to this:

/override codecov/project

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tmshort requested a review from a team as a code owner November 12, 2025 20:46

openshift-ci bot requested review from dtfranz and trgeiger November 12, 2025 20:46

tmshort force-pushed the fix-extra-cer branch from 5897825 to 8b067ab Compare November 12, 2025 22:14

perdasilva reviewed Nov 13, 2025

View reviewed changes

tmshort force-pushed the fix-extra-cer branch from 8b067ab to 4618467 Compare November 13, 2025 19:32

tmshort force-pushed the fix-extra-cer branch 2 times, most recently from ec218e0 to 0a3158f Compare November 17, 2025 19:26

tmshort changed the title ~~🐛 Fix duplicate ClusterExtensionRevisions during Helm-to-Boxcutter migr…~~ 🐛 Fix Boxcutter manifest ordering inconsistency Nov 17, 2025

openshift-ci bot assigned dtfranz Nov 18, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2025

camilamacedo86 approved these changes Nov 18, 2025

View reviewed changes

openshift-ci bot assigned camilamacedo86 Nov 18, 2025

perdasilva reviewed Nov 18, 2025

View reviewed changes

perdasilva approved these changes Nov 18, 2025

View reviewed changes

openshift-ci bot assigned perdasilva Nov 18, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 18, 2025

perdasilva reviewed Nov 18, 2025

View reviewed changes

tmshort force-pushed the fix-extra-cer branch from 0a3158f to 48e05e1 Compare November 18, 2025 18:51

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2025

openshift-merge-bot bot merged commit f3569d5 into operator-framework:main Nov 18, 2025
25 of 26 checks passed

tmshort deleted the fix-extra-cer branch November 18, 2025 19:34

	func compareClusterExtensionRevision(a, b ocv1.ClusterExtensionRevisionObject) int {
	func compareClusterExtensionRevisionObject(a, b ocv1.ClusterExtensionRevisionObject) int {

🐛 Fix Boxcutter manifest ordering inconsistency #2329

🐛 Fix Boxcutter manifest ordering inconsistency #2329

Uh oh!

Conversation

tmshort commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer Checklist

Uh oh!

netlify bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for olmv1 ready!

Uh oh!

codecov bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joelanford commented Nov 12, 2025

Uh oh!

perdasilva commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshort commented Nov 13, 2025

Uh oh!

tmshort commented Nov 14, 2025

Uh oh!

tmshort commented Nov 17, 2025

Uh oh!

dtfranz commented Nov 18, 2025

Uh oh!

camilamacedo86 commented Nov 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Nov 18, 2025

Uh oh!

perdasilva Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perdasilva commented Nov 18, 2025

Uh oh!

tmshort commented Nov 18, 2025

Uh oh!

openshift-ci bot commented Nov 18, 2025

Uh oh!

tmshort commented Nov 18, 2025

Uh oh!

openshift-ci bot commented Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tmshort commented Nov 12, 2025 •

edited

Loading

netlify bot commented Nov 12, 2025 •

edited

Loading

codecov bot commented Nov 12, 2025 •

edited

Loading

perdasilva Nov 18, 2025 •

edited

Loading