Skip to content

Conversation

@camilamacedo86
Copy link
Contributor

@camilamacedo86 camilamacedo86 commented Aug 28, 2025

To validate the cleanup of the solution Cursor was used to generate a comprehensive script.

Before the changes
Screenshot 2025-08-28 at 14 58 49

After we cleanup the webhook catalog

Screenshot 2025-08-28 at 15 40 21
✅ Tests completed successfully
Step 3: Capturing after-test state...
Capturing comprehensive cluster state to after-test.out...
State captured successfully to after-test.out
Step 4: Analyzing cleanup effectiveness...
Analyzing cleanup effectiveness...
Analysis complete. Results saved to /Users/camilam/go/src/github/operator-framework-operator-controller/openshift/tests-extension/output.result
=== FINAL RESULTS ===
=== CLEANUP VALIDATION ANALYSIS: Thu Aug 28 15:35:07 BST 2025 ===

=== DIFF SUMMARY ===
⚠️  DIFFERENCES DETECTED: Analyzing changes...

=== DETAILED DIFF ===
--- before-test.out     2025-08-28 15:31:07
+++ after-test.out      2025-08-28 15:35:07
@@ -1,4 +1,4 @@
-=== CLUSTER STATE CAPTURE: Thu Aug 28 15:30:47 BST 2025 ===
+=== CLUSTER STATE CAPTURE: Thu Aug 28 15:34:49 BST 2025 ===
 
 === NAMESPACES ===
 Namespace/default/2025-08-28T12:56:11Z
@@ -82,32 +82,40 @@
 Pod/default/verify-all-openshiftcertifiedoperators-57hvs-78xv4/Succeeded/2025-08-28T14:08:16Z
 Pod/default/verify-all-openshiftcertifiedoperators-6q5b9-bq5zs/Succeeded/2025-08-28T14:16:55Z
 Pod/default/verify-all-openshiftcertifiedoperators-gz5z5-6cxtw/Succeeded/2025-08-28T13:52:35Z
+Pod/default/verify-all-openshiftcertifiedoperators-ttkww-9qds2/Succeeded/2025-08-28T14:31:51Z
 Pod/default/verify-all-openshiftcommunityoperators-6pnvr-xwjvl/Succeeded/2025-08-28T14:16:10Z
+Pod/default/verify-all-openshiftcommunityoperators-mjg74-ftdk5/Succeeded/2025-08-28T14:31:48Z
 Pod/default/verify-all-openshiftcommunityoperators-ntnbs-c7m4t/Succeeded/2025-08-28T14:24:40Z
 Pod/default/verify-all-openshiftcommunityoperators-qqbnj-lv7pw/Succeeded/2025-08-28T13:51:39Z
 Pod/default/verify-all-openshiftcommunityoperators-xx78t-8w69n/Succeeded/2025-08-28T14:08:14Z
 Pod/default/verify-all-openshiftredhatmarketplace-hrsh4-hlkmp/Succeeded/2025-08-28T14:16:53Z
+Pod/default/verify-all-openshiftredhatmarketplace-nllmf-h2cng/Succeeded/2025-08-28T14:31:49Z
 Pod/default/verify-all-openshiftredhatmarketplace-p27s7-vlzmc/Succeeded/2025-08-28T14:08:15Z
 Pod/default/verify-all-openshiftredhatmarketplace-spsk2-g747g/Succeeded/2025-08-28T13:52:37Z
 Pod/default/verify-all-openshiftredhatmarketplace-tj4np-fvvrn/Succeeded/2025-08-28T14:24:36Z
 Pod/default/verify-all-openshiftredhatoperators-62vcj-rrc4v/Succeeded/2025-08-28T14:16:52Z
 Pod/default/verify-all-openshiftredhatoperators-d2gqw-mk98n/Succeeded/2025-08-28T13:52:36Z
+Pod/default/verify-all-openshiftredhatoperators-fqgrh-ptkts/Succeeded/2025-08-28T14:31:46Z
 Pod/default/verify-all-openshiftredhatoperators-tjw2c-vnjs2/Succeeded/2025-08-28T14:24:34Z
 Pod/default/verify-all-openshiftredhatoperators-xgdnk-cjrmn/Succeeded/2025-08-28T14:08:16Z
 Pod/default/verify-metas-openshiftcertifiedoperators-9rg76-xx4b2/Succeeded/2025-08-28T14:16:53Z
 Pod/default/verify-metas-openshiftcertifiedoperators-bt662-h2947/Succeeded/2025-08-28T14:08:21Z
 Pod/default/verify-metas-openshiftcertifiedoperators-htvsj-l7tpn/Succeeded/2025-08-28T13:52:38Z
 Pod/default/verify-metas-openshiftcertifiedoperators-hxl58-lk46h/Succeeded/2025-08-28T14:24:40Z
+Pod/default/verify-metas-openshiftcertifiedoperators-jwn9j-zwtlk/Succeeded/2025-08-28T14:31:50Z
 Pod/default/verify-metas-openshiftcommunityoperators-96wkb-4pvfd/Succeeded/2025-08-28T14:24:37Z
 Pod/default/verify-metas-openshiftcommunityoperators-dmgmh-46rpz/Succeeded/2025-08-28T14:16:54Z
 Pod/default/verify-metas-openshiftcommunityoperators-lnsxp-ft5ct/Succeeded/2025-08-28T13:51:39Z
 Pod/default/verify-metas-openshiftcommunityoperators-w7hp7-jtmf9/Succeeded/2025-08-28T14:08:22Z
+Pod/default/verify-metas-openshiftcommunityoperators-wdzkn-pxd44/Succeeded/2025-08-28T14:31:50Z
+Pod/default/verify-metas-openshiftredhatmarketplace-j52b7-9hdkv/Succeeded/2025-08-28T14:31:48Z
 Pod/default/verify-metas-openshiftredhatmarketplace-j9g9g-rlrp5/Succeeded/2025-08-28T14:16:55Z
 Pod/default/verify-metas-openshiftredhatmarketplace-mqnmc-wlmjm/Succeeded/2025-08-28T13:52:36Z
 Pod/default/verify-metas-openshiftredhatmarketplace-rzc4l-rwxq7/Succeeded/2025-08-28T14:08:22Z
 Pod/default/verify-metas-openshiftredhatmarketplace-zk7l8-gczz4/Succeeded/2025-08-28T14:24:38Z
 Pod/default/verify-metas-openshiftredhatoperators-b6rls-88vdl/Succeeded/2025-08-28T14:08:14Z
 Pod/default/verify-metas-openshiftredhatoperators-hwhhr-h5c7s/Succeeded/2025-08-28T14:16:51Z
+Pod/default/verify-metas-openshiftredhatoperators-stj5x-24r7h/Succeeded/2025-08-28T14:31:47Z
 Pod/default/verify-metas-openshiftredhatoperators-v2bd2-57mpp/Succeeded/2025-08-28T14:24:39Z
 Pod/default/verify-metas-openshiftredhatoperators-wm67d-5lwmc/Succeeded/2025-08-28T13:52:38Z
 Pod/openshift-apiserver-operator/openshift-apiserver-operator-5756666865-4drr9/Running/2025-08-28T12:59:18Z
@@ -307,7 +315,6 @@
 Pod/openshift-machine-config-operator/machine-config-server-mhrjt/Running/2025-08-28T13:03:21Z
 Pod/openshift-machine-config-operator/machine-config-server-txnk7/Running/2025-08-28T13:03:21Z
 Pod/openshift-marketplace/certified-operators-84z52/Running/2025-08-28T13:13:49Z
-Pod/openshift-marketplace/certified-operators-pj52j/Running/2025-08-28T14:30:43Z
 Pod/openshift-marketplace/community-operators-52rm4/Running/2025-08-28T13:13:49Z
 Pod/openshift-marketplace/marketplace-operator-597b8598f9-fr25s/Running/2025-08-28T12:59:19Z
 Pod/openshift-marketplace/redhat-marketplace-q7cjv/Running/2025-08-28T13:13:52Z
@@ -3344,4 +3351,4 @@
 --- Webhook Test Resources ---
 No webhook test resources found
 
-=== END CAPTURE: Thu Aug 28 15:31:07 BST 2025 ===
+=== END CAPTURE: Thu Aug 28 15:35:07 BST 2025 ===

=== LEFTOVER RESOURCE ANALYSIS ===
--- Checking for test-related leftovers ---
🔍 ClusterCatalogs:
✅ No test ClusterCatalogs found
🔍 ClusterExtensions:
✅ No test ClusterExtensions found
🔍 Test Namespaces:
✅ No test namespaces found
🔍 Test ClusterRoles:
✅ No test ClusterRoles found
🔍 Test ClusterRoleBindings:
✅ No test ClusterRoleBindings found
🔍 Webhook Configurations:
✅ No test webhook configurations found
🔍 Jobs in default namespace:
✅ No test jobs found in default namespace
🔍 Test CRDs:
✅ No test-related CRDs found
🔍 OpenShift Build/Image resources:
✅ No test build/image resources found

=== CLEANUP EFFECTIVENESS SUMMARY ===
🎉 EXCELLENT: Complete cleanup detected - no test resources left behind
Cleanup Score: 100%
Total leftover categories: 0

=== ANALYSIS COMPLETE: Thu Aug 28 15:35:11 BST 2025 ===
Comprehensive cleanup validation complete!

See attached the scripts used:

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 28, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: camilamacedo86
Once this PR has been reviewed and has the lgtm label, please assign grokspawn for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@camilamacedo86 camilamacedo86 changed the title UPSTREAM: <carry>: Enhance clenup cluster catalog created for webhook… NO-ISSUE: UPSTREAM: <carry>: Enhance clenup cluster catalog created for webhook… Aug 28, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 28, 2025
@openshift-ci-robot
Copy link

@camilamacedo86: This pull request explicitly references no jira issue.

In response to this:

To validate the cleanup of the solution Cursor was used to generate a comprehensive script.

Before the changes
Screenshot 2025-08-28 at 14 58 49

After we cleanup the webhook catalog

Screenshot 2025-08-28 at 15 40 21
✅ Tests completed successfully
Step 3: Capturing after-test state...
Capturing comprehensive cluster state to after-test.out...
State captured successfully to after-test.out
Step 4: Analyzing cleanup effectiveness...
Analyzing cleanup effectiveness...
Analysis complete. Results saved to /Users/camilam/go/src/github/operator-framework-operator-controller/openshift/tests-extension/output.result
=== FINAL RESULTS ===
=== CLEANUP VALIDATION ANALYSIS: Thu Aug 28 15:35:07 BST 2025 ===

=== DIFF SUMMARY ===
⚠️  DIFFERENCES DETECTED: Analyzing changes...

=== DETAILED DIFF ===
--- before-test.out     2025-08-28 15:31:07
+++ after-test.out      2025-08-28 15:35:07
@@ -1,4 +1,4 @@
-=== CLUSTER STATE CAPTURE: Thu Aug 28 15:30:47 BST 2025 ===
+=== CLUSTER STATE CAPTURE: Thu Aug 28 15:34:49 BST 2025 ===

=== NAMESPACES ===
Namespace/default/2025-08-28T12:56:11Z
@@ -82,32 +82,40 @@
Pod/default/verify-all-openshiftcertifiedoperators-57hvs-78xv4/Succeeded/2025-08-28T14:08:16Z
Pod/default/verify-all-openshiftcertifiedoperators-6q5b9-bq5zs/Succeeded/2025-08-28T14:16:55Z
Pod/default/verify-all-openshiftcertifiedoperators-gz5z5-6cxtw/Succeeded/2025-08-28T13:52:35Z
+Pod/default/verify-all-openshiftcertifiedoperators-ttkww-9qds2/Succeeded/2025-08-28T14:31:51Z
Pod/default/verify-all-openshiftcommunityoperators-6pnvr-xwjvl/Succeeded/2025-08-28T14:16:10Z
+Pod/default/verify-all-openshiftcommunityoperators-mjg74-ftdk5/Succeeded/2025-08-28T14:31:48Z
Pod/default/verify-all-openshiftcommunityoperators-ntnbs-c7m4t/Succeeded/2025-08-28T14:24:40Z
Pod/default/verify-all-openshiftcommunityoperators-qqbnj-lv7pw/Succeeded/2025-08-28T13:51:39Z
Pod/default/verify-all-openshiftcommunityoperators-xx78t-8w69n/Succeeded/2025-08-28T14:08:14Z
Pod/default/verify-all-openshiftredhatmarketplace-hrsh4-hlkmp/Succeeded/2025-08-28T14:16:53Z
+Pod/default/verify-all-openshiftredhatmarketplace-nllmf-h2cng/Succeeded/2025-08-28T14:31:49Z
Pod/default/verify-all-openshiftredhatmarketplace-p27s7-vlzmc/Succeeded/2025-08-28T14:08:15Z
Pod/default/verify-all-openshiftredhatmarketplace-spsk2-g747g/Succeeded/2025-08-28T13:52:37Z
Pod/default/verify-all-openshiftredhatmarketplace-tj4np-fvvrn/Succeeded/2025-08-28T14:24:36Z
Pod/default/verify-all-openshiftredhatoperators-62vcj-rrc4v/Succeeded/2025-08-28T14:16:52Z
Pod/default/verify-all-openshiftredhatoperators-d2gqw-mk98n/Succeeded/2025-08-28T13:52:36Z
+Pod/default/verify-all-openshiftredhatoperators-fqgrh-ptkts/Succeeded/2025-08-28T14:31:46Z
Pod/default/verify-all-openshiftredhatoperators-tjw2c-vnjs2/Succeeded/2025-08-28T14:24:34Z
Pod/default/verify-all-openshiftredhatoperators-xgdnk-cjrmn/Succeeded/2025-08-28T14:08:16Z
Pod/default/verify-metas-openshiftcertifiedoperators-9rg76-xx4b2/Succeeded/2025-08-28T14:16:53Z
Pod/default/verify-metas-openshiftcertifiedoperators-bt662-h2947/Succeeded/2025-08-28T14:08:21Z
Pod/default/verify-metas-openshiftcertifiedoperators-htvsj-l7tpn/Succeeded/2025-08-28T13:52:38Z
Pod/default/verify-metas-openshiftcertifiedoperators-hxl58-lk46h/Succeeded/2025-08-28T14:24:40Z
+Pod/default/verify-metas-openshiftcertifiedoperators-jwn9j-zwtlk/Succeeded/2025-08-28T14:31:50Z
Pod/default/verify-metas-openshiftcommunityoperators-96wkb-4pvfd/Succeeded/2025-08-28T14:24:37Z
Pod/default/verify-metas-openshiftcommunityoperators-dmgmh-46rpz/Succeeded/2025-08-28T14:16:54Z
Pod/default/verify-metas-openshiftcommunityoperators-lnsxp-ft5ct/Succeeded/2025-08-28T13:51:39Z
Pod/default/verify-metas-openshiftcommunityoperators-w7hp7-jtmf9/Succeeded/2025-08-28T14:08:22Z
+Pod/default/verify-metas-openshiftcommunityoperators-wdzkn-pxd44/Succeeded/2025-08-28T14:31:50Z
+Pod/default/verify-metas-openshiftredhatmarketplace-j52b7-9hdkv/Succeeded/2025-08-28T14:31:48Z
Pod/default/verify-metas-openshiftredhatmarketplace-j9g9g-rlrp5/Succeeded/2025-08-28T14:16:55Z
Pod/default/verify-metas-openshiftredhatmarketplace-mqnmc-wlmjm/Succeeded/2025-08-28T13:52:36Z
Pod/default/verify-metas-openshiftredhatmarketplace-rzc4l-rwxq7/Succeeded/2025-08-28T14:08:22Z
Pod/default/verify-metas-openshiftredhatmarketplace-zk7l8-gczz4/Succeeded/2025-08-28T14:24:38Z
Pod/default/verify-metas-openshiftredhatoperators-b6rls-88vdl/Succeeded/2025-08-28T14:08:14Z
Pod/default/verify-metas-openshiftredhatoperators-hwhhr-h5c7s/Succeeded/2025-08-28T14:16:51Z
+Pod/default/verify-metas-openshiftredhatoperators-stj5x-24r7h/Succeeded/2025-08-28T14:31:47Z
Pod/default/verify-metas-openshiftredhatoperators-v2bd2-57mpp/Succeeded/2025-08-28T14:24:39Z
Pod/default/verify-metas-openshiftredhatoperators-wm67d-5lwmc/Succeeded/2025-08-28T13:52:38Z
Pod/openshift-apiserver-operator/openshift-apiserver-operator-5756666865-4drr9/Running/2025-08-28T12:59:18Z
@@ -307,7 +315,6 @@
Pod/openshift-machine-config-operator/machine-config-server-mhrjt/Running/2025-08-28T13:03:21Z
Pod/openshift-machine-config-operator/machine-config-server-txnk7/Running/2025-08-28T13:03:21Z
Pod/openshift-marketplace/certified-operators-84z52/Running/2025-08-28T13:13:49Z
-Pod/openshift-marketplace/certified-operators-pj52j/Running/2025-08-28T14:30:43Z
Pod/openshift-marketplace/community-operators-52rm4/Running/2025-08-28T13:13:49Z
Pod/openshift-marketplace/marketplace-operator-597b8598f9-fr25s/Running/2025-08-28T12:59:19Z
Pod/openshift-marketplace/redhat-marketplace-q7cjv/Running/2025-08-28T13:13:52Z
@@ -3344,4 +3351,4 @@
--- Webhook Test Resources ---
No webhook test resources found

-=== END CAPTURE: Thu Aug 28 15:31:07 BST 2025 ===
+=== END CAPTURE: Thu Aug 28 15:35:07 BST 2025 ===

=== LEFTOVER RESOURCE ANALYSIS ===
--- Checking for test-related leftovers ---
🔍 ClusterCatalogs:
✅ No test ClusterCatalogs found
🔍 ClusterExtensions:
✅ No test ClusterExtensions found
🔍 Test Namespaces:
✅ No test namespaces found
🔍 Test ClusterRoles:
✅ No test ClusterRoles found
🔍 Test ClusterRoleBindings:
✅ No test ClusterRoleBindings found
🔍 Webhook Configurations:
✅ No test webhook configurations found
🔍 Jobs in default namespace:
✅ No test jobs found in default namespace
🔍 Test CRDs:
✅ No test-related CRDs found
🔍 OpenShift Build/Image resources:
✅ No test build/image resources found

=== CLEANUP EFFECTIVENESS SUMMARY ===
🎉 EXCELLENT: Complete cleanup detected - no test resources left behind
Cleanup Score: 100%
Total leftover categories: 0

=== ANALYSIS COMPLETE: Thu Aug 28 15:35:11 BST 2025 ===
Comprehensive cleanup validation complete!

See attached the scripts used:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Add AfterAll hook to cleanup leftover ClusterCatalog resource
using EnsureCleanupClusterCatalog helper function.

Assisted-by: Cursor
@camilamacedo86 camilamacedo86 force-pushed the enhance-cleanup-catalog branch from 337ed7c to a478980 Compare August 28, 2025 14:56
})

AfterAll(func(ctx SpecContext) {
helpers.EnsureCleanupClusterCatalog(ctx, webhookCatalogName)
Copy link
Contributor Author

@camilamacedo86 camilamacedo86 Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tmshort

I have one concern:
The only way to clean it is with AfterALL
However, can we really use AfterAll with OTE?

If I remember correctly, I read in a comment that it might not work well, which is why it was not added in the first place.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are you trying to clean up here? The catalogs are not all cleaned up?

Copy link
Contributor Author

@camilamacedo86 camilamacedo86 Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the webhook tests, we create a Catalog.
This Catalog is shared across all tests. That is why we can only clean AfterAll

By design, it was not cleaned up before.
(I am not sure if we can use AfterAll, but if we can, that might be the only way.)

Please see EnsureCleanupClusterCatalog.
This helper makes sure the Catalog is cleaned up:

func EnsureCleanupClusterCatalog(ctx context.Context, name string) {
k8s := env.Get().K8sClient
cc := &olmv1.ClusterCatalog{}
key := client.ObjectKey{Name: name}
if err := k8s.Get(ctx, key, cc); err != nil {
if !errors.IsNotFound(err) {
fmt.Fprintf(GinkgoWriter, "Warning: failed to get ClusterCatalog %q during cleanup: %v\n", name, err)
}
return
}
By(fmt.Sprintf("deleting lingering ClusterCatalog %q", name))
if err := k8s.Delete(ctx, cc); err != nil && !errors.IsNotFound(err) {
fmt.Fprintf(GinkgoWriter, "Warning: failed to delete ClusterCatalog %q: %v\n", name, err)
}
Eventually(func() bool {
err := k8s.Get(ctx, key, &olmv1.ClusterCatalog{})
return errors.IsNotFound(err)
}).WithTimeout(3*time.Minute).WithPolling(2*time.Second).
Should(BeTrue(), "ClusterCatalog %q failed to delete", name)
}

Why was this not added earlier?
There is some history here:
👉 openshift/origin#30059 (comment)

Because of that, I am not sure if it is completely safe to add now.
I think we need confirm.

So, for now, I would suggest:

  • Put this PR on /hold
  • It doesn’t seem safe to take the risk before TDB
  • Our checks already confirm that everything is cleaned properly

@tmshort
Copy link
Contributor

tmshort commented Aug 28, 2025

/hold
I'm not sure this is needed.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 28, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 28, 2025

@camilamacedo86: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-e2e-aws-ovn a478980 link false /test okd-scos-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@camilamacedo86
Copy link
Contributor Author

Closing this one
If we need tomorrow we reopen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants