[release-4.17] OCPBUGS-59253: Reduce Frequency of Update Requests for Copied CSVs (#3597) #1036

tmshort · 2025-07-11T14:48:47Z

Cherry-pick from release-4.18

(bugfix): reduce frequency of update requests for CSVs

by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields.

This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs.

update unit tests
updates to test so far
Small changes
Add metadata drift guard to copyToNamespace

Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation.

This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now:

  • Updates the spec and hash annotations
  • Updates the status subresource
  • Records the new generation and resourceVersion in the guard annotations

Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn.

Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact.

Tests for metadata guard

Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match.

Persist observed annotations on all status updates
GCI the file
Use TransformFunc

Unit tests not updated

Update operatorgroup tests to compile
Restore operatorgroup_test from master

Remove metadatalister

Remove more PartialObjectMetadata
Remove hashes from operator_test
Fix error messages for static-analysis
Update test annotations and test client
Rename pruning to listerwatcher
Set resync to 6h
Add CSV copy revert syncer
Log tweaks
Consolidate revert and gc syncers
Add logging and reduce the amount of metadata in the TransformFunc
Handle the copy CSV revert via a requeue of the primary CSV
Revert "Set resync to 6h"

This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf.

Add delete handler for copied csv
Revert whitespace change
Rename function, fix comment

Upstream-repository: operator-lifecycle-manager
Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21)

…3597) * (bugfix): reduce frequency of update requests for CSVs by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields. This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs. Signed-off-by: everettraven <everettraven@gmail.com> * update unit tests Signed-off-by: everettraven <everettraven@gmail.com> * updates to test so far Signed-off-by: everettraven <everettraven@gmail.com> * Small changes Signed-off-by: Brett Tofel <btofel@redhat.com> * Add metadata drift guard to copyToNamespace Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation. This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now: • Updates the spec and hash annotations • Updates the status subresource • Records the new generation and resourceVersion in the guard annotations Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn. Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact. Signed-off-by: Brett Tofel <btofel@redhat.com> * Tests for metadata guard Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match. Signed-off-by: Brett Tofel <btofel@redhat.com> * Persist observed annotations on all status updates Signed-off-by: Brett Tofel <btofel@redhat.com> * GCI the file Signed-off-by: Brett Tofel <btofel@redhat.com> * Use TransformFunc Unit tests not updated Signed-off-by: Todd Short <todd.short@me.com> * Update operatorgroup tests to compile Signed-off-by: Todd Short <todd.short@me.com> * Restore operatorgroup_test from master Remove metadatalister Signed-off-by: Todd Short <todd.short@me.com> * Remove more PartialObjectMetadata Signed-off-by: Todd Short <todd.short@me.com> * Remove hashes from operator_test Signed-off-by: Todd Short <todd.short@me.com> * Fix error messages for static-analysis Signed-off-by: Todd Short <todd.short@me.com> * Update test annotations and test client Signed-off-by: Todd Short <todd.short@me.com> * Rename pruning to listerwatcher Signed-off-by: Todd Short <todd.short@me.com> * Set resync to 6h Signed-off-by: Todd Short <todd.short@me.com> * Add CSV copy revert syncer Signed-off-by: Todd Short <todd.short@me.com> * Log tweaks Signed-off-by: Todd Short <todd.short@me.com> * Consolidate revert and gc syncers Signed-off-by: Todd Short <todd.short@me.com> * Add logging and reduce the amount of metadata in the TransformFunc Signed-off-by: Todd Short <todd.short@me.com> * Handle the copy CSV revert via a requeue of the primary CSV Signed-off-by: Todd Short <todd.short@me.com> * Revert "Set resync to 6h" This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf. Signed-off-by: Todd Short <todd.short@me.com> * Add delete handler for copied csv Signed-off-by: Todd Short <todd.short@me.com> * Revert whitespace change Signed-off-by: Todd Short <todd.short@me.com> * Rename function, fix comment Signed-off-by: Todd Short <todd.short@me.com> --------- Signed-off-by: everettraven <everettraven@gmail.com> Signed-off-by: Brett Tofel <btofel@redhat.com> Signed-off-by: Todd Short <todd.short@me.com> Co-authored-by: everettraven <everettraven@gmail.com> Co-authored-by: Brett Tofel <btofel@redhat.com> Upstream-repository: operator-lifecycle-manager Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21)

openshift-ci-robot · 2025-07-11T14:49:01Z

@tmshort: This pull request references Jira Issue OCPBUGS-59253, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.17.z) matches configured target version for branch (4.17.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-58882 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-58882 targets the "4.18.z" version, which is one of the valid target versions: 4.18.0, 4.18.z
bug has dependents

Requesting review from QA contact:
/cc @kuiwang02

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Cherry-pick from release-4.18

(bugfix): reduce frequency of update requests for CSVs

by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields.

This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs.

update unit tests

updates to test so far

Small changes

Add metadata drift guard to copyToNamespace

Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation.

This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now:
 • Updates the spec and hash annotations
 • Updates the status subresource
 • Records the new generation and resourceVersion in the guard annotations
Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn.

Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact.

Tests for metadata guard

Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match.

Persist observed annotations on all status updates

GCI the file

Use TransformFunc

Unit tests not updated

Update operatorgroup tests to compile

Restore operatorgroup_test from master

Remove metadatalister

Remove more PartialObjectMetadata

Remove hashes from operator_test

Fix error messages for static-analysis

Update test annotations and test client

Rename pruning to listerwatcher

Set resync to 6h

Add CSV copy revert syncer

Log tweaks

Consolidate revert and gc syncers

Add logging and reduce the amount of metadata in the TransformFunc

Handle the copy CSV revert via a requeue of the primary CSV

Revert "Set resync to 6h"

This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf.

Add delete handler for copied csv

Revert whitespace change

Rename function, fix comment

Upstream-repository: operator-lifecycle-manager
Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tmshort · 2025-07-11T14:49:12Z

/approve

tmshort · 2025-07-11T19:22:06Z

/retest-required

tmshort · 2025-07-14T13:25:46Z

/retest-required

tmshort · 2025-07-14T18:07:28Z

/retest

camilamacedo86

/lgtm

camilamacedo86

/approved

openshift-ci · 2025-07-14T19:51:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~DOWNSTREAM_OWNERS~~ [tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tmshort · 2025-07-15T02:38:21Z

/retest-required

camilamacedo86 · 2025-07-15T06:52:02Z

/test e2e-gcp-olm

camilamacedo86 · 2025-07-15T10:01:35Z

/test e2e-gcp-olm

tmshort · 2025-07-15T15:40:04Z

/retest-required

tmshort · 2025-07-15T20:09:21Z

/retest-required

tmshort · 2025-07-16T00:23:22Z

/retest-required

tmshort · 2025-07-16T08:09:15Z

/retest-required

tmshort · 2025-07-16T13:20:01Z

/retest-required

tmshort · 2025-07-16T17:18:40Z

/retest-required

tmshort · 2025-07-17T00:27:17Z

/retest-required

camilamacedo86 · 2025-07-17T08:03:17Z

/test e2e-gcp-olm

tmshort · 2025-07-17T13:01:46Z

/retest-required

tmshort · 2025-07-17T20:02:00Z

/retest-required

tmshort · 2025-07-18T01:32:49Z

/retest-required

camilamacedo86 · 2025-07-19T04:53:25Z

/test e2e-gcp-olm

tmshort · 2025-07-21T03:34:10Z

/retest

openshift-ci · 2025-07-21T06:58:39Z

@tmshort: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

oceanc80 · 2025-07-22T03:54:22Z

/label backport-risk-assessed

openshift-ci-robot · 2025-07-22T03:57:36Z

@tmshort: Jira Issue OCPBUGS-59253: All pull requests linked via external trackers have merged:

openshift/operator-framework-olm#1036

Jira Issue OCPBUGS-59253 has been moved to the MODIFIED state.

In response to this:

Cherry-pick from release-4.18

(bugfix): reduce frequency of update requests for CSVs

by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields.

This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs.

update unit tests

updates to test so far

Small changes

Add metadata drift guard to copyToNamespace

Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation.

This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now:
 • Updates the spec and hash annotations
 • Updates the status subresource
 • Records the new generation and resourceVersion in the guard annotations
Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn.

Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact.

Tests for metadata guard

Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match.

Persist observed annotations on all status updates

GCI the file

Use TransformFunc

Unit tests not updated

Update operatorgroup tests to compile

Restore operatorgroup_test from master

Remove metadatalister

Remove more PartialObjectMetadata

Remove hashes from operator_test

Fix error messages for static-analysis

Update test annotations and test client

Rename pruning to listerwatcher

Set resync to 6h

Add CSV copy revert syncer

Log tweaks

Consolidate revert and gc syncers

Add logging and reduce the amount of metadata in the TransformFunc

Handle the copy CSV revert via a requeue of the primary CSV

Revert "Set resync to 6h"

This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf.

Add delete handler for copied csv

Revert whitespace change

Rename function, fix comment

Upstream-repository: operator-lifecycle-manager
Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2025-07-22T08:57:15Z

[ART PR BUILD NOTIFIER]

Distgit: operator-lifecycle-manager
This PR has been included in build operator-lifecycle-manager-container-v4.17.0-202507211835.p0.g21ac339.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-07-22T09:09:18Z

[ART PR BUILD NOTIFIER]

Distgit: ose-operator-framework-tools
This PR has been included in build ose-operator-framework-tools-container-v4.17.0-202507211835.p0.g21ac339.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-07-22T09:21:22Z

[ART PR BUILD NOTIFIER]

Distgit: operator-registry
This PR has been included in build operator-registry-container-v4.17.0-202507211835.p0.g21ac339.assembly.stream.el9.
All builds following this will include this PR.

tmshort · 2025-07-22T13:36:09Z

/cherry-pick release-4.16

openshift-cherrypick-robot · 2025-07-22T13:37:05Z

@tmshort: #1036 failed to apply on top of branch "release-4.16":

Applying: OCPBUGS-37982: Reduce Frequency of Update Requests for Copied CSVs (#3597)
Using index info to reconstruct a base tree...
M	staging/operator-lifecycle-manager/pkg/controller/operators/catalog/operator.go
M	staging/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go
M	staging/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup_test.go
M	vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/operator.go
M	vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go
M	vendor/modules.txt
Falling back to patching base and 3-way merge...
Auto-merging vendor/modules.txt
Auto-merging vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go
Removing vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/internal/pruning/listerwatcher.go
Auto-merging vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/operator.go
Auto-merging staging/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup_test.go
CONFLICT (content): Merge conflict in staging/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup_test.go
Auto-merging staging/operator-lifecycle-manager/pkg/controller/operators/olm/operatorgroup.go
Removing staging/operator-lifecycle-manager/pkg/controller/operators/internal/pruning/listerwatcher.go
Auto-merging staging/operator-lifecycle-manager/pkg/controller/operators/catalog/operator.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 OCPBUGS-37982: Reduce Frequency of Update Requests for Copied CSVs (#3597)

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2025-07-22T22:27:13Z

Fix included in accepted release 4.17.0-0.nightly-2025-07-22-124947

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jul 11, 2025

openshift-ci bot requested review from kuiwang02, joelanford and perdasilva July 11, 2025 14:49

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 11, 2025

camilamacedo86 reviewed Jul 14, 2025

View reviewed changes

camilamacedo86 approved these changes Jul 14, 2025

View reviewed changes

openshift-ci bot assigned camilamacedo86 Jul 14, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 14, 2025

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jul 22, 2025

openshift-ci bot assigned bandrade, emmajiafan, jianzhangbjz, KeenonLee, kuiwang02 and Xia-Zhao-rh Jul 22, 2025

openshift-merge-bot bot merged commit 21ac339 into openshift:release-4.17 Jul 22, 2025
14 checks passed

tmshort deleted the OCPBUGS-59253 branch July 22, 2025 13:35

[release-4.17] OCPBUGS-59253: Reduce Frequency of Update Requests for Copied CSVs (#3597) #1036

[release-4.17] OCPBUGS-59253: Reduce Frequency of Update Requests for Copied CSVs (#3597) #1036

Uh oh!

Conversation

tmshort commented Jul 11, 2025

Uh oh!

openshift-ci-robot commented Jul 11, 2025

Uh oh!

tmshort commented Jul 11, 2025

Uh oh!

tmshort commented Jul 11, 2025

Uh oh!

tmshort commented Jul 14, 2025

Uh oh!

tmshort commented Jul 14, 2025

Uh oh!

camilamacedo86 left a comment

Choose a reason for hiding this comment

Uh oh!

camilamacedo86 left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jul 14, 2025

Uh oh!

tmshort commented Jul 15, 2025

Uh oh!

camilamacedo86 commented Jul 15, 2025

Uh oh!

camilamacedo86 commented Jul 15, 2025

Uh oh!

tmshort commented Jul 15, 2025

Uh oh!

tmshort commented Jul 15, 2025

Uh oh!

tmshort commented Jul 16, 2025

Uh oh!

tmshort commented Jul 16, 2025

Uh oh!

tmshort commented Jul 16, 2025

Uh oh!

tmshort commented Jul 16, 2025

Uh oh!

tmshort commented Jul 17, 2025

Uh oh!

camilamacedo86 commented Jul 17, 2025

Uh oh!

tmshort commented Jul 17, 2025

Uh oh!

tmshort commented Jul 17, 2025

Uh oh!

tmshort commented Jul 18, 2025

Uh oh!

camilamacedo86 commented Jul 19, 2025

Uh oh!

tmshort commented Jul 21, 2025

Uh oh!

openshift-ci bot commented Jul 21, 2025

Uh oh!

oceanc80 commented Jul 22, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 22, 2025

Uh oh!

openshift-bot commented Jul 22, 2025

Uh oh!

openshift-bot commented Jul 22, 2025

Uh oh!

openshift-bot commented Jul 22, 2025

Uh oh!

tmshort commented Jul 22, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 22, 2025

Uh oh!

openshift-merge-robot commented Jul 22, 2025

Uh oh!

Uh oh!