Skip to content

Conversation

camilamacedo86
Copy link
Contributor

@camilamacedo86 camilamacedo86 commented Oct 6, 2025

Description

ClusterExtension upgrade could fail due to oversized CRD validation status updates

When upgrading a ClusterExtension, CRD validation status updates could exceed the Kubernetes 32 KB limit, causing the update to fail with the error “Too long: may not be more than 32768 bytes.” This prevented the ClusterExtension status from being updated, leaving no status conditions or information about why the upgrade did not occur.

This issue occurred when the crddiff library returned a large JSON diff for unhandled CRD changes.

The problem has been resolved by truncating and summarising the diff output for unhandled scenarios, instead of including the full JSON diff. This ensures status updates remain within size limits and users receive clear, actionable error messages.

Cherry-picks

More info

To address the fix verified for 4.21 to 4.20 ( version where the bug was reported initially).
More info: https://issues.redhat.com/browse/OCPBUGS-59518?focusedId=28188598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28188598

…itions (#2169)

When upgrading operators, CRD validation errors can be very large (50KB+).
Kubernetes rejects status updates over 32KB with "Too long: may not be more than 32768 bytes".
This causes ClusterExtension upgrades to fail and get stuck.

Assisted-by: Cursor
…CPBUGS-59518 ) (#2179)

* (fix): unhandle changes for crd upgrade safety

- Keep unhandled spec changes as errors; message: "unhandled changes found"

Assisted-by: Cursor

* enhance to extract details from unhandle issue
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 6, 2025
@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is invalid:

  • expected the bug to target the "4.20.0" version, but no target version was set
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-59518 to target a version in 4.21.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

To address the fix verified for 4.21 to 4.20 ( version where the bug was reported initially).
More info: https://issues.redhat.com/browse/OCPBUGS-59518?focusedId=28188598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28188598

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-59518 to target a version in 4.21.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from dtfranz and tmshort October 6, 2025 09:34
@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Description

When upgrading a ClusterExtension, CRD validation status updates could exceed the Kubernetes 32 KB limit, causing the update to fail with the error “Too long: may not be more than 32768 bytes.” This prevented the ClusterExtension status from being updated, leaving no status conditions or information about why the upgrade did not occur.

This issue occurred when the crddiff library returned a large JSON diff for unhandled CRD changes.

The problem has been resolved by truncating and summarising the diff output for unhandled scenarios, instead of including the full JSON diff. This ensures status updates remain within size limits and users receive clear, actionable error messages.

Cherry-picks

More info

To address the fix verified for 4.21 to 4.20 ( version where the bug was reported initially).
More info: https://issues.redhat.com/browse/OCPBUGS-59518?focusedId=28188598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28188598

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 6, 2025
@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-59518 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-59518 targets the "4.21.0" version, which is one of the valid target versions: 4.21.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (jiazha@redhat.com), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bandrade
Copy link

bandrade commented Oct 6, 2025

/label qe-approved
/verified by @bandrade

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Oct 6, 2025
@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 6, 2025
@openshift-ci-robot
Copy link

@camilamacedo86: This pull request references Jira Issue OCPBUGS-62722, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-59518 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-59518 targets the "4.21.0" version, which is one of the valid target versions: 4.21.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (jiazha@redhat.com), skipping review request.

In response to this:

Description

ClusterExtension upgrade could fail due to oversized CRD validation status updates

When upgrading a ClusterExtension, CRD validation status updates could exceed the Kubernetes 32 KB limit, causing the update to fail with the error “Too long: may not be more than 32768 bytes.” This prevented the ClusterExtension status from being updated, leaving no status conditions or information about why the upgrade did not occur.

This issue occurred when the crddiff library returned a large JSON diff for unhandled CRD changes.

The problem has been resolved by truncating and summarising the diff output for unhandled scenarios, instead of including the full JSON diff. This ensures status updates remain within size limits and users receive clear, actionable error messages.

Cherry-picks

More info

To address the fix verified for 4.21 to 4.20 ( version where the bug was reported initially).
More info: https://issues.redhat.com/browse/OCPBUGS-59518?focusedId=28188598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28188598

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

@bandrade: This PR has been marked as verified by @bandrade.

In response to this:

/label qe-approved
/verified by @bandrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@tmshort
Copy link
Contributor

tmshort commented Oct 6, 2025

/lgtm
/approved

@tmshort
Copy link
Contributor

tmshort commented Oct 6, 2025

Still have a problem in techpreview

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 6, 2025
@camilamacedo86
Copy link
Contributor Author

/test openshift-e2e-aws-techpreview

Copy link
Contributor

openshift-ci bot commented Oct 7, 2025

@camilamacedo86: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@tmshort
Copy link
Contributor

tmshort commented Oct 7, 2025

/approve

Copy link
Contributor

openshift-ci bot commented Oct 7, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 7, 2025
@oceanc80
Copy link
Contributor

oceanc80 commented Oct 7, 2025

/label backport-risk-assessed

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Oct 7, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit 0df53a9 into openshift:release-4.20 Oct 7, 2025
12 checks passed
@openshift-ci-robot
Copy link

@camilamacedo86: Jira Issue OCPBUGS-62722: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-62722 has been moved to the MODIFIED state.

In response to this:

Description

ClusterExtension upgrade could fail due to oversized CRD validation status updates

When upgrading a ClusterExtension, CRD validation status updates could exceed the Kubernetes 32 KB limit, causing the update to fail with the error “Too long: may not be more than 32768 bytes.” This prevented the ClusterExtension status from being updated, leaving no status conditions or information about why the upgrade did not occur.

This issue occurred when the crddiff library returned a large JSON diff for unhandled CRD changes.

The problem has been resolved by truncating and summarising the diff output for unhandled scenarios, instead of including the full JSON diff. This ensures status updates remain within size limits and users receive clear, actionable error messages.

Cherry-picks

More info

To address the fix verified for 4.21 to 4.20 ( version where the bug was reported initially).
More info: https://issues.redhat.com/browse/OCPBUGS-59518?focusedId=28188598&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28188598

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86 camilamacedo86 deleted the cherry-pick-OCPBUGS-62722 branch October 7, 2025 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR verified Signifies that the PR passed pre-merge verification criteria
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants