Skip to content

OCPBUGS-90083: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps#8811

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:release-4.22from
vsolanki12:cherry-pick-8543-to-release-4.22
Jul 14, 2026
Merged

OCPBUGS-90083: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps#8811
openshift-merge-bot[bot] merged 1 commit into
openshift:release-4.22from
vsolanki12:cherry-pick-8543-to-release-4.22

Conversation

@vsolanki12

Copy link
Copy Markdown
Contributor

Backport of #8543 to release-4.22.

Adapted cherry-pick to use supportutil.DeleteIfNeededWithPredicate and supportutil.ShortenName instead of k8sutil/netutil which don't exist on the release branch (they were extracted from support/util in a later refactor on main).

Mirrored ConfigMaps created by the NodePool NTO reconciler were
set with immutable=true, which prevented Kubernetes from accepting
in-place updates when the source ConfigMap data changed. This
caused NodePools sharing the same kubelet ConfigMap to get stuck
in UpdatingConfig=True with "field is immutable" errors.

Remove the immutable flag from mutateMirroredConfig and add a
migration helper that uses DeleteIfNeededWithPredicate to safely
delete existing immutable mirrored ConfigMaps before
CreateOrUpdate recreates them as mutable. The predicate guards
deletion to only ConfigMaps with the nto-mirrored-config label
belonging to the current NodePool.

Signed-off-by: Vimal Solanki <vsolanki@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 23, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-85778, which is invalid:

  • expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "5.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-85778 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Backport of #8543 to release-4.22.

Adapted cherry-pick to use supportutil.DeleteIfNeededWithPredicate and supportutil.ShortenName instead of k8sutil/netutil which don't exist on the release branch (they were extracted from support/util in a later refactor on main).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 17370d9c-7ed4-4ad0-ad81-615123641871

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jun 23, 2026
@openshift-ci openshift-ci Bot requested review from enxebre and jparrill June 23, 2026 12:47
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.46%. Comparing base (5cb8735) to head (3d50484).
⚠️ Report is 23 commits behind head on release-4.22.

Files with missing lines Patch % Lines
hypershift-operator/controllers/nodepool/nto.go 82.35% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@               Coverage Diff                @@
##           release-4.22    #8811      +/-   ##
================================================
+ Coverage         35.45%   35.46%   +0.01%     
================================================
  Files               767      767              
  Lines             93724    93740      +16     
================================================
+ Hits              33226    33246      +20     
+ Misses            57785    57782       -3     
+ Partials           2713     2712       -1     
Files with missing lines Coverage Δ
hypershift-operator/controllers/nodepool/nto.go 51.32% <82.35%> (+2.70%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vsolanki12 vsolanki12 changed the title OCPBUGS-85778: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps OCPBUGS-90083: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps Jun 26, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-90083, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-85778 to target a version in 5.0.0, but it targets "5.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Backport of #8543 to release-4.22.

Adapted cherry-pick to use supportutil.DeleteIfNeededWithPredicate and supportutil.ShortenName instead of k8sutil/netutil which don't exist on the release branch (they were extracted from support/util in a later refactor on main).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-90083, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-85778 to target a version in 5.0.0, but it targets "5.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-90083, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/cc @jparrill @csrwng

@openshift-ci openshift-ci Bot requested a review from csrwng June 26, 2026 10:41
@jparrill

jparrill commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill, vsolanki12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 1, 2026
@vsolanki12

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 7, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-90083, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note type set to "Release Note Not Required"
  • dependent bug Jira Issue OCPBUGS-85778 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-85778 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
  • bug has dependents
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@enxebre enxebre added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jul 9, 2026
@enxebre

enxebre commented Jul 9, 2026

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 9, 2026
@vsolanki12

Copy link
Copy Markdown
Contributor Author

/retest

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/test e2e-aks

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/test e2e-aks-4-21
/test e2e-aws
/test e2e-aws-4-21
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

@vsolanki12

Copy link
Copy Markdown
Contributor Author

Manual Verification on Live Cluster

Tested the immutable mirrored ConfigMap fix on a live AWS based HostedCluster.

Setup:

  • Built custom CPO image from PR OCPBUGS-90083: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps #8811 branch cherry-pick-8543-to-release-4.22 (quay.io/vsolanki/hypershift:OCPBUGS-90083)
  • Created HostedCluster ocpbugs-90083 with release image 4.22.5-multi
  • Annotated with custom CPO: hypershift.openshift.io/control-plane-operator-image=quay.io/vsolanki/hypershift:OCPBUGS-90083
  • Waited for 2 worker nodes to reach Ready state
  • Added KubeletConfig to NodePool with maxPods: 300, confirmed mirrored CM created in guest openshift-config-managed namespace

Test:

Marked the guest-side mirrored ConfigMap as immutable, then updated the source KubeletConfig data to trigger a reconcile:

KUBECONFIG=/tmp/ocpbugs-90083-kubeconfig oc replace -f /tmp/cm-immutable.json

  oc patch configmap test-kubelet-config -n clusters --type=merge \
    -p '{"data":{"config":"apiVersion: machineconfiguration.openshift.io/v1\nkind: KubeletConfig\nmetadata:\n  name: test-kubelet-config\nspec:\n  kubeletConfig:\n
  maxPods: 500\n"}}'

Result:

HCCO detected the immutable ConfigMap, deleted it, and recreated it as mutable with the updated data:

  {"level":"info","ts":"2026-07-13T08:05:16Z","msg":"deleting immutable KubeletConfig ConfigMap to recreate as
  mutable","controller":"resources","configMap":"openshift-config-managed/test-kubelet-config-ocpbugs-90083-ap-south-1a"}

Guest-side ConfigMap confirmed mutable with updated content (maxPods: 300 → maxPods: 500). Fix working as expected on release-4.22.

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/verified by @vsolanki12
check here #8811 (comment)

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 13, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This PR has been marked as verified by @vsolanki12.

Details

In response to this:

/verified by @vsolanki12
check here #8811 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 370b2b2 and 2 for PR HEAD 3d50484 in total

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@vsolanki12: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot Bot merged commit 13ae391 into openshift:release-4.22 Jul 14, 2026
25 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: Jira Issue Verification Checks: Jira Issue OCPBUGS-90083
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-90083 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Backport of #8543 to release-4.22.

Adapted cherry-pick to use supportutil.DeleteIfNeededWithPredicate and supportutil.ShortenName instead of k8sutil/netutil which don't exist on the release branch (they were extracted from support/util in a later refactor on main).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants