Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-17433: Sync featuregate controller during the node config controller sync #3846

Merged
merged 1 commit into from Aug 11, 2023

Conversation

sairameshv
Copy link
Member

  1. Changes in the nodes.config.openshift.io object results in the create/update of 97-{master/worker}-generated-kubelet machine configs
  2. However, these changes are not rendered into the master/worker mcps if 98-{master/worker}-generated-kubelet or 99-{master/worker}-generated-kubelet machine configs exist due to precedence.
  3. Hence, in order to bring in the change caused by nodes.config object's update, the respective sync functions of featuregate controller and the Kubelet config controller need to be called/executed in the syncNodeConfig function.
  4. syncKubeletConfig function is already getting called and this code change introduces the calling of syncFeatureHandler function
  5. Also added a minor fix in featuregate controller sync to apply the workerlatencyprofiles change only to the worker mcp

Fixes: https://issues.redhat.com/browse/OCPBUGS-17433

1. Changes in the `nodes.config.openshift.io` object results in the create/update of `97-{master/worker}-generated-kubelet` machine configs
2. However, these changes are not rendered into the `master/worker` mcps if `98-{master/worker}-generated-kubelet` or `99-{master/worker}-generated-kubelet`
machine configs exist due to precedence.
3. Hence, in order to bring in the change caused by nodes.config object's update, the respective sync functions of featuregate controller and the Kubelet config controller
need to be called/executed in the syncNodeConfig function.
4. `syncKubeletConfig` function is already getting called and this code change introduces the calling of `syncFeatureHandler` function
5. Also added a minor fix in featuregate controller sync to apply the workerlatencyprofiles change only to the worker mcp

Signed-off-by: Sai Ramesh Vanka <svanka@redhat.com>
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 9, 2023
@openshift-ci-robot
Copy link
Contributor

@sairameshv: This pull request references Jira Issue OCPBUGS-17433, which is invalid:

  • expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

  1. Changes in the nodes.config.openshift.io object results in the create/update of 97-{master/worker}-generated-kubelet machine configs
  2. However, these changes are not rendered into the master/worker mcps if 98-{master/worker}-generated-kubelet or 99-{master/worker}-generated-kubelet machine configs exist due to precedence.
  3. Hence, in order to bring in the change caused by nodes.config object's update, the respective sync functions of featuregate controller and the Kubelet config controller need to be called/executed in the syncNodeConfig function.
  4. syncKubeletConfig function is already getting called and this code change introduces the calling of syncFeatureHandler function
  5. Also added a minor fix in featuregate controller sync to apply the workerlatencyprofiles change only to the worker mcp

Fixes: https://issues.redhat.com/browse/OCPBUGS-17433

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sairameshv
Copy link
Member Author

/jira refresh

@openshift-ci-robot
Copy link
Contributor

@sairameshv: This pull request references Jira Issue OCPBUGS-17433, which is invalid:

  • expected the bug to target the "4.14.0" version, but it targets "4.14" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sairameshv
Copy link
Member Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 9, 2023
@openshift-ci-robot
Copy link
Contributor

@sairameshv: This pull request references Jira Issue OCPBUGS-17433, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.0) matches configured target version for branch (4.14.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (schoudha@redhat.com), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sairameshv
Copy link
Member Author

/cc @yuqi-zhang @harche @rphillips

@sairameshv
Copy link
Member Author

/retest

@LorbusChris
Copy link
Member

/test okd-scos-images

@wking
Copy link
Member

wking commented Aug 10, 2023

@LorbusChris pushed an older build to registry.ci.openshift.org/origin/scos-4.14:centos-stream-coreos-9 and the backing quay.io/okd/centos-stream-coreos-9 images to recover the io.openshift.build.versions=machine-os=... labels whose recent removals had lead to unknown version reference "machine-os".

/retest

Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions below:

@@ -177,6 +177,14 @@ func (ctrl *Controller) syncNodeConfigHandler(key string) error {
}
}

// syncing the featuregate controller
features, err := ctrl.featLister.Get(ctrlcommon.ClusterFeatureInstanceName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should work. This would also be an issue for kubeletconfigcontroller generating the 99-xx off a user config right? This cluster that had the bug only had 97 and 98, but due to how our hierarchy works we would need to also call sync kubelet config here (and then in features function, also call kubelet config sync too.

The alternative would be for features controller to also watch changes to nodes.config, and kubelet config controller to also watch for features and nodes.config.

Longer term though I think this hierarchy is probably not the best way to do this as it is not very scalable, but for now I think we should focus on fixing this issue since I suspect it will show up more often with the setting cgroups via nodes.config we added to 4.13

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree with your comment. The issue with Kubeconfigcontroller generating 99-XX has already been taken care as of now in the node config controller here
But going forward, we might have to come up with another approach to overcome this complexity.

Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@yuqi-zhang
Copy link
Contributor

/hold

For QE pre-merge verification. Feel free to unhold if that's not relevant here

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 10, 2023
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 10, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sairameshv, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2023
@sunilcio
Copy link

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Aug 11, 2023
@sairameshv
Copy link
Member Author

/hold cancel

This change has been validated and hence ready for the merge.

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 11, 2023
@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD beca7a6 and 2 for PR HEAD 1b56ad7 in total

@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 4048c66 and 1 for PR HEAD 1b56ad7 in total

@sairameshv
Copy link
Member Author

/retest

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 11, 2023

@sairameshv: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 63d7be1 into openshift:master Aug 11, 2023
13 checks passed
@openshift-ci-robot
Copy link
Contributor

@sairameshv: Jira Issue OCPBUGS-17433: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-17433 has been moved to the MODIFIED state.

In response to this:

  1. Changes in the nodes.config.openshift.io object results in the create/update of 97-{master/worker}-generated-kubelet machine configs
  2. However, these changes are not rendered into the master/worker mcps if 98-{master/worker}-generated-kubelet or 99-{master/worker}-generated-kubelet machine configs exist due to precedence.
  3. Hence, in order to bring in the change caused by nodes.config object's update, the respective sync functions of featuregate controller and the Kubelet config controller need to be called/executed in the syncNodeConfig function.
  4. syncKubeletConfig function is already getting called and this code change introduces the calling of syncFeatureHandler function
  5. Also added a minor fix in featuregate controller sync to apply the workerlatencyprofiles change only to the worker mcp

Fixes: https://issues.redhat.com/browse/OCPBUGS-17433

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants