OCPBUGS-24792: Make MC names deterministic #875
Conversation
OpenShift Nodes can be a part of only one custom MachineConfigPool. However, MCP configuration allows this not to be the case. This is caught by the machine-config-controller and reported as an error (<node> belongs to <N> custom roles, cannot proceed with this Node). In order to target an MCP with a configuration, NTO uses machineConfigLabels. However, one or more MCPs can select a particular single MC. This is due to the MCP's machineConfigSelector. This is another challenge scenario. In the above two scenarios, it was possible for NTO to generate a randomly-named MC based on the membership of one of the matching MCPs. Then, a pruning function would mistakenly remove the other MCs, seemingly unused. This could result in a flip between the rendered MCs and cause a Node reboot. This PR makes the process of establishing the names of the MC for the purposes of MachineConfigPool based matching deterministic. Other changes/fixes: - Synced with MCO's latest getPoolsForNode() changes. - Logging in syncMachineConfigHyperShift(). Resolves: OCPBUGS-24792
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
@jmencak: GitHub didn't allow me to request PR reviews from the following users: jmencak. Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jmencak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/cc @liqcui |
jmencak
changed the title
openshift-ci-robot
added
jira/severity-moderate
|
@jmencak: This pull request references Jira Issue OCPBUGS-24792, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request. The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
@jmencak: This pull request references Jira Issue OCPBUGS-24792, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cc @yanirq |
|
Code wise this looks good to me. Would we like to have functional tests for this deterministic behavior ? Its a heavy test to perform so Im not saying Im in favor, just thinking out loud. cc @MarSik In case you want to weigh in. |
|
Thank you for the review, much appreciated.
Good observation. However, that is not my aim. My aim was to stop the restart loop. I'd still like the customers did not use this, because then this basically wides the the pool of machines that need the same HW across all the pools that match. We should probably document that we strongly recommend they do not do this.
Possibly, but not sure whether in this repo too. We have similar tests in openshift/origin for this. And yes, it will be time consuming. Also, if as part of this PR, please note that I have a fix for OCPBUGS-24636 ready waiting for this to merge. |
pkg/operator/controller.go
Outdated
| return nil | ||
| } | ||
|
|
||
| if ok := c.allNodesAgreeOnBootcmdline(nodes); !ok { |
There was a problem hiding this comment.
If you want to support multiple pools matching the recommendation section of a single Tuned, then I think this loop has to extend further below this. Because you fail the whole sync on two MCPs not having the same kargs. Which they are not required to have, that condition only applies to each MCP separately.
There was a problem hiding this comment.
If you want to support multiple pools matching the recommendation section of a single Tuned
@MarSik , I believe there is a fundamental misunderstanding what this change does. We are not adding support, we are fixing a potential cyclic reboot issue that was not fixed by previous code.
So my aim is not to support multiple MCPs, but we need to check all the matching MCPs affected by this. Would logging an error or warning when we see multiple matching MCPs be sufficient?
There was a problem hiding this comment.
But it essentially does add the support for multiple MCPs.
You can block the sync when multiple MCPs are detected instead to make it clear that is is not supported. Not need to validate kargs in such case.
There was a problem hiding this comment.
But it essentially does add the support for multiple MCPs.
Let's agree to disagree. It fixes a bug.
You can block the sync when multiple MCPs are detected instead to make it clear that is is not supported. Not need to validate kargs in such case.
No, we still need to check the kargs. You can have machines with different topology within a single MCP.
There was a problem hiding this comment.
@MarSik , please see the "Enforce per-pool machineConfigLabels selectors" commit. Hopefully it resolves the issues you see with this PR.
|
Hmm, we can either improve the support for multiple MCPs (like I commented in the code) or block the approach by either a check or by generating a sufficiently unique label set for the new MC so it is matched by a single MCP only. |
As mentioned above, we are not adding support for multipe MCPs. Also, we cannot generate a MCP label, because that's what customers/PPC specifies. |
|
Liquan found an issue when manually running one test scenario. Cannot reproduce it, but let's see if it is real first. |
openshift-ci
bot
added
the
do-not-merge/hold
|
/lgtm |
It's not a real issue. re-test is ok |
Thank you, still keeping the hold for now. |
|
Keeping a hold for now as I want to do more manual testing on SNO. Will unhold after that. |
|
Manual tests on SNO worked fine. |
openshift-ci
bot
removed
the
do-not-merge/hold
| // Log an error and do not requeue, this is a configuration issue. | ||
| klog.Errorf("profile %v uses machineConfigLabels that match across multiple MCPs (%v); this is not supported", | ||
| profile.Name, printMachineConfigPoolsNames(pools)) | ||
| return nil |
There was a problem hiding this comment.
Should this really return success? What was wrong on returning the error? (the other log update below on line 798)
There was a problem hiding this comment.
In my view yes. You don't want to requeue this error 15 times until this is dropped and spam the logs. This is a configuration issue, user has been notified by logging an error. Requeing would not help anything.
There was a problem hiding this comment.
Ack, so the only place we will report this at will be the NTO log?
|
/lgtm But we should get together and figure out a better error reporting scheme. Right now we only have the information about errors in logs only. Users do not notice those from the cluster management level. |
I propose doing a similar thing MCO does when it detects issues with its MCPs, it raises alerts. |
|
/retest |
1 similar comment
|
/retest |
|
/retest |
|
@jmencak: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@jmencak: Jira Issue OCPBUGS-24792: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-24792 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[ART PR BUILD NOTIFIER] This PR has been included in build cluster-node-tuning-operator-container-v4.16.0-202312150911.p0.g778695d.assembly.stream for distgit cluster-node-tuning-operator. |
* Make MC names deterministic OpenShift Nodes can be a part of only one custom MachineConfigPool. However, MCP configuration allows this not to be the case. This is caught by the machine-config-controller and reported as an error (<node> belongs to <N> custom roles, cannot proceed with this Node). In order to target an MCP with a configuration, NTO uses machineConfigLabels. However, one or more MCPs can select a particular single MC. This is due to the MCP's machineConfigSelector. This is another challenge scenario. In the above two scenarios, it was possible for NTO to generate a randomly-named MC based on the membership of one of the matching MCPs. Then, a pruning function would mistakenly remove the other MCs, seemingly unused. This could result in a flip between the rendered MCs and cause a Node reboot. This PR makes the process of establishing the names of the MC for the purposes of MachineConfigPool based matching deterministic. Other changes/fixes: - Synced with MCO's latest getPoolsForNode() changes. - Logging in syncMachineConfigHyperShift(). Resolves: OCPBUGS-24792 * Enforce per-pool machineConfigLabels selectors --------- Co-authored-by: Jiri Mencak <jmencak@users.noreply.github.com>
|
/cherry-pick release-4.15 |
|
@jmencak: new pull request created: #888 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
OpenShift Nodes can be a part of only one custom MachineConfigPool. However, MCP configuration allows this not to be the case. This is caught by the machine-config-controller and reported as an error ( belongs to custom roles, cannot proceed with this Node).
In order to target an MCP with a configuration, NTO uses machineConfigLabels. However, one or more MCPs can select a particular single MC. This is due to the MCP's machineConfigSelector. This is another challenge scenario.
In the above two scenarios, it was possible for NTO to generate a randomly-named MC based on the membership of one of the matching MCPs. Then, a pruning function would mistakenly remove the other MCs, seemingly unused. This could result in a flip between the rendered MCs and cause a Node reboot.
This PR makes the process of establishing the names of the MC for the purposes of MachineConfigPool based matching deterministic.
Other changes/fixes:
Resolves: OCPBUGS-24792