OCPBUGS-17585: Tighten the rules for modifying Tuned Profiles #775
Conversation
|
@jmencak: GitHub didn't allow me to request PR reviews from the following users: jmencak. Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
added
the
approved
|
/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade 10 |
|
@jmencak: trigger 1 job(s) for the /payload-(job|aggregate) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4c0e1b00-4668-11ee-83f0-1bac13c94f74-0 |
|
@jmencak: The specified target(s) for
Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
jmencak
force-pushed
the
4.14-OCPBUGS-17585-profiles-subresource
branch
2 times, most recently
from
a11c6d7
to
38b7d94
Compare
|
/retest |
|
/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade 10 |
|
@jmencak: trigger 1 job(s) for the /payload-(job|aggregate) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/829adea0-46fa-11ee-9c97-6d9651431b8b-0 |
|
/cc @MarSik |
jmencak
force-pushed
the
4.14-OCPBUGS-17585-profiles-subresource
branch
from
38b7d94
to
8d65cd4
Compare
NTO operands allow updating Tuned Profiles. This is intentional and by design
as some host information needs to be communicated back to the NTO operator,
but this also allows a successful local attacker potentially affect node-level
configuration of other cluster nodes.
This change addresses the situation in two ways. First, scoped RBAC
permissions on Profile.status subresource is used to disallow Node-level
write access to Profile.spec. Second, the Node resource is used to provide
status loops back to NTO using the kubelet credential to write an annotation
to the Node resource.
This change also simplifies the mechanism for accepting kernel
command-line parameters as calculated by the NTO operands. Now, all NTO
operands must agree on the calculated kernel command-line parameters.
ClusterOperator/node-tuning now also reports operand version. The
operand version changes only when all operand replicas have successfully
upgraded and are ready. This information is used to block PPC
Reconcilation loop until the operator and operand RELEASE_VERSION agree.
This is a short-term measure to prevent from multiple node reboots
during upgrades.
Other fixes:
- Disallow MC updates during upgrades when kernel command-line
parameters of nodes within a MCP do not match.
- ClusterOperator/node-tuning object was sometimes giving false
information based on an old Metadata.Generation.
Resolves: OCPBUGS-17585
jmencak
force-pushed
the
4.14-OCPBUGS-17585-profiles-subresource
branch
from
8d65cd4
to
c756bbd
Compare
jmencak
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@jmencak: This pull request references Jira Issue OCPBUGS-17585, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request. The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade 10 |
|
@jmencak: trigger 1 job(s) for the /payload-(job|aggregate) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9e0ccd00-4732-11ee-97fb-2046709d4fbf-0 |
|
@jmencak: This pull request references Jira Issue OCPBUGS-17585, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@jmencak: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@MarSik, all tests passed and the blocking job also succeeded successfully all 3 times I tried. I belive I've hopefully also addressed your concerns. |
| return nil | ||
| } | ||
|
|
||
| bootcmdline := profile.Status.Bootcmdline | ||
| if ok := c.allNodesAgreeOnBootcmdline(nodes); !ok { | ||
| return fmt.Errorf("not all %d Nodes in MCP %v agree on bootcmdline: %s", len(nodes), pools[0].ObjectMeta.Name, bootcmdline) |
There was a problem hiding this comment.
Will this propagate to the Status section of the Tuned object?
There was a problem hiding this comment.
Not sure what you mean. Tuned CR/object doesn't really use status.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jmencak, MarSik The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Let's do this as a follow up PR, this needs to get in. /label acknowledge-critical-fixes-only |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
|
@jmencak: Jira Issue OCPBUGS-17585: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-17585 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
NTO operands allow updating Tuned Profiles. This is intentional and by design
as some host information needs to be communicated back to the NTO operator,
but this also allows a successful local attacker potentially affect node-level
configuration of other cluster nodes.
This change addresses the situation in two ways. First, scoped RBAC
permissions on Profile.status subresource is used to disallow Node-level
write access to Profile.spec. Second, the Node resource is used to provide
status loops back to NTO using the kubelet credential to write an annotation
to the Node resource.
This change also simplifies the mechanism for accepting kernel
command-line parameters as calculated by the NTO operands. Now, all NTO
operands must agree on the calculated kernel command-line parameters.
ClusterOperator/node-tuning now also reports operand version. The
operand version changes only when all operand replicas have successfully
upgraded and are ready. This information is used to block PPC
Reconcilation loop until the operator and operand RELEASE_VERSION agree.
This is a short-term measure to prevent from multiple node reboots
during upgrades.
Other fixes:
parameters of nodes within a MCP do not match.
information based on an old Metadata.Generation.
Resolves: OCPBUGS-17585