OCPBUGS-17941: Tighten the rules for modifying Tuned Profiles

[jmencak]

NTO operands allow updating Tuned Profiles. This is intentional and by design
as some host information needs to be communicated back to the NTO operator,
but this also allows a successful local attacker potentially affect node-level
configuration of other cluster nodes.

This change addresses the situation in two ways. First, scoped RBAC
permissions on Profile.status subresource is used to disallow Node-level
write access to Profile.spec. Second, the Node resource is used to provide
status loops back to NTO using the kubelet credential to write an annotation
to the Node resource.

This change also simplifies the mechanism for accepting kernel
command-line parameters as calculated by the NTO operands. Now, all NTO
operands must agree on the calculated kernel command-line parameters.

ClusterOperator/node-tuning now also reports operand version. The
operand version changes only when all operand replicas have successfully
upgraded and are ready. This information is used to block PPC
Reconcilation loop until the operator and operand RELEASE_VERSION agree.
This is a short-term measure to prevent from multiple node reboots
during upgrades.

Other fixes:

• Disallow MC updates during upgrades when kernel command-line
  parameters of nodes within a MCP do not match.
• ClusterOperator/node-tuning object was sometimes giving false
  information based on an old Metadata.Generation.

Resolves: OCPBUGS-17941

This is a backport of #775

[openshift-ci]

@jmencak: GitHub didn't allow me to request PR reviews from the following users: jmencak.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

NTO operands allow updating Tuned Profiles. This is intentional and by design as some host information needs to be communicated back to the NTO operator, but this also allows a successful local attacker potentially affect node-level configuration of other cluster nodes.

This change addresses the situation in two ways. First, scoped RBAC permissions on Profile.status subresource is used to disallow Node-level write access to Profile.spec. Second, the Node resource is used to provide status loops back to NTO using the kubelet credential to write an annotation to the Node resource.

Resolves: OCPBUGS-17941

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmencak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jmencak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jmencak]

/cc @dagrayvid
This is a straightforward port of #775 to 4.13. Would you be able to take a look when available? Thank you!

[jmencak]

/hold

[openshift-ci-robot]

@jmencak: This pull request references Jira Issue OCPBUGS-17941, which is invalid:

• expected dependent Jira Issue OCPBUGS-17585 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

NTO operands allow updating Tuned Profiles. This is intentional and by design as some host information needs to be communicated back to the NTO operator, but this also allows a successful local attacker potentially affect node-level configuration of other cluster nodes.

This change addresses the situation in two ways. First, scoped RBAC permissions on Profile.status subresource is used to disallow Node-level write access to Profile.spec. Second, the Node resource is used to provide status loops back to NTO using the kubelet credential to write an annotation to the Node resource.

Resolves: OCPBUGS-17941

This is a backport of #765

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@jmencak: This pull request references Jira Issue OCPBUGS-17941, which is invalid:

• expected dependent Jira Issue OCPBUGS-17585 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

NTO operands allow updating Tuned Profiles. This is intentional and by design
as some host information needs to be communicated back to the NTO operator,
but this also allows a successful local attacker potentially affect node-level
configuration of other cluster nodes.

This change addresses the situation in two ways. First, scoped RBAC
permissions on Profile.status subresource is used to disallow Node-level
write access to Profile.spec. Second, the Node resource is used to provide
status loops back to NTO using the kubelet credential to write an annotation
to the Node resource.

This change also simplifies the mechanism for accepting kernel
command-line parameters as calculated by the NTO operands. Now, all NTO
operands must agree on the calculated kernel command-line parameters.

ClusterOperator/node-tuning now also reports operand version. The
operand version changes only when all operand replicas have successfully
upgraded and are ready. This information is used to block PPC
Reconcilation loop until the operator and operand RELEASE_VERSION agree.
This is a short-term measure to prevent from multiple node reboots
during upgrades.

Other fixes:

• Disallow MC updates during upgrades when kernel command-line
  parameters of nodes within a MCP do not match.
• ClusterOperator/node-tuning object was sometimes giving false
  information based on an old Metadata.Generation.

Resolves: OCPBUGS-17941

This is a backport of #775

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jmencak]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade 10

[openshift-ci]

@jmencak: trigger 1 job(s) for the /payload-(job|aggregate) command

• periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5df55560-4b20-11ee-8463-642b1079c525-0

[jmencak]

/retest

[openshift-ci]

@jmencak: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[jmencak]

/jira refresh
/retest

[openshift-ci-robot]

@jmencak: This pull request references Jira Issue OCPBUGS-17941, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.13.z) matches configured target version for branch (4.13.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-17585 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE))
• dependent Jira Issue OCPBUGS-17585 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
• bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request.

In response to this:

/jira refresh
/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jmencak]

@dagrayvid , when you have a minute, would you be able to review this backport of: #775 ? Thank you.

[dagrayvid]

Sorry for the delay!

/lgtm

[jmencak]

Thank you David.
/label backport-risk-assessed
@mrniranjan in Liquan's absence, would you be able to provide cherry-pick-approved label, please?

[mrniranjan]

/label cherry-pick-approved

[jmencak]

/hold cancel

[openshift-ci-robot]

@jmencak: Jira Issue OCPBUGS-17941: All pull requests linked via external trackers have merged:

• openshift/cluster-node-tuning-operator#765
• openshift/cluster-node-tuning-operator#766

Jira Issue OCPBUGS-17941 has been moved to the MODIFIED state.

In response to this:

NTO operands allow updating Tuned Profiles. This is intentional and by design
as some host information needs to be communicated back to the NTO operator,
but this also allows a successful local attacker potentially affect node-level
configuration of other cluster nodes.

This change addresses the situation in two ways. First, scoped RBAC
permissions on Profile.status subresource is used to disallow Node-level
write access to Profile.spec. Second, the Node resource is used to provide
status loops back to NTO using the kubelet credential to write an annotation
to the Node resource.

This change also simplifies the mechanism for accepting kernel
command-line parameters as calculated by the NTO operands. Now, all NTO
operands must agree on the calculated kernel command-line parameters.

ClusterOperator/node-tuning now also reports operand version. The
operand version changes only when all operand replicas have successfully
upgraded and are ready. This information is used to block PPC
Reconcilation loop until the operator and operand RELEASE_VERSION agree.
This is a short-term measure to prevent from multiple node reboots
during upgrades.

Other fixes:

• Disallow MC updates during upgrades when kernel command-line
  parameters of nodes within a MCP do not match.
• ClusterOperator/node-tuning object was sometimes giving false
  information based on an old Metadata.Generation.

Resolves: OCPBUGS-17941

This is a backport of #775

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.