Limit node access to API #279
Comments
saad-ali
added
sig/api-machinery
saad-ali
added
sig/auth
idvoretskyi
added
the
stage/alpha
Automatic merge from submit-queue (batch tested with PRs 44337, 45775, 45832, 45574, 45758) Tighten validation of mirror pod annotations Tightens validation for pods with a mirror pod annotation: 1. spec.nodeName must be set 2. makes the mirror pod annotation immutable 3. starts validating pod-specific annotations during pod status update None of these changes affect usage of the mirror pod annotation by kubelets, which only set it on pod creation (verified this is true back to 1.5.x) the second commit updates the pod validation tests to look for specific error messages (best reviewed ignoring whitespace changes) This is the validation portion of https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-authorizer.md and kubernetes/enhancements#279 ```release-note Mirror pods must now indicate the nodeName they are bound to on creation. The mirror pod annotation is now treated as immutable and cannot be added to an existing pod, removed from a pod, or modified. ```
Automatic merge from submit-queue Node authorizer This PR implements the authorization portion of https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-authorizer.md and kubernetes/enhancements#279: * Adds a new authorization mode (`Node`) that authorizes requests from nodes based on a graph of related pods,secrets,configmaps,pvcs, and pvs: * Watches pods, adds edges (secret -> pod, configmap -> pod, pvc -> pod, pod -> node) * Watches pvs, adds edges (secret -> pv, pv -> pvc) * When both Node and RBAC authorization modes are enabled, the default RBAC binding that grants the `system:node` role to the `system:nodes` group is not automatically created. * Tightens the `NodeRestriction` admission plugin to require identifiable nodes for requests from users in the `system:nodes` group. This authorization mode is intended to be used in combination with the `NodeRestriction` admission plugin, which limits the pods and nodes a node may modify. To enable in combination with RBAC authorization and the NodeRestriction admission plugin: * start the API server with `--authorization-mode=Node,RBAC --admission-control=...,NodeRestriction,...` * start kubelets with TLS boostrapping or with client credentials that place them in the `system:nodes` group with a username of `system:node:<nodeName>` ```release-note kube-apiserver: a new authorization mode (`--authorization-mode=Node`) authorizes nodes to access secrets, configmaps, persistent volume claims and persistent volumes related to their pods. * Nodes must use client credentials that place them in the `system:nodes` group with a username of `system:node:<nodeName>` in order to be authorized by the node authorizer (the credentials obtained by the kubelet via TLS bootstrapping satisfy these requirements) * When used in combination with the `RBAC` authorization mode (`--authorization-mode=Node,RBAC`), the `system:node` role is no longer automatically granted to the `system:nodes` group. ``` ```release-note RBAC: the automatic binding of the `system:node` role to the `system:nodes` group is deprecated and will not be created in future releases. It is recommended that nodes be authorized using the new `Node` authorization mode instead. Installations that wish to continue giving all members of the `system:nodes` group the `system:node` role (which grants broad read access, including all secrets and configmaps) must create an installation-specific ClusterRoleBinding. ``` Follow-up: - [ ] enable e2e CI environment with admission and authorizer enabled (blocked by kubelet TLS bootstrapping enablement in #40760) - [ ] optionally enable this authorizer and admission plugin in kubeadm - [ ] optionally enable this authorizer and admission plugin in kube-up
liggitt
added
stage/beta
liggitt
changed the title
Automatic merge from submit-queue (batch tested with PRs 46441, 43987, 46921, 46823, 47276) Enable Node authorizer and NodeRestriction admission in kubemark xref kubernetes/enhancements#279 We want to ensure scale testing covers use of the authorizer/admission pair that partitions nodes. This includes enabling the authorizer, which populates a graph of existing nodes and pods. Kubemark is still running all nodes with a single credential, so a follow-up step is to generate unique credentials per node (or enable TLS bootstrapping) and remove the temporary rolebinding added in this PR so the node authorizer is the one authorizing each call by a hollow node.
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
1.19-targeted work: kubernetes/kubernetes#90307 |
|
1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19? In order to have this part of the release: The KEP PR must be merged in an implementable state The current release schedule is: Monday, April 13: Week 1 - Release cycle begins Please let me know and I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. Thanks! |
|
Yes, this is completed in 1.19. Relevant PRs: |
|
@liggitt just to clarify will this be GA in 1.19? |
kikisdeliveryservice
added
tracked/yes
|
/milestone v1.19 |
|
This didn't follow alpha, beta, GA cycle, but basically it's GA now. |
kikisdeliveryservice
added
the
stage/stable
|
I notice that the underlying KEP does not have a Test Plan, can you please update the KEP to have one by the Enhancements Freeze (Tuesday May 19th EOD PST) as this is required for inclusion in 1.19 release. As an additional note, #1620 merged recently, adding production readiness review questions to the KEP template. We are not making it mandatory for the 1.19 release cycle, but it would be great if the PRR questionnaire is filled since the KEP needs to be updated to include the test plan. You can find the template at https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template If you have any questions please let me know. Thanks! |
|
As this predates the current KEP format and the linked issues do have tests, we will be marking this as tracked. |
|
There is partial e2e coverage for this feature, but the tests are currently broken and not run by any of our test jobs, and don't cover the label or taint restrictions. See kubernetes/kubernetes#82551. |
|
The unit and integration tests exercise all the API restrictions implemented by the feature. This feature does not lend itself to e2e test environments which do not control the kube-apiserver configuration (so cannot guarantee the feature is in use) and cannot easily create false Node objects in cloud environments where the node controller detects and deletes the dummy Node object. |
|
Do you think we should just delete the e2e tests? |
|
It's unclear to me what the goal of the e2e tests are currently. To verify functionality, integration tests seem much better suited to me. Are they trying to validate that the feature is enabled? |
|
Hi @liggitt Friendly reminder that if new/modification to docs are required, a placeholder PR against k/website (branch |
|
No docs updates are required, https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction already describes the user-facing aspects of the feature. |
|
Thank you! I will update the tracking sheet accordingly |
|
Hi @liggitt ! To follow-up on the email sent to k-dev today, I wanted to let you know that Code Freeze has been extended to Thursday, July 9th. You can see the revised schedule here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.19 We expect all PRs to be merged by that time. Please let me know if you have any questions. Best, |
mikedanese
removed
the
stage/beta
|
This is done. |
kikisdeliveryservice
removed
the
tracked/yes
Limit node access to API
Nodeauthorization mode andNodeRestrictionadmission plugin, when used in combination, limit nodes' access to specific APIs, so that they may only modify their own Node API object, only modify Pod objects bound to themselves, and only retrieve secrets and configmaps referenced by pods bound to themselves.The text was updated successfully, but these errors were encountered: