Node Authorization Admission Controller

[miwithro]

Enabled by default on AKS 1.24 +

https://kubernetes.io/docs/reference/access-authn-authz/node/

@Azure/aks-pm issue needs labels

[raesene]

Hi,

Just to check, should this be enabled on current AKS 1.24 clusters? I spun up a test cluster today using 1.24.6 but it seems like the kubelet credentials can still be used to get a variety of information from the API server, which I would expect to fail in a cluster using node authorization.

 kubectl --kubeconfig=/var/lib/kubelet/kubeconfig get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-32616802-vmss000000   Ready    agent   12m   v1.24.6

As an example get secrets -A works

kubectl --kubeconfig=/var/lib/kubelet/kubeconfig get secrets -A
NAMESPACE     NAME                     TYPE                            DATA   AGE
kube-system   ama-logs-secret          Opaque                          2      13m
kube-system   bootstrap-token-77ss05   bootstrap.kubernetes.io/token   4      13m
kube-system   konnectivity-certs       Opaque                          3      13m

[raesene]

Just an update on this, it still seems that Node authorization isn't enabled? looking at the output of auth can-i --list there's a lot of RBAC permissions assigned.

 kubectl --kubeconfig=/var/lib/kubelet/kubeconfig auth can-i --list
warning: the list may be incomplete: node authorizer does not support user rule resolution
Resources                                                       Non-Resource URLs   Resource Names   Verbs
leases.coordination.k8s.io                                      []                  []               [create delete get patch update]
csinodes.storage.k8s.io                                         []                  []               [create delete get patch update]
nodes                                                           []                  []               [create get list watch patch update]
certificatesigningrequests.certificates.k8s.io                  []                  []               [create get list watch]
events                                                          []                  []               [create patch update]
events.events.k8s.io                                            []                  []               [create patch update]
pods/eviction                                                   []                  []               [create]
serviceaccounts/token                                           []                  []               [create]
tokenreviews.authentication.k8s.io                              []                  []               [create]
localsubjectaccessreviews.authorization.k8s.io                  []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io                   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io                    []                  []               [create]
subjectaccessreviews.authorization.k8s.io                       []                  []               [create]
certificatesigningrequests.certificates.k8s.io/selfnodeclient   []                  []               [create]
pods                                                            []                  []               [get list watch create delete]
configmaps                                                      []                  []               [get list watch]
endpoints                                                       []                  []               [get list watch]
secrets                                                         []                  []               [get list watch]
services                                                        []                  []               [get list watch]
runtimeclasses.node.k8s.io                                      []                  []               [get list watch]
csidrivers.storage.k8s.io                                       []                  []               [get list watch]
persistentvolumeclaims/status                                   []                  []               [get patch update]
                                                                [/api/*]            []               [get]
                                                                [/api]              []               [get]
                                                                [/apis/*]           []               [get]
                                                                [/apis]             []               [get]
                                                                [/healthz]          []               [get]
                                                                [/healthz]          []               [get]
                                                                [/livez]            []               [get]
                                                                [/livez]            []               [get]
                                                                [/openapi/*]        []               [get]
                                                                [/openapi]          []               [get]
                                                                [/readyz]           []               [get]
                                                                [/readyz]           []               [get]
                                                                [/version/]         []               [get]
                                                                [/version/]         []               [get]
                                                                [/version]          []               [get]
                                                                [/version]          []               [get]
persistentvolumeclaims                                          []                  []               [get]
persistentvolumes                                               []                  []               [get]
volumeattachments.storage.k8s.io                                []                  []               [get]
endpointslices.discovery.k8s.io                                 []                  []               [list watch]
nodes/status                                                    []                  []               [patch update]
pods/status                                                     []                  []               [patch update]

[raesene]

For anyone else finding this issue, I made some notes as it's not as straight-forward as it might seem (I'm working black-box here, so if anyone from AKS wants to clarify/provide a better answer please feel free :P )

I think that the NodeAuthorization plugin actually is enabled, it's just that additional rights have been granted via RBAC and these rights are cumulative.

The output in the comment above shows the RBAC rights that are assigned to the node (Node authorization rights do not appear in the output of kubectl auth can-i --list as mentioned in the output of the command)

Where this has an impact is, for example, kubectl get secrets -A which is not a right available under node authorization but is a right available via RBAC (per the comment two up). In a base cluster this is less dangerous where service account tokens are no longer secrets, but in a production cluster it's likely this would be dangerous as secrets not assigned to a node could be accessed by someone compromising the node.

Other rights that could be dangerous (e.g. Create pods or create tokens) are mitigated by the NodeAuthorzation mode and NodeRestriction Admission Controller.

[miwithro]

@raesene we actually fixed the issues you noted above. Can you please test again and validate our fixes from your end?

[microsoft-github-policy-service]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.

@Azure/aks-pm issue needs labels

[raesene]

Hi,

Just to check, should this be enabled on current AKS 1.24 clusters? I spun up a test cluster today using 1.24.6 but it seems like the kubelet credentials can still be used to get a variety of information from the API server, which I would expect to fail in a cluster using node authorization.

 kubectl --kubeconfig=/var/lib/kubelet/kubeconfig get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-32616802-vmss000000   Ready    agent   12m   v1.24.6

As an example get secrets -A works

kubectl --kubeconfig=/var/lib/kubelet/kubeconfig get secrets -A
NAMESPACE     NAME                     TYPE                            DATA   AGE
kube-system   ama-logs-secret          Opaque                          2      13m
kube-system   bootstrap-token-77ss05   bootstrap.kubernetes.io/token   4      13m
kube-system   konnectivity-certs       Opaque                          3      13m

[raesene]

Just an update on this, it still seems that Node authorization isn't enabled? looking at the output of auth can-i --list there's a lot of RBAC permissions assigned.

 kubectl --kubeconfig=/var/lib/kubelet/kubeconfig auth can-i --list
warning: the list may be incomplete: node authorizer does not support user rule resolution
Resources                                                       Non-Resource URLs   Resource Names   Verbs
leases.coordination.k8s.io                                      []                  []               [create delete get patch update]
csinodes.storage.k8s.io                                         []                  []               [create delete get patch update]
nodes                                                           []                  []               [create get list watch patch update]
certificatesigningrequests.certificates.k8s.io                  []                  []               [create get list watch]
events                                                          []                  []               [create patch update]
events.events.k8s.io                                            []                  []               [create patch update]
pods/eviction                                                   []                  []               [create]
serviceaccounts/token                                           []                  []               [create]
tokenreviews.authentication.k8s.io                              []                  []               [create]
localsubjectaccessreviews.authorization.k8s.io                  []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io                   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io                    []                  []               [create]
subjectaccessreviews.authorization.k8s.io                       []                  []               [create]
certificatesigningrequests.certificates.k8s.io/selfnodeclient   []                  []               [create]
pods                                                            []                  []               [get list watch create delete]
configmaps                                                      []                  []               [get list watch]
endpoints                                                       []                  []               [get list watch]
secrets                                                         []                  []               [get list watch]
services                                                        []                  []               [get list watch]
runtimeclasses.node.k8s.io                                      []                  []               [get list watch]
csidrivers.storage.k8s.io                                       []                  []               [get list watch]
persistentvolumeclaims/status                                   []                  []               [get patch update]
                                                                [/api/*]            []               [get]
                                                                [/api]              []               [get]
                                                                [/apis/*]           []               [get]
                                                                [/apis]             []               [get]
                                                                [/healthz]          []               [get]
                                                                [/healthz]          []               [get]
                                                                [/livez]            []               [get]
                                                                [/livez]            []               [get]
                                                                [/openapi/*]        []               [get]
                                                                [/openapi]          []               [get]
                                                                [/readyz]           []               [get]
                                                                [/readyz]           []               [get]
                                                                [/version/]         []               [get]
                                                                [/version/]         []               [get]
                                                                [/version]          []               [get]
                                                                [/version]          []               [get]
persistentvolumeclaims                                          []                  []               [get]
persistentvolumes                                               []                  []               [get]
volumeattachments.storage.k8s.io                                []                  []               [get]
endpointslices.discovery.k8s.io                                 []                  []               [list watch]
nodes/status                                                    []                  []               [patch update]
pods/status                                                     []                  []               [patch update]

[raesene]

For anyone else finding this issue, I made some notes as it's not as straight-forward as it might seem (I'm working black-box here, so if anyone from AKS wants to clarify/provide a better answer please feel free :P )

I think that the NodeAuthorization plugin actually is enabled, it's just that additional rights have been granted via RBAC and these rights are cumulative.

The output in the comment above shows the RBAC rights that are assigned to the node (Node authorization rights do not appear in the output of kubectl auth can-i --list as mentioned in the output of the command)

Where this has an impact is, for example, kubectl get secrets -A which is not a right available under node authorization but is a right available via RBAC (per the comment two up). In a base cluster this is less dangerous where service account tokens are no longer secrets, but in a production cluster it's likely this would be dangerous as secrets not assigned to a node could be accessed by someone compromising the node.

Other rights that could be dangerous (e.g. Create pods or create tokens) are mitigated by the NodeAuthorzation mode and NodeRestriction Admission Controller.

[miwithro]

@raesene we actually fixed the issues you noted above. Can you please test again and validate our fixes from your end?

[microsoft-github-policy-service]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.