A potential risk in volcano that could lead to takeover of the cluster

[HouqiyuA]

Dear Team Members:

Greetings! Our team is very interested in your project and we recently identified a potential RBAC security risk while doing a security assessment of your project. Therefore, we would like to report it to you and provide you with the relevant details so that you can fix and improve it accordingly.

Details:

In this Kubernetes project, there exists a ClusterRole that has been granted list secrets high-risk permissions. These permissions allow the role to list confidential information across the cluster. An attacker could impersonate the ServiceAccount bound to this ClusterRole and use its high-risk permissions to list secrets information across the cluster. By combining the permissions of other roles, an attacker can elevate the privileges and further take over the entire cluster.

We construct the following attack vectors to address the above risks:

First, you need to get the Token for the ServiceAccount that has this high-risk privilege. if you are already in a Pod with this over-privileged privilege, you can run the following command directly to get the Token: cat /var/run/secrets/kubernetes.io/serviceaccount/ token. If you are on the node and not in the Pod, you can run the following command to get the kubectl describe secret .

Use the obtained Token information to authenticate with the API Server. By including the Token in the request, you can be recognized as a legitimate user with the ServiceAccount and gain all privileges associated with the ServiceAccount. This ServiceAccount identity can therefore be used to list all the Secrets in the cluster.

We give two ways to further utilize ServiceAccount Token with other permissions to take over the cluster:

Method 1: Elevation of Privileges using ServiceAccount Token bound to ClusterAdmin.

Use a Token with the privileges of the ClusterAdmin role that has the authority to control the entire cluster. By authenticating with this Token, you can gain full control of the cluster.

Method 2: Create privileged containers using the ServiceAccount Token with the create pods permission. If you have a ServiceAccount with create pods privileges in your k8s cluster, you can use the Token in the Secrets of this ServiceAccount to create a privileged container that mounts the root directory. This container is excluded from the master node by setting a taint tolerance, which leaks the master's kubeconfig configuration file. In this way, the attacker can take over the entire cluster.

For the above attack chain we have developed exploit code and uploaded it to github: https://github.com/HouqiyuA/k8s-rbac-poc

Mitigation methods are explored:

Carefully evaluate the permissions required for each user or service account to ensure that it is following the principle of least privilege and to avoid over-authorization.

If list secrets is a required permission, consider using more granular RBAC rules. Role Binding can be used to grant list secrets permissions instead of ClusterRole, which restricts permissions to specific namespaces or resources rather than the entire cluster.

Isolate different applications into different namespaces and use namespace-level RBAC rules to restrict access. This reduces the risk of privilege leakage across namespaces

Looking forward to hearing from you and discussing this risk in more detail with us, thank you very much for your time and attention.

Best wishes.

HouqiyuA

[Monokaix]

Glad to receive your feedback! Volcano uses a standard serviceAccount & RBAC rule to grant pod permissions, and seems that k8s has some measures to avoid serviceaccount permission issue like https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection, is this can satisfy your need?

[HouqiyuA]

I found a ClusterRole named "Volcanic-controllers" found in the volcano project code, which has the list secrets permission. Is granting clusterRole list secrets permission necessary in this project? Marty Mcfly ***@***.***> 于2024年4月29日周一 10:43写道：

…

Glad to receive your feedback! Volcano uses a standard serviceAccount & RBAC rule to grant pod permissions, and seems that k8s has some measures to avoid serviceaccount permission issue like https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection, is this can satisfy your need? — Reply to this email directly, view it on GitHub <#3446 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBBY3LTWUNUPSTSLULLRGO3Y7WXTNAVCNFSM6AAAAABG4Z6BTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRG44TQNRTHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Monokaix]

I found a ClusterRole named "Volcanic-controllers" found in the volcano project code, which has the list secrets permission. Is granting clusterRole list secrets permission necessary in this project? Marty Mcfly @.> 于2024年4月29日周一 10:43写道：
…
Glad to receive your feedback! Volcano uses a standard serviceAccount & RBAC rule to grant pod permissions, and seems that k8s has some measures to avoid serviceaccount permission issue like https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection, is this can satisfy your need? — Reply to this email directly, view it on GitHub <#3446 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBBY3LTWUNUPSTSLULLRGO3Y7WXTNAVCNFSM6AAAAABG4Z6BTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRG44TQNRTHE . You are receiving this because you authored the thread.Message ID: @.>

Seems that volcano controller only used GET and CREATE verbs currently: )

[Monokaix]

Maybe we can remove list permission of volcano controller.

[Monokaix]

/good-first-issue

[volcano-sh-bot]

@Monokaix:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[HouqiyuA]

Yes! Can volcano team reward our team after solving this problem? For example, apply for a CVE ID to our team for this problem or other ways? Marty Mcfly ***@***.***> 于2024年4月29日周一 14:56写道：

…

Maybe we can remove list permission of volcano controller. — Reply to this email directly, view it on GitHub <#3446 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBBY3LXG3JAEHAVQCFYBCN3Y7XVITAVCNFSM6AAAAABG4Z6BTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBSGAYDKNZTGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Monokaix]

fixed by #3449

[Monokaix]

/close

[volcano-sh-bot]

@Monokaix: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.