Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A potential risk of kubevirt makes a worker node get the cluster's admin secret. #9109

Open
younaman opened this issue Jan 27, 2023 · 30 comments
Labels

Comments

@younaman
Copy link

Description:

  • There are three components of kubevirt with default installation: kubevirt-handler, kubevirt-controller, kubevirt-api, and kubevirt-operator, the kubevirt-handler is a daemon set running on each node, the controller&api&operator are deployments, each of them is two-pod deployments running on worker nodes randomly.
  • There are different cluster roles of these components:
    • The cluster role of the kubevirt-handler can patch nodes, which can be leveraged to make nodes unschedulable.
    • The cluster role of the kubevirt-controller can create pods/eviction, which can be leveraged to make pods evicted. While the cluster role of kubevirt-api can delete pods, which can be leveraged to make pods recreated and rescheduled to worker nodes.
    • The cluster role of the kubevirt-operator can list all secrets, which can be leveraged to get the admin secret of the entire cluster and make cluster-level privilege escalation.
  • If a malicious user controls a worker node, which has kubevirt-handler, and one of kubevirt-api/controller:
    1. He/she can leverage the kubevirt-handler to path all other nodes with unschedulable: true". And when the kubevirt-operator is rescheduled, the kubevirt-operator will be rescheduled to the malicious worker node.
    2. He/She can use the token of kubevirt-api/controller to evict/delete kubevirt-operator pods, which will be recreated and run on the malicious worker node.
    3. After the kubevirt-operator runs on the malicious worker node, he/she can leverage the operator's token to list all secrets in the whole cluster, get the cluster admin's token and make a cluster-level privilege escalation.
  • Mitigation Discussion:
    • The cluster role of the kubevirt-operator can use the Resource Names to restrain the secrets which can be listed by the operator itself, however, it may need a careful review of the kubevirt-operator to identify all secrets needed by the operator exactly.
    • The kubevirt-handler can remove the patch verbs of nodes, and the controller/api can remove the evict and delete of pods. However, similar to the first mitigation, it still needs a careful revisit of the components' source code.
    • The kubevirt can let the kubevirt-operator/controller/api runs on worker nodes (which we called "controller worker node") without running users' workloads, and leverage the isolation of nodes to mitigate the risk. However, if a malicious user leverages some method(i.e., scale-up of kubevirt-handler) and moves to the "controller worker node" horizontally, it still can break the worker nodes' isolation and make privilege escalation.
  • Installation and environment:
    • We follow the Kubevirt's official guide to install the Kubevirt on a four-node Kubernetes cluster(1.24.2).
      I'm looking forward to hearing back from you!
@younaman
Copy link
Author

Besides, I have written an email following the private security policy of the kubevirt vulnerability report. However, I tried my school email and it's not inside the mailing white list :(. I am sorry to write a "security issue" by opening a public-accessible GitHub issue.

@rmohr
Copy link
Member

rmohr commented Jan 27, 2023

If a malicious user controls a worker node, which has kubevirt-handler, and one of kubevirt-api/controller:

  1. He/she can leverage the kubevirt-handler to path all other nodes with unschedulable: true". And when the kubevirt-operator is rescheduled, the kubevirt-operator will be rescheduled to the malicious worker node.
  2. He/She can use the token of kubevirt-api/controller to evict/delete kubevirt-operator pods, which will be recreated and run on the malicious worker node.
  3. After the kubevirt-operator runs on the malicious worker node, he/she can leverage the operator's token to list all secrets in the whole cluster, get the cluster admin's token and make a cluster-level privilege escalation.

That is a nice abuse flow. Thanks for writing it up!

@rmohr
Copy link
Member

rmohr commented Jan 27, 2023

Besides, I have written an email following the private security policy of the kubevirt vulnerability report. However, I tried my school email and it's not inside the mailing white list :(. I am sorry to write a "security issue" by opening a public-accessible GitHub issue.

Sent it in your name to the list.

@younaman
Copy link
Author

@rmohr Thanks for your reply! My school e-mail is nzyang@stu.xidian.edu.cn :) So can you add my e-mail to the white list? After that, should I discuss this potential risk by private email and close this opening issue?

@younaman
Copy link
Author

@rmohr I have tried to send the email by using nzyang@stu.xidian.edu.cn and my GitHub account's primary email 952508578@qq.com. However, they both told me that my email is bouncing back :( How can I connect to you by sending an email?

@rmohr
Copy link
Member

rmohr commented Jan 30, 2023

@rmohr I have tried to send the email by using nzyang@stu.xidian.edu.cn and my GitHub account's primary email 952508578@qq.com. However, they both told me that my email is bouncing back :( How can I connect to you by sending an email?

@aburdenthehand is this something which you can do?

@aburdenthehand
Copy link
Contributor

@rmohr I'm looking into it.
While that's happening, I can cc nzyang@stu.xidian.edu.cn onto the thread, which should work for this issue.

@younaman
Copy link
Author

younaman commented Feb 3, 2023

@rmohr @aburdenthehand Thanks for your reply! I am a little busy these days because revising my paper:)
So, It means that I can send mail right now? I will try it after I finish my tasks on hand, eta 3 days.

@aburdenthehand
Copy link
Contributor

@younaman Sort of. We are still investigating the problem.
I added you to the thread using your nzyang@stu.xidian.edu.cn address. You should be able to reply to that thread now.
I also sent you two other emails to the same address asking for a little bit of information to help us troubleshoot why you were being blocked. If you could please follow up with them, it would be most appreciated. Thank you, and good luck with your paper!

@younaman
Copy link
Author

younaman commented Feb 3, 2023

@aburdenthehand Thaks a lot! I checked my nzyang@stu.xidian.edu.cn just now, and I have received three emails you sent:) I will reply your mail after I finished my paper, thank you again for your patience:) Have a nice day!

@younaman
Copy link
Author

younaman commented Feb 5, 2023

@rmohr @aburdenthehand I can send mail to your post box, however, the cncf mailing list (cncf-kubevirt-security@lists.cncf.io) still rejects my mail, my post box says:"Warm tips: Please contact the recipient's administrator to adjust the anti-spam rule ,or ask the recipient to add your email address to the whitelist."

Anyway, it seems like you(aburden@redhat.com) and robert.kielty(robert.kielty@cncf.io) have received my mail, can you answer the questions which I wrote in my mail? Looking forward to your reply!

@younaman
Copy link
Author

younaman commented Feb 5, 2023

@rmohr @aburdenthehand I write my questions in the following, and looks for more comments:
Is it a real problem in kubevirt? If the problem is real, can any of my mitigations (listed in the github issue page) works to solve this problem? Does kubevirt have any plan to fix this issue? (It looks like there exists a PR to mitigate this risk,#9162)
By the way, if it's a real issue and has been fixed, can I get a CVE-number:)
Reporter List:
Nanzi Yang(nzyang@stu.xidian.edu.cn/952508578@qq.com, github account is younaman, me)
Xin Guo(guox@stu.xidian.edu.cn)
Jietao Xiao(jietaoXiao@stu.xidian.edu.cn)
Wenbo Shen(shenwenbo@zju.edu.cn)
Jinku Li(jkli@xidian.edu.cn)
Looking forward to your reply!

@younaman
Copy link
Author

younaman commented Feb 6, 2023

@rmohr @aburdenthehand Knock knock! Are there any updates about my "possible CVE number"? Besides, the PR looks like following my first recommendations to mitigate the risks? Looking forward to your reply!

@rmohr
Copy link
Member

rmohr commented Feb 7, 2023

@rmohr @aburdenthehand I write my questions in the following, and looks for more comments:
Is it a real problem in kubevirt? If the problem is real, can any of my mitigations (listed in the github issue page) works to solve this problem? Does kubevirt have any plan to fix this issue? (It looks like there exists a PR to mitigate this risk,#9162)

Yes that is because of your initiative. Just note that virt-operator only has a Role (not a ClusterRole). As such it could only read secrets in the kubevirt install namespace and not from the whole cluster. But thanks to your initiative it will now be limited even more :)

Still, the core issue remains and is not addressed yet: virt-handler can modify pod specs. It is a serious issue but luckily not critical, since a node needs at least to get compromised first.

By the way, if it's a real issue and has been fixed, can I get a CVE-number:)

It is definitely a problem that daemonsets which can modify pod specs can "lure-in" specific deployments with extended privileges if a node gets hacked. We are discussing mitigations and fixes at the moment.

Reporter List:
Nanzi Yang(nzyang@stu.xidian.edu.cn/952508578@qq.com, github account is younaman, me)
Xin Guo(guox@stu.xidian.edu.cn)
Jietao Xiao(jietaoXiao@stu.xidian.edu.cn)
Wenbo Shen(shenwenbo@zju.edu.cn)
Jinku Li(jkli@xidian.edu.cn)
Looking forward to your reply!

I am currently checking regarding a CVE.
We will make sure that you get mentioned as the discoverer. :)

@younaman
Copy link
Author

younaman commented Feb 7, 2023

@rmohr Thanks for your reply! Perhaps it's because I ignore the Role and regard it as a ClusterRole. Anyway, it does not downgrade the key point of my report.

Besides, for the critical problem, notice the node escape may be easier via other co-host third-party apps:). For example, I have found and reported another issue to the cilium community, "The potential node escape issue" has been confirmed but the fix has not been discussed and opened yet:(

By the way, thank you again for the CVE number, it's an honor and appreciate for your patience and works:)

@younaman
Copy link
Author

younaman commented Feb 7, 2023

@rmohr As I still rejected by the mailing list, can you or @aburdenthehand CC to me a mail when there are any updates about my issue:) Both of you have my email address.

@RobertKielty
Copy link

@younaman I've just sent you an invite to join the mailing list, check your email inbox.

@younaman
Copy link
Author

younaman commented Feb 7, 2023

@RobertKielty Thanks a lot! I have joined the mailing list, and you should have received my email about "Are there any updates about mitigations?" Have a nice day:)

@younaman
Copy link
Author

younaman commented Feb 20, 2023

@rmohr For the cilium node escalation I mentioned in comment, it has been confirmed and fixed by cilium in cilium version 1.13, so I can report it to you now:) The cilium DaemonSet pod mounts the host's /proc/sys directory with read&write, so if a malicious user can access the cilium pod, he/she can:

  • Create a malicious reverse script in the cilium pod.
  • Modify the /proc/sys/kernel/core_pattern, and point it to the malicious shell.
  • Trigger a core dump, and the core_pattern will make the host kernel execute the malicious script, and reverse a shell with the host root user to the malicious user. Thus, he/she can leverage the cilium pod to escape to the worker node, and leverage the kubevirt to make the privilege escalation that I reported here.
    I know it's not the problem of kubevirt. I just want you to know this issue and make a further consideration about "multiple third-party apps" scenario:) If you want the details, you have my email and please feel free to discuss more details with me.

@younaman
Copy link
Author

@rmohr Knock knock! Are there any updates? Looking forward to your reply:)

@rmohr
Copy link
Member

rmohr commented Feb 22, 2023

So, sorry for the delay. Took me a little bit to check things, since I am myself not that familiar with filing CVEs. I think the best way would be if you just file the CVE yourself, like described here: https://infosecwriteups.com/how-to-register-and-publish-a-cve-for-your-awesome-vulnerability-e68a6a5f748f. You can then hand us over the CVE number which you get (I would file a low-to-med severity CVE).

We would then publish a security note on our project referencing your CVE together with mitigations, which will serve as the public reference for your CVE to make it public as well.

I think that would be the best way to have you as an owner of this.

@younaman
Copy link
Author

@rmohr Thanks for your reply:) However, I think it's better for you to register a CVE number because you stand for the kubevirt community. If I register the CVE myself, I am afraid that the cveform.mitre.org will NOT give me a CVE number because of some other reasons. (I believe you know what I mean.)

@younaman
Copy link
Author

younaman commented Feb 22, 2023

@rmohr I believe you can register the CVE, and I will do my best to assist you if needed:) More specifically, I have not registered a CVE before. I am very worried about the cveform.miter.org will not give me the CVE number because of some "misbehavior" of mine.

@younaman
Copy link
Author

younaman commented Feb 23, 2023

@rmohr I have tried to request a CVE number following the https://infosecwriteups.com/how-to-register-and-publish-a-cve-for-your-awesome-vulnerability-e68a6a5f748f, and I fill the CVE web page following the pdf file which I sent to your email(rmohr@google.com). Please feel free to contact me if you want some changes:) And please let me know you you received the mail.

By the way, my CVE ID request is "CVE Request 1411295", I don't know if you need this information, so I just write it.

@younaman
Copy link
Author

younaman commented Feb 24, 2023

@rmohr I tried to send you and kubevirt maintainers another mail based on your valuable comments, however, it seems that I am banned by your mail, and my e-mail has been rejected. Your e-mail says "(Proxy)Host google.com said 450 Broken pipe(CONNECT). Bounce by SDN Rule 14 (tag:8)", and the kubevirt maintainer list's e-mail says "SMTP through SDN 4, SMTP: (Proxy)Host lists.cncf.io said 500 Invalid request, no reverse DNS for 121.9.212.76 (tag:-1)" Have I do something wrong? Looking forward to your reply!
@RobertKielty Can you help me to fix this problem? Or do I have to use my Gmail? :( I just want to share new findings based on Roman's valuable comments:( I am sorry if it "breaks" some rules of the community.

@fabiand
Copy link
Member

fabiand commented Apr 12, 2023

Hey.

Does #9162 fix this issue completely?

@rmohr
Copy link
Member

rmohr commented Apr 12, 2023 via email

@rmohr
Copy link
Member

rmohr commented Apr 12, 2023

/reopen

@kubevirt-bot kubevirt-bot reopened this Apr 12, 2023
@kubevirt-bot
Copy link
Contributor

@rmohr: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@younaman
Copy link
Author

@rmohr @fabiand Perhaps we can use the kubelet's configuration, and make the virt-handler can only "modify" the node which it runs? As the kubelet has NodeRestrictions by default, so it may needs only a few changes to solve this problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants