[WIP] Adding support for encrypted container images

[harche]

What type of PR is this?

/kind feature

What this PR does / why we need it:
This PR adds support for upcoming encryption support for container images

Which issue(s) this PR fixes:

Fixes kubernetes/enhancements#1067

Special notes for your reviewer:
This is still a work-in-progress. We are on the verge of getting OCI spec followed by the support in containerd for encrypted images merged.

Does this PR introduce a user-facing change?: yes

Add support to encrypted images

[k8s-ci-robot]

Welcome @harche!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Hi @harche. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harche]

"I signed it"

[harche]

/assign @derekwaynecarr @liggitt

[harche]

/joke

[dims]

@harche Which version of containerd is likely to have support for this feature? Is there a plan to help other runtimes get there? (especially support via docker API)

[harche]

@harche Which version of containerd is likely to have support for this feature? Is there a plan to help other runtimes get there? (especially support via docker API)

CRI->containerd is yet to support the decryption in their upstream. Although I have a working POC of CRI-Containerd as well as CRI-O, they both won't be able to merge my changes because they vendor k8s APIs which need modification for this to work. So as soon as we have the code in k8s merged, I can submit the PRs for CRI-Containerd as well CRI-O to support this feature by updating their respective vendor directories from k8s that handle CRI interface.

Once the CRI interface in k8s gets updated to pass decryption keys to runtimes, adding support in the runtimes should fairly simple, including docker.

For your reference this is how it's supposed to work, I gave the example of CRI-O in that diagram but you can very well replace with any runtime that implements CRI interface.

[dims]

Gotcha. Thanks @harche

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[harche]

/remove-lifecycle stale

[duyanghao]

@harche Great Work, but how things progress? Does kubernetes support image encryption now?
There is an article about how to use kubernetes image encryption based on cri-o, which seems not to be consistent with your KEP status right now.

[k8s-ci-robot]

@harche: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harche]

@harche Great Work, but how things progress? Does kubernetes support image encryption now?
There is an article about how to use kubernetes image encryption based on cri-o, which seems not to be consistent with your KEP status right now.

Thanks @duyanghao. The article talks about the work I did with @lumjjb on adding decryption capabilities in CRIO using the keys available on the worker node.

This code change is about adding the decryption capabilities in kubernetes by adding the new secret type to hold the required private keys. However, based on this comment, we are trying add the image encryption/decryption support in other container related projects to increase the adoption. Such as,

containers/skopeo#732
containers/skopeo#814
cri-o/cri-o#2813
containerd/cri#1340

Apart from these, we are in talks with Quay registry folks to add the support for encrypted images. Hoping to get that through soon.

[duyanghao]

@harche Great Work, but how things progress? Does kubernetes support image encryption now?
There is an article about how to use kubernetes image encryption based on cri-o, which seems not to be consistent with your KEP status right now.

Thanks @duyanghao. The article talks about the work I did with @lumjjb on adding decryption capabilities in CRIO using the keys available on the worker node.

This code change is about adding the decryption capabilities in kubernetes by adding the new secret type to hold the required private keys. However, based on this comment, we are trying add the image encryption/decryption support in other container related projects to increase the adoption. Such as,

containers/skopeo#732
containers/skopeo#814
cri-o/cri-o#2813
containerd/cri#1340

Apart from these, we are in talks with Quay registry folks to add the support for encrypted images. Hoping to get that through soon.

@harche Thanks for your reply, I still have some questions about your work:
1. Is there a practical version of your work in production environments?
As I understand it, both containerd/cri and cri-o has supported basic image encryption and decryption right now(maybe not fully as you are still developing). So I am curious about the practical version of the kubernetes, cri-o or containerd in production environments.
2. What's the current stage of your work? Alpha, Beta or Stable?
I am trying to put your work onto our production environments(kubernetes + docker), so I am worry about the security of image encryption and decryption itself. After all, this new feature seem to have not used by others yet.

[lumjjb]

Hi @duyanghao, thanks for your interest in our work! Here is some more information, but would be glad to chat offline to hear some of the usecases and plans.

1. Is there a practical version of your work in production environments?

We have an ongoing offering that will be using this technology that is going to be available in the later half of this year. We have plans to work with the IBM IKS team to get this integrated this year as well. The feature should also be available in OpenShift 4.4 when they start using k8s >=1.17.

2. What's the current stage of your work? Alpha, Beta or Stable?

This would be made first available with kubernetes 1.17 (through cri-o 1.17 and the respective containerd cri version), without any changes in kubernetes, so if I were to put a label on it, it would be alpha. But containerd and cri-o doesn't officially have any special labeling on the feature.

Also, just to clarify, there are two models of using encrypted images with kubernetes, what is do-able today with containerd/cri and cri-o is independent of this KEP. This KEP is an effort to provide the ability to have stronger multitenancy using encrypted images.

[duyanghao]

@lumjjb Thanks for your reply

Also, just to clarify, there are two models of using encrypted images with kubernetes, what is do-able today with containerd/cri and cri-o is independent of this KEP. This KEP is an effort to provide the ability to have stronger multitenancy using encrypted images.

It seems that I can now use the kubernetes(>=1.17) and docker(containerd) to achieve image encryption&decryption with the model independent of this KEP, but is there any official kubernetes documents about how to use it besides this article?

[lumjjb]

It seems that I can now use the kubernetes(>=1.17) and docker(containerd) to achieve image encryption&decryption with the model independent of this KEP, but is there any official kubernetes documents about how to use it besides this article?

For containerd, it is currently pending on this PR, which is waiting on 1 more approval.

containerd/cri#1340

The actual doc that will merge with this is: https://github.com/containerd/cri/blob/de21864c7684ddbb9c72c22769d704ed0d42cbbf/docs/encryption.md

I think @Random-Liu can probably shed a bit more light on timeline and containerd release cycles.

[harche]

@lumjjb Thanks for your reply

Also, just to clarify, there are two models of using encrypted images with kubernetes, what is do-able today with containerd/cri and cri-o is independent of this KEP. This KEP is an effort to provide the ability to have stronger multitenancy using encrypted images.

It seems that I can now use the kubernetes(>=1.17) and docker(containerd) to achieve image encryption&decryption with the model independent of this KEP, but is there any official kubernetes documents about how to use it besides this article?

That article talks about the threat model where you only trust the worker nodes of your k8s cluster (but not other components, like api server, docker registry etc). It requires that the decryption keys are kept on the target worker node. The support for this is already present in upstream in CRIO. So you might have to use CRIO instead of Containerd with your k8s setup. You don't need to change anything in k8s for this to work.

We are also hoping to get our changes merged soon in CRI-Containerd too.

As Brandon mentioned, the code in this KEP is part of the effort to provide stronger multi-tenancy using encrypted images. In this the kubernetes api server and worker nodes are both trusted entities (but docker registry is not). But as I mentioned earlier, this code is not upstreamed yet.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.