AMD SEV technology enablement

[vasiliy-ul]

This is the placeholder issue for tracking the progress of AMD Memory Encryption features (SEV, SEV-ES, SEV-SNP) enablement in KubeVirt.

Original discussion from the mailing list: https://groups.google.com/g/kubevirt-dev/c/a6XlT0qRNAY/m/IWFAOu2xAAAJ

/kind enhancement

Implementation

• Basic SEV support
• Launch Security with AMD SEV #6166
• Encrypted State (SEV-ES)
• Launch Security with AMD SEV-ES #7742
• Secure Nested Paging (SEV-SNP)
• Attestation
• [WIP] Libvirt API: https://listman.redhat.com/archives/libvir-list/2021-December/msg00800.html
• virDomainSetLaunchSecurityState API implemented in libvirt 8.0.0 (allows injecting a secret into a running VM)
• virsh nodesevinfo command implemented in libvirt 8.0.0 (provides platform certificates)
• (!) Proposal to generalize the attestation flow in libvirt: https://listman.redhat.com/archives/libvir-list/2022-February/228809.html
• Introduce API endpoints for SEV attestation #7197
• Extend virtctl to Support Interactive SEV Attestation #7595
• Support rotation of the platform certificates
• Support external kernel / initrd / cmdline
• Avoid using EFI vars storage with confidential guests
• Expose attestation data SEVPlatformInfo and SEVMeasurementInfo in VMIStatus
• Expose /dev/sev device to pods only if the feature gate is enabled
• Respect the feature gate when exposing SEV device #8099

Testing

• Update the test image for CI
• tests: Use fedora-with-test-tooling based on Fedora 35 #6958
• fedora-with-test-tooling: Bump base image to fc35 kubevirtci#713
• Run functional tests on AMD hardware
• Add template prowjobs for testing against SEV cluster project-infra#2606
• Add document on AMD SEV testing cluster project-infra#2608
• Add sevctl to the test image #9892
• Use encrypted image for testing
• [WIP] GRUB: https://lists.gnu.org/archive/html/grub-devel/2022-02/msg00064.html

Documentation

• Initial design proposal
• Design Proposal: AMD SEV support in KubeVirt community#141
• Update user-guide
• Document basic SEV support user-guide#539
• Ensure the design proposal and user-guide are in sync with the implementation

[xpivarc]

How the plan for testing looks like? Would it be possible to test this with non-root? Thanks, @vasiliy-ul !

[vasiliy-ul]

HI @xpivarc . The actual test cases are provided with the implementation PR (well, only one test case so far...). Though for testing the feature there are a couple of prerequisites that are still missing: 1) we need to have a guest VM with recent kernel and UEFI boot support. The relevant PRs (switch to Fedora 35) are under discussion at the moment; 2) the feature requires specific hardware to run on. I am not sure if there is a test machine with AMD EPYC CPU available in kubevirt CI cluster for that. For now I tested and verified manually on AMD machine.

Regarding non-root: I cannot think of any potetnial issue at the moment. Qemu will just need to have RW access to /dev/sev device which can be setup in the same way as other devices (e.g. /dev/kvm). So in theory it should work with non-root... but I might be too optimistic :)

[xpivarc]

HI @xpivarc . The actual test cases are provided with the implementation PR (well, only one test case so far...). Though for testing the feature there are a couple of prerequisites that are still missing: 1) we need to have a guest VM with recent kernel and UEFI boot support. The relevant PRs (switch to Fedora 35) are under discussion at the moment; 2) the feature requires specific hardware to run on. I am not sure if there is a test machine with AMD EPYC CPU available in kubevirt CI cluster for that. For now I tested and verified manually on AMD machine.

As far as I know, we are missing the hw in our clusters.

Regarding non-root: I cannot think of any potetnial issue at the moment. Qemu will just need to have RW access to /dev/sev device which can be setup in the same way as other devices (e.g. /dev/kvm). So in theory it should work with non-root... but I might be too optimistic :)

Thanks for the response. It is also our assumption but we want to be sure :) I would appreciate it if you give it a try.
/cc @vladikr

[vasiliy-ul]

It is also our assumption but we want to be sure :) I would appreciate it if you give it a try.

Hi @xpivarc . I tried to run a SEV guest with non-root. Everything was fine. The access to /dev/sev is properly configured for qemu user and I think that libvirt does not actually talk to the device directly (it only queries the domain for SEV info). So, seems there are no issues with that.

[xpivarc]

It is also our assumption but we want to be sure :) I would appreciate it if you give it a try.

Hi @xpivarc . I tried to run a SEV guest with non-root. Everything was fine. The access to /dev/sev is properly configured for qemu user and I think that libvirt does not actually talk to the device directly (it only queries the domain for SEV info). So, seems there are no issues with that.

@vasiliy-ul Thank you!

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[iholder101]

/remove-lifecycle stale

[hw-claudio]

Hi all, this is up-to-date right? We don't have SEV-ES yet

[alicefr]

@hw-claudio not yet. We are still integrating the SEV CI nodes to be able to test the code changes

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[hw-claudio]

@alicefr noticed the bot inactivity message, are the SEV CI nodes integrated now, is AMD SEV-ES functional?

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[vasiliy-ul]

/remove-lifecycle rotten

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[vasiliy-ul]

/remove-lifecycle stale

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[FishmanL]

Is this live with ES/SNP? Currently working with libvirt on one of our clusters and would love to be able to fold this into our CoCo build.

[vasiliy-ul]

SEV-ES is already enabled. SNP is WIP in kernel/qemu/libvirt (not yet upstreamed), hence not supported in KubeVirt so far.

/remove-lifecycle rotten

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale