-
Notifications
You must be signed in to change notification settings - Fork 996
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: provide a guide for how to use IBM SE
This PR is to add a document for how to run kata containers under IBM secure execution(SE) environment. Fixes: #7025 Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
- Loading branch information
Showing
2 changed files
with
298 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,297 @@ | ||
# Kata Containers with IBM SE VMs | ||
|
||
## Manual configuration | ||
|
||
This section assumes an environment with a functioning kata container, as per the | ||
[developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md). | ||
|
||
### Prerequisite | ||
|
||
1. SE-enabled Host | ||
|
||
To leverage the IBM Secure Execution (SE) capability, the host machine on which you intend | ||
to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III (or a newer | ||
model). In addition to the hardware requirement, you need to verify the CPU facility and | ||
kernel configuration, as outlined below: | ||
|
||
``` | ||
$ # To check the protected virtualization support from kernel | ||
$ grep -q 'prot_virt=1' /etc/zipl.conf && echo "found" || echo "not found" | ||
found | ||
$ # To check if an ultravisor reserves a memory for the current boot | ||
$ dmesg | grep -i ultravisor | ||
[ 0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage | ||
$ # To check a facility bit for SE | ||
$ cat /proc/cpuinfo | grep 158 | ||
facilities : ... numbers ... 158 ... numbers ... | ||
``` | ||
|
||
If any of the results are not identifiable, please reach out to the responsible cloud | ||
provider to enable the secure execution capability. Alternatively, if you possess | ||
administrative privileges and the facility bit is set, you can enable the SE capability | ||
by configuring `prot_virt=1` and performing a system reboot with: | ||
|
||
``` | ||
$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf | ||
$ sudo zipl -V | ||
$ sudo systemctl reboot | ||
``` | ||
|
||
2. Artifacts from Kata Containers | ||
|
||
A secure image is constructed using the following artifacts | ||
|
||
- A raw kernel | ||
- An initial RAM disk | ||
|
||
The most straightforward approach to obtain these artifacts is by reusing kata-containers: | ||
|
||
``` | ||
$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path')) | ||
config-5.19.2 | ||
kata-containers.img | ||
kata-containers-initrd.img | ||
kata-ubuntu-20.04.initrd | ||
kata-ubuntu-latest.image | ||
vmlinux-5.19.2-107 | ||
vmlinux.container | ||
vmlinuz-5.19.2-107 | ||
vmlinuz.container | ||
``` | ||
|
||
The output indicates the deployment of the kernel (`vmlinux-5.19.2-107`), | ||
rootfs-image (`kata-ubuntu-latest.image`), and rootfs-initrd (`kata-ubuntu-20.04.initrd`). | ||
In this scenario, the available kernel and initrd can be utilized for a secure image. | ||
However, if any of these components are absent, they must be built from the | ||
[project source](https://github.com/kata-containers/kata-containers) as follows: | ||
|
||
``` | ||
$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers | ||
$ cd $GOPATH/src/github.com/kata-containers/kata-containers | ||
$ sudo -E PATH=$PATH make kernel-tarball | ||
$ sudo -E PATH=$PATH make rootfs-initrd-tarball | ||
$ tar -tf build/kata-static-kernel.tar.xz | grep vmlinuz | ||
./opt/kata/share/kata-containers/vmlinuz-5.19.2-107 | ||
./opt/kata/share/kata-containers/vmlinuz.container | ||
$ tar -tf build/kata-static-rootfs-initrd.tar.xz | grep initrd | ||
./opt/kata/share/kata-containers/kata-containers-initrd.img | ||
./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd | ||
$ mkdir artifacts | ||
$ tar -xvf build/kata-static-kernel.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-5.19.2-107 | ||
$ tar -xvf build/kata-static-rootfs-initrd.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd | ||
$ ls artifacts/opt/kata/share/kata-containers/ | ||
kata-ubuntu-20.04.initrd vmlinuz-5.19.2-107 | ||
``` | ||
|
||
3. Genprotimg Tool | ||
|
||
`genprotimg` is a utility designed to generate an IBM SE image. It can be installed either | ||
from the package manager of a distribution or from the source code. The tool is included | ||
in the `s390-tools` package. Please ensure that you have a version of the tool equal to | ||
or greater than `2.17.0`. If not, you will need to specify an additional argument, | ||
`--x-pcf '0xe0'`, when executing the command. | ||
Here is an example of a native build from the source: | ||
|
||
``` | ||
$ tool_version=v2.25.0 | ||
$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git && cd s390-tools | ||
$ pushd genprotimg && pushd boot && make | ||
$ popd && pushd src && make | ||
$ popd && sudo make install && popd | ||
``` | ||
|
||
4. Host Key Document | ||
|
||
A host key document is a public key employed for encrypting a secure image, which is | ||
subsequently decrypted using a corresponding private key during the VM bootstrap process. | ||
You can obtain the host key document either through IBM's designated | ||
[resource link](http://www.ibm.com/servers/resourcelink) or by requesting it from the | ||
cloud provider responsible for the zSystem where your workloads are executed. | ||
|
||
### Build a Secure Image | ||
|
||
Assuming you have placed the host key document at | ||
`$HOME/host-key-documenet/HKD-0000-0000000.crt`, | ||
you can construct a secure image using the following procedure: | ||
|
||
``` | ||
$ # Change a directory to the project root | ||
$ cd $GOPATH/src/github.com/kata-containers/kata-containers | ||
$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt | ||
$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-5.19.2-107 | ||
$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd | ||
$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile | ||
$ sudo genprotimg --host-key-document=${host_key_document} \ | ||
--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \ | ||
--parmfile=parmfile --no-verify | ||
WARNING: host-key document verification is disabled. Your workload is not secured. | ||
$ file kata-containers-se.img | ||
kata-containers-se.img: data | ||
$ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/ | ||
``` | ||
|
||
The aforementioned steps, including the dependencies for the kernel and initrd, | ||
can be easily accomplished by executing the following make target: | ||
|
||
``` | ||
$ cd $GOPATH/src/github.com/kata-containers/kata-containers | ||
$ mkdir hkd_dir && cp $host_key_document hkd_dir | ||
$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \ | ||
make boot-image-se-tarball | ||
$ ls build/kata-static-boot-image-se.tar.xz | ||
build/kata-static-boot-image-se.tar.xz | ||
``` | ||
|
||
### Adjust the configuration | ||
|
||
There still remains an opportunity to fine-tune the configuration file: | ||
|
||
``` | ||
$ sudo cp /opt/kata/share/defaults/kata-containers/configuration.toml \ | ||
/opt/kata/share/defaults/kata-containers/configuration.toml.old | ||
$ # Make the following adjustment to the original config file | ||
$ diff /opt/kata/share/defaults/kata-containers/configuration.toml.old \ | ||
/opt/kata/share/defaults/kata-containers/configuration.toml | ||
16,17c16,17 | ||
< kernel = "/opt/kata/share/kata-containers/vmlinux.container" | ||
< image = "/opt/kata/share/kata-containers/kata-containers.img" | ||
--- | ||
> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img" | ||
> # image = "/opt/kata/share/kata-containers/kata-containers.img" | ||
41c41 | ||
< # confidential_guest = true | ||
--- | ||
> confidential_guest = true | ||
544c544 | ||
< dial_timeout = 45 | ||
--- | ||
> dial_timeout = 90 | ||
679c679 | ||
< #service_offload = true | ||
--- | ||
> service_offload = true | ||
``` | ||
|
||
### Verification | ||
|
||
To verify the successful decryption and loading of the secure image within a test VM, | ||
please refer to the following commands: | ||
|
||
``` | ||
$ cd $GOPATH/src/github.com/kata-containers/kata-containers | ||
$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path') | ||
$ secure_kernel=kata-containers-se.img | ||
$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \ | ||
-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \ | ||
--nographic --nodefaults --kernel "${secure_kernel}" | ||
[ 0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023 [ 0.110279] setup: Linux is running under KVM in 64-bit mode | ||
... log skipped ... | ||
[ 1.467228] Run /init as init process | ||
{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent | ||
","version":"0.1.0"} | ||
... log skipped ... | ||
$ # Press ctrl + a + x to exit | ||
``` | ||
|
||
If the hypervisor log does not indicate any errors, it provides assurance that the image | ||
has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime | ||
will function properly. | ||
|
||
Let us proceed with the final verification by running a test container in a Kubernetes | ||
cluster: | ||
|
||
``` | ||
$ kubectl get node | ||
NAME STATUS ROLES AGE VERSION | ||
test-cluster Ready control-plane,master 7m28s v1.23.1 | ||
$ cat <<EOF | kubectl apply -f - | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
name: nginx-kata | ||
spec: | ||
runtimeClassName: kata | ||
containers: | ||
- name: nginx | ||
image: nginx | ||
EOF | ||
pod/nginx-kata created | ||
$ kubectl get po | ||
NAME READY STATUS RESTARTS AGE | ||
nginx-kata 1/1 Running 0 29s | ||
``` | ||
|
||
Finally, an operational kata container with IBM SE is now running. | ||
|
||
## Using Kata-Deploy | ||
|
||
It is reasonable to anticipate that the aforementioned manual steps can be readily | ||
accomplished by the | ||
[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md). | ||
The following commands explains how to build an archive named `kata-static.tar.xz`: | ||
|
||
``` | ||
$ cd $GOPATH/src/github.com/kata-containers/kata-containers | ||
$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt | ||
$ mkdir hkd_dir && cp $host_key_document hkd_dir | ||
$ # kernel and rootfs-initrd are built automactially by the command below | ||
$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \ | ||
make boot-image-se-tarball | ||
$ sudo -E PATH=$PATH make qemu-tarball | ||
$ sudo -E PATH=$PATH make virtiofsd-tarball | ||
$ # shim-v2 should be built after kernel due to dependency | ||
$ sudo -E PATH=$PATH make shim-v2-tarball | ||
$ mkdir kata-artifacts | ||
$ build_dir=$(readlink -f build) | ||
$ sudo cp -r $build_dir/*.tar.xz kata-artifacts | ||
$ sudo chown -R $(id -u):$(id -g) kata-artifacts | ||
$ ./tools/packaging/kata-deploy/local-build/kata-deploy-merge-builds.sh kata-artifacts | ||
``` | ||
|
||
If a rootfs-image is required for testing purposes without SE functionality, | ||
it is necessary to execute the following command | ||
before running `kata-deploy-merge-builds.sh`: | ||
|
||
``` | ||
$ sudo -E PATH=$PATH make rootfs-image-tarball | ||
``` | ||
|
||
At this point, you have an archive used by a | ||
[Dockerfile](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/Dockerfile) for kata-deploy. | ||
|
||
### Verification | ||
|
||
When using kata-deploy, a different runtime class name should be used for verification: | ||
|
||
``` | ||
$ cat <<EOF | kubectl apply -f - | ||
> apiVersion: v1 | ||
> kind: Pod | ||
> metadata: | ||
> name: nginx-kata | ||
> spec: | ||
> runtimeClassName: kata-qemu-se | ||
> containers: | ||
> - name: nginx | ||
> image: nginx | ||
> EOF | ||
pod/nginx-kata created | ||
$ kubectl get po | ||
NAME READY STATUS RESTARTS AGE | ||
nginx-kata 1/1 Running 0 15s | ||
``` | ||
|
||
## Considerations for CI | ||
|
||
If you intend to integrate the aforementioned procedure with a CI system, it is | ||
recommended to configure the following setup for an environment variable. | ||
The setup helps speed up CI jobs by caching container images used during the build: | ||
|
||
``` | ||
$ export BUILDER_REGISTRY=$YOUR_PRIVATE_REGISTRY_FOR_CI | ||
$ export PUSH_TO_REGISTRY=yes | ||
``` |