docs: provide a guide for how to use IBM Secure Execution

This PR is to add a document for how to run kata containers under IBM
Secure Execution environment.

Fixes: #7025

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>

docs/how-to/README.md

@@ -44,5 +44,6 @@
 - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md)
 - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md)
 - [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md)
+- [How to run Kata Containers with IBM Secure Execution](how-to-run-kata-containers-with-SE-VMs.md)
 - [How to use EROFS to build rootfs in Kata Containers](how-to-use-erofs-build-rootfs.md)
 - [How to run Kata Containers with kinds of Block Volumes](how-to-run-kata-containers-with-kinds-of-Block-Volumes.md)

docs/how-to/how-to-run-kata-containers-with-SE-VMs.md

@@ -0,0 +1,336 @@
+# Kata Containers with IBM Secure Execution VMs
+
+This document assumes a trusted environment with a functioning kata container, as per the
+[developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md).
+The term "trusted" implies that the system is authorized, authenticated and attested to use your artifacts and secrets safely.
+
+## Manual configuration
+
+### Prerequisite
+
+1. Host capable of Secure Execution
+
+To take advantage of the IBM Secure Execution capability, the host machine on which
+you intend to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III
+(or a newer model). In addition to the hardware requirement, you need to verify the CPU
+facility and kernel configuration, as outlined below:
+
+```
+$ # To check the protected virtualization support from kernel
+$ cat /sys/firmware/uv/prot_virt_host
+1
+$ # To check if an ultravisor reserves memory for the current boot
+$ dmesg | grep -i ultravisor
+[    0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage
+$ # To check a facility bit for Secure Execution
+$ cat /proc/cpuinfo | grep 158
+facilities      :  ... numbers ... 158 ... numbers ...
+```
+
+If any of the results are not identifiable, please reach out to the responsible cloud
+provider to enable the Secure Execution capability. Alternatively, if you possess
+administrative privileges and the facility bit is set, you can enable the Secure Execution
+capability by configuring `prot_virt=1` and performing a system reboot with:
+
+```
+$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf
+$ sudo zipl -V
+$ sudo systemctl reboot
+```
+
+2. Artifacts from Kata Containers
+
+A secure image is constructed using the following artifacts
+
+- A raw kernel
+- An initial RAM disk
+
+The most straightforward approach to obtain these artifacts is by reusing kata-containers:
+
+```
+$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path'))
+config-5.19.2
+kata-containers.img
+kata-containers-initrd.img
+kata-ubuntu-20.04.initrd
+kata-ubuntu-latest.image
+vmlinux-5.19.2-107
+vmlinux.container
+vmlinuz-5.19.2-107
+vmlinuz.container
+```
+
+The output indicates the deployment of the kernel (`vmlinux-5.19.2-107`),
+rootfs-image (`kata-ubuntu-latest.image`), and rootfs-initrd (`kata-ubuntu-20.04.initrd`).
+In this scenario, the available kernel and initrd can be utilized for a secure image.
+However, if any of these components are absent, they must be built from the
+[project source](https://github.com/kata-containers/kata-containers) as follows:
+
+```
+$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ sudo -E PATH=$PATH make kernel-tarball
+$ sudo -E PATH=$PATH make rootfs-initrd-tarball
+$ tar -tf build/kata-static-kernel.tar.xz | grep vmlinuz
+./opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+./opt/kata/share/kata-containers/vmlinuz.container
+$ tar -tf build/kata-static-rootfs-initrd.tar.xz | grep initrd
+./opt/kata/share/kata-containers/kata-containers-initrd.img
+./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ mkdir artifacts
+$ tar -xvf build/kata-static-kernel.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+$ tar -xvf build/kata-static-rootfs-initrd.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ ls artifacts/opt/kata/share/kata-containers/
+kata-ubuntu-20.04.initrd  vmlinuz-5.19.2-107
+```
+
+3. Genprotimg Tool
+
+`genprotimg` is a utility designed to generate an IBM Secure Execution image. It can be
+installed either from the package manager of a distribution or from the source code.
+The tool is included in the `s390-tools` package. Please ensure that you have a version
+of the tool equal to or greater than `2.17.0`. If not, you will need to specify
+an additional argument, `--x-pcf '0xe0'`, when running the command.
+Here is an example of a native build from the source:
+
+```
+$ tool_version=v2.25.0
+$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git && cd s390-tools
+$ pushd genprotimg && make && sudo make install && popd
+```
+
+4. Host Key Document
+
+A host key document is a public key employed for encrypting a secure image, which is
+subsequently decrypted using a corresponding private key during the VM bootstrap process.
+You can obtain the host key document either through IBM's designated
+[Resource Link](http://www.ibm.com/servers/resourcelink) or by requesting it from the
+cloud provider responsible for the IBM Z and LinuxONE instances where your workloads are intended to run.
+
+To ensure security, it is essential to verify the authenticity and integrity of the host key document
+belonging to an authentic IBM machine. To achieve this, please additionally obtain the following
+certificates from the Resource Link:
+
+- IBM zSystem signing key certificate
+- DigiCert intermediate CA certificate
+
+These files will be used for verification during secure image construction in the next section.
+
+### Build a Secure Image
+
+
+Assuming you have placed a host key document at `$HOME/host-key-document`:
+
+- Host key document as `HKD-0000-0000000.crt`
+
+and two certificates at `$HOME/certificates`:
+
+- DigiCert intermediate CA certificate as `DigiCertCA.crt`
+- IBM Z signing-key certificate as `ibm-z-host-key-signing.crt`
+
+you can construct a secure image using the following procedure:
+
+```
+$ # Change a directory to the project root
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile
+$ genprotimg --host-key-document=${host_key_document} \
+--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
+--parmfile=parmfile --no-verify
+WARNING: host-key document verification is disabled. Your workload is not secured.
+$ file kata-containers-se.img
+kata-containers-se.img: data
+$ cp kata-containers-se.img /opt/kata/share/kata-containers/
+```
+
+It is important to note that the `--no-verify` parameter, which allows skipping
+the key verification process, is intended to be used solely in a development or
+testing environment.
+In production, the image construction should incorporate the verification
+in the following manner:
+
+```
+$ cacert=$HOME/certificates/DigiCertCA.crt
+$ signcert=$HOME/certificates/ibm-z-host-key-signing.crt
+$ genprotimg --host-key-document=${host_key_document} \
+--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
+--cert=${cacert} --cert=${signcert} --parmfile=parmfile
+```
+
+The steps with no verification, including the dependencies for the kernel and initrd,
+can be easily accomplished by issuing the following make target:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ ls build/kata-static-boot-image-se.tar.xz
+build/kata-static-boot-image-se.tar.xz
+```
+
+In production, you could build an image by running the same command, but with two
+additional environment variables for key verification:
+
+```
+$ export SIGNING_KEY_CERT_PATH=$HOME/certificates/ibm-z-host-key-signing.crt
+$ export INTERMEDIATE_CA_CERT_PATH=$HOME/certificates/DigiCertCA.crt
+```
+
+### Adjust the configuration
+
+There still remains an opportunity to fine-tune the configuration file:
+
+```
+$ cp /opt/kata/share/defaults/kata-containers/configuration.toml \
+/opt/kata/share/defaults/kata-containers/configuration.toml.old
+$ # Make the following adjustment to the original config file
+$ diff /opt/kata/share/defaults/kata-containers/configuration.toml.old \
+/opt/kata/share/defaults/kata-containers/configuration.toml
+16,17c16,17
+< kernel = "/opt/kata/share/kata-containers/vmlinux.container"
+< image = "/opt/kata/share/kata-containers/kata-containers.img"
+---
+> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img"
+> # image = "/opt/kata/share/kata-containers/kata-containers.img"
+41c41
+< # confidential_guest = true
+---
+> confidential_guest = true
+544c544
+< dial_timeout = 45
+---
+> dial_timeout = 90
+679c679
+< #service_offload = true
+---
+> service_offload = true
+```
+
+### Verification
+
+To verify the successful decryption and loading of the secure image within a test VM,
+please refer to the following commands:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path')
+$ secure_kernel=kata-containers-se.img
+$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \
+-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \
+--nographic --nodefaults --kernel "${secure_kernel}"
+[    0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023                                                                     [    0.110279] setup: Linux is running under KVM in 64-bit mode
+
+... log skipped ...
+
+[    1.467228] Run /init as init process
+{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent
+","version":"0.1.0"}
+
+... log skipped ...
+
+$ # Press ctrl + a + x to exit
+```
+
+If the hypervisor log does not indicate any errors, it provides assurance that the image
+has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime
+will function properly.
+
+Let us proceed with the final verification by running a test container in a Kubernetes
+cluster:
+
+```
+$ kubectl get node
+NAME           STATUS   ROLES                  AGE     VERSION
+test-cluster   Ready    control-plane,master   7m28s   v1.23.1
+$ cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx-kata
+spec:
+  runtimeClassName: kata
+  containers:
+  - name: nginx
+    image: nginx
+EOF
+pod/nginx-kata created
+$ kubectl get po
+NAME         READY   STATUS    RESTARTS   AGE
+nginx-kata   1/1     Running   0          29s
+```
+
+Finally, an operational kata container with IBM Secure Execution is now running.
+
+## Using Kata-Deploy
+
+It is reasonable to anticipate that the aforementioned manual steps can be readily
+accomplished by the
+[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md).
+The following commands explain how to build an archive named `kata-static.tar.xz`:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ # kernel and rootfs-initrd are built automactially by the command below
+$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ sudo -E PATH=$PATH make qemu-tarball
+$ sudo -E PATH=$PATH make virtiofsd-tarball
+$ # shim-v2 should be built after kernel due to dependency
+$ sudo -E PATH=$PATH make shim-v2-tarball
+$ mkdir kata-artifacts
+$ build_dir=$(readlink -f build)
+$ cp -r $build_dir/*.tar.xz kata-artifacts
+$ ./tools/packaging/kata-deploy/local-build/kata-deploy-merge-builds.sh kata-artifacts
+```
+
+In production, the environment variables `SIGNING_KEY_CERT_PATH` and
+`INTERMEDIATE_CA_CERT_PATH` should be exported like the manual configuration.
+If a rootfs-image is required for testing purposes without the Secure Execution
+functionality, it is necessary to enter the following command before running
+`kata-deploy-merge-builds.sh`:
+
+```
+$ sudo -E PATH=$PATH make rootfs-image-tarball
+```
+
+At this point, you have an archive used by a
+[Dockerfile](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/Dockerfile) for kata-deploy.
+
+### Verification
+
+When using kata-deploy, a different runtime class name should be used for verification:
+
+```
+$ cat <<EOF | kubectl apply -f -
+> apiVersion: v1
+> kind: Pod
+> metadata:
+>   name: nginx-kata
+> spec:
+>   runtimeClassName: kata-qemu-se
+>   containers:
+>   - name: nginx
+>     image: nginx
+> EOF
+pod/nginx-kata created
+$ kubectl get po
+NAME         READY   STATUS    RESTARTS   AGE
+nginx-kata   1/1     Running   0          15s
+```
+
+## Considerations for CI
+
+If you intend to integrate the aforementioned procedure with a CI system, it is
+recommended to configure the following setup for an environment variable.
+The setup helps speed up CI jobs by caching container images used during the build:
+
+```
+$ export BUILDER_REGISTRY=$YOUR_PRIVATE_REGISTRY_FOR_CI
+$ export PUSH_TO_REGISTRY=yes
+```