docs: provide a guide for how to use IBM SE

This PR is to add a document for how to run kata containers under IBM
secure execution(SE) environment.

Fixes: #7025

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>

docs/how-to/README.md

@@ -44,5 +44,6 @@
 - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md)
 - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md)
 - [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md)
+- [How to run Kata Containers with IBM SE](how-to-run-kata-containers-with-SE-VMs.md)
 - [How to use EROFS to build rootfs in Kata Containers](how-to-use-erofs-build-rootfs.md)
 - [How to run Kata Containers with kinds of Block Volumes](how-to-run-kata-containers-with-kinds-of-Block-Volumes.md)

docs/how-to/how-to-run-kata-containers-with-SE-VMs.md

@@ -0,0 +1,275 @@
+# Kata Containers with IBM SE VMs
+
+## Manual configuration
+
+This section assumes an environment with a functioning kata container, as per the
+[developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md).
+
+### Prerequisite
+
+1. SE-enabled Host
+
+To leverage the IBM Secure Execution (SE) capability, the host machine on which you intend
+to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III (or a newer
+model). In addition to the hardware requirement, you need to verify the CPU facility and
+kernel configuration, as outlined below:
+
+```
+$ # To check the protected virtualization support from kernel
+$ grep -q 'prot_virt=1' /etc/zipl.conf && echo "found" || echo "not found"
+found
+$ # To check if an ultravior reserves a memory for the current boot
+$ dmesg | grep -i ultravisor
+[    0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage
+$ # To check a facility bit for SE
+$ cat /proc/cpuinfo | grep 158
+facilities      :  ... numbers ... 158 ... numbers ...
+```
+
+If any of the results are not identifiable, please reach out to the responsible cloud
+provider to enable the secure execution capability. Alternatively, if you possess
+administrative privileges and the facility bit is set, you can enable the SE capability
+by configuring `prot_virt=1` and performing a system reboot with:
+
+```
+$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf
+$ sudo zipl -V
+$ sudo systemctl reboot
+```
+
+2. Artifacts from Kata Containers
+
+A secure image is constructed using the following artifacts
+
+- A raw kernel
+- An initial RAM disk
+
+The most straightforward approach to obtain these artifacts is by reusing kata-containers:
+
+```
+$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path'))
+config-5.19.2
+kata-containers.img
+kata-containers-initrd.img
+kata-ubuntu-20.04.initrd
+kata-ubuntu-latest.image
+vmlinux-5.19.2-107
+vmlinux.container
+vmlinuz-5.19.2-107
+vmlinuz.container
+```
+
+The output indicates the deployment of the kernel (`vmlinux-5.19.2-107`),
+rootfs-image (`kata-ubuntu-latest.image`), and rootfs-initrd (`kata-ubuntu-20.04.initrd`).
+In this scenario, the available kernel and initrd can be utilized for a secure image.
+However, if any of these components are absent, they must be built from the
+[project source](https://github.com/kata-containers/kata-containers) as follows:
+
+```
+$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ sudo -E PATH=$PATH make kernel-tarball
+$ sudo -E PATH=$PATH make rootfs-initrd-tarball
+$ tar -tf build/kata-static-kernel.tar.xz | grep vmlinuz
+./opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+./opt/kata/share/kata-containers/vmlinuz.container
+$ tar -tf build/kata-static-rootfs-initrd.tar.xz | grep initrd
+./opt/kata/share/kata-containers/kata-containers-initrd.img
+./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ mkdir artifacts
+$ tar -xvf build/kata-static-kernel.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+$ tar -xvf build/kata-static-rootfs-initrd.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ ls artifacts/opt/kata/share/kata-containers/
+kata-ubuntu-20.04.initrd  vmlinuz-5.19.2-107
+```
+
+3. Genprotimg Tool
+
+`genprotimg` is a utility designed to generate an IBM SE image. It can be installed either
+from the package manager of a distribution or from the source code. The tool is included
+in the `s390-tools` package. Please ensure that you have a version of the tool equal to 
+or greater than `2.17.0`. If not, you will need to specify an additional argument,
+`--x-pcf '0xe0'`, when executing the command.
+Here is an example of a native build from the source:
+
+```
+$ tool_version=v2.25.0
+$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git && cd s390-tools
+$ pushd genprotimg && pushd boot && make
+$ popd && pushd src && make
+$ popd && sudo make install && popd
+```
+
+4. Host Key Document
+
+A host key document is a public key employed for encrypting a secure image, which is
+subsequently decrypted using a corresponding private key during the VM bootstrap process.
+You can obtain the host key document either through IBM's designated
+[resource link](http://www.ibm.com/servers/resourcelink) or by requesting it from the
+cloud provider responsible for the zSystem where your workloads are executed.
+
+### Build a Secure Image
+
+Assuming you have placed the host key document at
+`$HOME/host-key-documenet/HKD-0000-0000000.crt`,
+you can construct a secure image using the following procedure:
+
+```
+$ # Change a directory to the project root
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-5.19.2-107
+$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd
+$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile
+$ sudo genprotimg --host-key-document=${host_key_document} \
+--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \
+--parmfile=parmfile --no-verify
+WARNING: host-key document verification is disabled. Your workload is not secured.
+$ file kata-containers-se.img
+kata-containers-se.img: data
+$ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/
+```
+
+The aforementioned steps, including the dependencies for the kernel and initrd,
+can be easily accomplished by executing the following make target:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ ls build/kata-static-boot-image-se.tar.xz
+build/kata-static-boot-image-se.tar.xz
+```
+
+### Adjust the configuration
+
+There still remains an opportunity to fine-tune the configuration file:
+
+```
+$ sudo cp /opt/kata/share/defaults/kata-containers/configuration.toml \
+/opt/kata/share/defaults/kata-containers/configuration.toml.old
+$ # Make the following adjustment to the original config file
+$ diff /opt/kata/share/defaults/kata-containers/configuration.toml.old \
+/opt/kata/share/defaults/kata-containers/configuration.toml
+16,17c16,17
+< kernel = "/opt/kata/share/kata-containers/vmlinux.container"
+< image = "/opt/kata/share/kata-containers/kata-containers.img"
+---
+> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img"
+> # image = "/opt/kata/share/kata-containers/kata-containers.img"
+41c41
+< # confidential_guest = true
+---
+> confidential_guest = true
+544c544
+< dial_timeout = 45
+---
+> dial_timeout = 90
+679c679
+< #service_offload = true
+---
+> service_offload = true
+```
+
+### Verification
+
+To verify the successful decryption and loading of the secure image within a test VM,
+please refer to the following commands:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path')
+$ secure_kernel=kata-containers-se.img
+$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \
+-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \
+--nographic --nodefaults --kernel "${secure_kernel}"
+[    0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023                                                                     [    0.110279] setup: Linux is running under KVM in 64-bit mode
+
+... log skipped ...
+
+[    1.467228] Run /init as init process
+{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent
+","version":"0.1.0"}
+
+... log skipped ...
+
+$ # Press ctrl + a + x to exit
+```
+
+If the hypervisor log does not indicate any errors, it provides assurance that the image
+has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime
+will function properly.
+
+Let us proceed with the final verification by running a test container in a Kubernetes
+cluster:
+
+```
+$ kubectl get node
+NAME           STATUS   ROLES                  AGE     VERSION
+test-cluster   Ready    control-plane,master   7m28s   v1.23.1
+$ cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx-kata
+spec:
+  runtimeClassName: kata
+  containers:
+  - name: nginx
+    image: nginx
+EOF
+pod/nginx-kata created
+$ kubectl get po
+NAME         READY   STATUS    RESTARTS   AGE
+nginx-kata   1/1     Running   0          29s
+```
+
+Finally, an operational kata container with IBM SE is now running. 
+
+## Using Kata-Deploy
+
+It is reasonable to anticipate that the aforementioned manual steps can be readily
+accomplished by the
+[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md).
+The following commands explains how to build an archive named `kata-static.tar.xz`:
+
+```
+$ cd $GOPATH/src/github.com/kata-containers/kata-containers
+$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt
+$ mkdir hkd_dir && cp $host_key_document hkd_dir
+$ # kernel and rootfs-initrd are built automactially by the command below
+$ sudo -E PATH=$PATH TEE_TYPE=se HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \
+make boot-image-se-tarball
+$ sudo -E PATH=$PATH make qemu-tarball
+$ sudo -E PATH=$PATH make virtiofsd-tarball
+$ # shim-v2 should be built after kernel due to dependency
+$ sudo -E PATH=$PATH make shim-v2-tarball
+$ mkdir kata-artifacts
+$ build_dir=$(readlink -f build)
+$ sudo cp -r $build_dir/*.tar.xz kata-artifacts
+$ sudo chown -R $(id -u):$(id -g) kata-artifacts
+$ ./tools/packaging/kata-deploy/local-build/kata-deploy-merge-builds.sh kata-artifacts
+```
+
+If a rootfs-image is required for testing purposes without SE functionality,
+it is necessary to execute the following command
+before running `kata-deploy-merge-builds.sh`:
+
+```
+$ sudo -E PATH=$PATH make rootfs-image-tarball
+```
+
+At this point, you have an archive used by a
+[Dockerfile](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/Dockerfile) for kata-deploy.
+
+## Considerations for CI
+
+If you intend to integrate the aforementioned procedure with a CI system, it is
+recommended to configure the following setup for an environment variable.
+The setup helps speed up CI jobs by caching container images used during the build:
+
+```
+$ export BUILDER_REGISTRY=$YOUR_PRIVATE_REGISTRY_FOR_CI
+$ export PUSH_TO_REGISTRY=yes
+```