# Release 2.3.0

kata-containers Changes

2.3 release of Kata Containers introduces several new features and cleanup. Highlights include of 2.3.0 include:

• Better support for VFIO to enable DPDK workloads
• Improvements on getting metrics via Kata Monitor
• Improvements on kata-deploy and its documentation
• Seccomp support
• Improvements on tracing
• Fixes / Improvements from the host cgroups
• Fixes for the agent's files watcher
• Fixes for the Firecracker integration
• Support for swap device within guest
• A whole lot of cleanups

There are a few changes which will impact users when upgrading to 2.3. Take note of the following:

• The runtime was updated to utilize QEMU 6.1. Utilization of older versions of QEMU are not supported, and some features will be broken if 6.1 isn't utilized (at least block device support).
• The 2.3 agent introduces SECCOMP support, which is enabled by default in the initial 2.3 release. With its initial support, static build of the agent is broken when SECCOMP is utilized. If users require a statically built agent, they should consider building the agent without sec comp (for example, cd ./src/agent && make SECCOMP=no
• As part of our cleanup and refactoring, built configuration files can be found at ./src/runtime/config/ instead of ./src/runtime/cli/config/.

Shortlog

9bc543f release: Kata Containers 2.3.0
198e0d1 release: Adapt kata-deploy for 2.3.0
df34e91 osbuilder: fix missing cpio package when building rootfs-initrd image
f61e31c osbuilder: add coreutils to guest rootfs
2667e02 workflows: only allow org members to run /test_kata_deploy
3542cba workflows: Add back the checks for running test-kata-deploy
117b920 kata-deploy: Ensure we test HEAD with /test_kata_deploy
cb7891e tools/osbuilder: build QAT kernel in fedora 34
db9cd10 watcher: tests: ensure there is 20ms delay between fs writes
a51a1f6 watchers: handle symlinked directories, dir removal
5bc1c20 watchers: don't dereference symlinks when copying files
34a1b53 stable-2.3: add VFIO kernel dependencies for ppc64le
8a705f7 runtime: Update containerd to 1.5.8
ac5ab86 qemu: fix snap build by disabling libudev
d22ec59 virtcontainers: fix failing template test on ppc64le
b821511 cgroups: Fix systemd cgroup support
a9d5377 cgroups: pass vhost-vsock device to cgroup
ea83ff1 runtime: remove prefix when cgroups are managed by systemd
f9bde32 workflows: Remove non-used main.yaml
91003c2 versions: bump golang to 1.17.x
5e9b807 release: Use ${GOPATH}/bin/yq for upload-libseccomp-tarball action
de0eea5 release: Kata Containers 2.3.0-rc1
96b66d2 docs: Fix typo
62a51d5 runtime: Revert "runtime: use containerd package instead of cri-containerd"
99c46be release: Kata Containers 2.3.0-rc0
d17100a vendor: update OpenTelemetry to v1.0.0
84ccdd8 vendor: update OpenTelemetry to v0.20.0
9d3ec58 runtime: make sure the "Shutdown" trace span have a correct end
09d5d88 runtime: tracing: Change method for adding tags
bcf3e82 logging: Enable agent debug output for release builds
b468dc5 agent: Use dup3 system call in unit tests of seccomp
1aaa059 agent: "Revert agent: Disable seccomp feature on aarch64 temporarily"
375ad2b runtime: Enhancement for Makefile
a239a38 osbuilder: build image-builder image from Fedora 34
1e331f7 agent: refactor process IO processing
7e40195 agent-ctl: Add stub for AddSwap API
82de838 agent-ctl: Update for Hybrid VSOCK
d1bcf10 forwarder: Remove quotes from socket path in doc
2b13944 docs: Fix outdated links
9b270d7 ci/install_libseccomp: use a temporary work directory
98b4406 ci/install_libseccomp: Fix fail when DESTDIR is set
e66d047 virtcontainers: simplify read-only mount handling
3f21af9 runtime: add fast-test to let test exit on error
17a8c5c runtime: Fix random failure for TestIoCopy
6cc8000 cli: Show available guest protection in env output
2063b13 virtcontainers: Add func AvailableGuestProtections
d45c86d versions: Update CRI-O to its 1.22 release
c4a6426 versions: Update k8s & critools to v1.22
d789b42 package: assign proper value to redefined_string
881b996 agent: Make wording of error message match CRI-O test suite
7a80aeb docs: Moving from EOT to EOF
338ac87 virtcontainers: api: update the functions in the api.md docs
23496f9 release: Upload libseccomp sources with notice to release page
309dae6 virtcontainers: check that both initrd and image are not set
4280415 agent: Fix the configuration sample file
46720c6 runtime: set tags for trace span
c509a20 agent-ctl: Implement Linux OCI spec handling
e610fc8 runtime: Remove comments about unsupported features in config for clh
bdf4824 tools/packaging: Add options for VFIO to guest kernel
42add7f agent: Disable seccomp feature on aarch64 temporarily
5dfedc2 docs: Add explanation about seccomp
45e7c2c static-checks: Add step for installing libseccomp
a3647e3 osbuilder: Set up libseccomp library
3be50ad agent: Add support for Seccomp
b0bc71f ci: test-kata-deploy: Get rid of slash-command-action action
37fa453 osbuilder: Update QAT driver in Dockerfile
a10cfff forwarder: Fix changing log level
6abccb9 forwarder: Drop privileges when using hybrid VSOCK
b67fa9e forwarder: Make explicit root check
e377578 forwarder: Fix docs socket path
d2a7b6f packaging/static-build: s390x fixes
bf00b8d agent-ctl: improve the oci_to_grpc code
5f5eca6 agent: do not return error but print it if task wait failed
5f30633 virtcontainers: delete duplicated notify in watchHypervisor function
a13e2f7 agent: Handle uevent remove actions
57c0f93 agent: fix race condition when test watcher
1a96b8b template: disable template unit test on arm
43b13a4 runtime: DefaultMaxVCPUs should not greater than defaultMaxQemuVCPUs
c59c367 runtime: current vcpu number should be limited
fa92251 runtime: kernel version with '+' as suffix panic in parse
b40eedc rustjail: Consistent coding style of LinuxDevice type
f5172d1 cli: Fix outdated kata-runtime bash completion
34273da runtime/device: Allow VFIO devices to be presented to guest as VFIO devices
68696e0 runtime: Add parameter to constrainGRPCSpec to control VFIO handling
d9e2e9e runtime: Rename constraintGRPCSpec to improve grammar
57ab408 runtime: Introduce "vfio_mode" config variable and annotation
730b9c4 agent/device: Create device nodes for VFIO devices
175f9b0 rustjail: Allow container devices in subdirectories
9891efc rustjail: Correct sanity checks on device path
d6b62c0 rustjail: Change mknod_dev() and bind_dev() to take relative device path
2680c0b rustjail: Provide useful context on device node creation errors
42b92b2 agent/device: Allow container devname to differ from the host
827a41f agent/device: Refactor update_spec_device_list()
8ceadcc agent/device: Sanity check guest IOMMU groups
ff59db7 agent/device: Add function to get IOMMU group for a PCI device
13b06a3 agent/device: Rebind VFIO devices to VFIO driver inside guest
e22bd78 agent/device: Add helper function for binding a guest device to a driver
52268d0 hypervisor: Expose the hypervisor itself
a72bed5 hypervisor: update tests based on createSandbox->CreateVM change
f434bcb hypervisor: createSandbox is CreateVM
76f1ce9 hypervisor: startSandbox is StartVM
fd24a69 hypervisor: waitSandbox is waitVM
a6385c8 hypervisor: stopSandbox is StopVM
f989078 hypervisor: resumeSandbox is ResumeVM
73b4f27 hypervisor: saveSandbox is SaveVM
7308610 hypervisor: pauseSandbox is nothing but PauseVM
8f78e1c hypervisor: The SandboxConsole is the VM's console
4d47aee hypervisor: Export generic interface methods
6baf258 hypervisor: Minimal exports of generic hypervisor internal fields
8030b6c virtcontainers: clh: Re-generate the client code
8296754 versions: Upgrade to Cloud Hypervisor v19.0
4f75ccb docs: use-cases: Update Intel SGX use case
51cbe14 runtime: Add option "disable_seccomp" to config hypervisor.clh
98b7350 virtcontainers: clh: Enable the seccomp feature
b625f62 runtime: delete cri containerd plugin from versions.yaml
09a5e03 docs: Write tracing documentation
4f018b5 runtime: delete useless src/runtime/cli/exit.go
24fff57 snap: make curl commands consistent
2b9f79c snap: add cloud-hypervisor and experimental kernel
50da26d osbuilder: Call detect_rust_version() right before install_rust.sh
b4fadc9 docs: Updating Developer Guide re qemu-img
b8e69ce versions: Add libseccomp and gperf version
e61f5e2 runtime: Show socket path in kata-env output
5b3a349 trace-forwarder: Support Hybrid VSOCK
273a1a9 runtime: optimize test code
76f16fd runtime: use containerd package instead of cri-containerd
6d55b1b docs: use containerd to replace cri-containerd
ed02bc9 packaging: add containerd to versions.yaml
adc9e0b runtime: fix two bugs in rootless hypervisor
4d7ddff utils: kata-manager: Update kata-manager.sh for new containerd config
f34f67d osbuilder: Specify version when installing Rust
135a080 osbuilder: Pass CI env to container agent build
eb5dd76 osbuilder: Re-enable building the agent in Docker
7d0b616 agent: Do not fail when trying to adding existing routes
bcffa26 tracing: Fix typo in "package" tag name
e42bc05 kata-deploy: add .dockerignore file
3f95469 runtime: logging: Add variable for syslog tag
321be0f tracing: Remove trace mode and trace type
8873dda release: Kata Containers 2.3.0-alpha2
f7f6bd0 kata-monitor: add index page
7b2bfd4 virtcontainers: clh: Use 'quiet' as the default kernel parameter
3e24e46 virtcontainers: clh: Turn-off serial and virtio-console by default
176dee6 agent: exec should inherit container process capabilities
a9c2a4b GitHubActions: fix invalid format of require-pr-porting-labels.yaml
2d7b65e agent: flush root span before process finish
7204418 agent/device: Return PCI address from wait_for_pci_device()
e50b05d agent/pci: Add type to represent PCI addresses
8528157 agent/pci: Extend Slot type to represent PCI function as well
c4236cb packaging/kernel: Add CONFIG_PCI_MMCONFIG to x86 guest kernel configuration
5c77cc2 runtime: don't start shim management server in tests
80463b4 qemu: use GitLab repos instead of qemu.org
bf8f582 runtime: optimize code for managing temp users for rootless mode
08360c9 agent: Add an agent configutation file example
8a4e69d agent: rpc: Return UNIMPLEMENTED for not allowed endpoints
0ea2e3a agent: config: Allow for building the configuration from a file
63539dc agent: config: Add allowed endpoints
a953fea agent: config: Simplify configuration creation
b888edc agent: config: Implement Default
762922a runtime: delete func ConstraintsToVCPUs
4f48543 runtime: delete virtcontainers-setup.sh
191d001 vendor: Update containerd to v1.5.7
18bff58 runtime: Optimize func noNeedForOutput and add test cases
7eac2ec protection: add confidential compute frame for arm
8acfc15 check: fix typecheck failure in qemu_arm64_test.go
5b02d54 virtcontainers: fix lint failure on ppc64le
ff9728f virtcontainers: nolint guestProtection
5c138c8 runtime: Fix field alignment on s390x
80f6b97 osbuilder: fixing centos gpg key url for ppc64le
a44cde7 agent: netlink: Use the grpc IP family field when updating the route
71ce6cf runtime: Pass the route IP family to the agent
99450bd agent: protos: Add a Family field to the Route payload
f85fe70 runtime: vendor: Bump the netlink package dependency
e439cec cmd: fix field alignment on ppc64le
e5159ea cmd: get return value for setCPUtype
cd1064b packaging: Configure QEMU with --enable-pie
2ce8d42 clh: Suppress hypervisor output to make guest output visible
13e65f2 cmd: Fix mismatched types in testModuleData
870771d runtime: update .gitignore to ignore monitor_address file
bb99bfb runtime: fix the make check-go-static command error
814cea9 virtcontainers: clean up useless code
907459c agent/device: Don't force PCI rescans
75f426d agent: Simplify do_add_swap()
aad1a87 runtime/device: Give the agent information about VFIO devices
ebd7b61 runtime: Don't repeat GetDeviceByID between appendDevices() and append*()
ad45c52 runtime/device: Record guest PCI path for VFIO devices
5c2af3e runtime/device: Refactor hotplugVFIODevice() to have common exit path
8bc7110 agent/device: Add device type for VFIO devices
f7a2707 agent: Move driver type constants into device.rs
5b1eb08 agent/uevent: Improve logging of wait_for_uevent()
cf36fd8 runtime: Fix some leftover go fmt errors
da42cbc actions: Build experimental kernel on kata-deploy push action
dffc509 kernel: Enable SGX in experimental kernel.
ff6a677 kernel-build: Enable multiple config types.
9004696 experimental-kernel: bump 5.13.10
1fbb730 build: kata-deploy kernel experimental
e5fe53f runtime: fix nil reference in cleanup rootless user
6d94957 kernel: reduce alignment size of memory hotplug to 128M
48090f6 qemu: disable plug on arm64 when pie is added
2304a59 runtime: set the sandbox storage path static
315295e runtime: rename GetSanboxesStoragePath() --> GetSandboxesStoragePath()
4751698 virtcontainers: Fix incorrect scripts path
3b0c4bf runtime: clear virtcontainers cgroup duplicated function
8b0bc1f kata-monitor: bump version to 0.2.0
bfb556d kata-monitor: refresh kata sandbox list on fs events
0e854f3 kata-monitor: improve detection of kata workloads
afad910 kata-monitor: add getSandboxFS()
e38686f runtime: add GetSandboxesStoragePath()
245a12b kata-monitor: improve sandbox caching
fc067d6 kata-monitor: warn when unable to retrive the lower level runtime
53ec4df kata-monitor: minor fixes
57e3712 virtiofs: fix error report in TestVirtiofsdStart when go test running
a525991 workflows: Fix the config file path for using vendored sources
39dcbaa workflows: Fix tag attribution
04139ba release: Kata Containers 2.3.0-alpha1
48fb1d9 virtiofs: Create shared directory with 0700 mode, not 0750
272771d watcher: ensure we create target mount point for storage
439e5ac packaging: fix qemu build on ppc64le
39cd05e runtime: tracing: Use root context to stop tracing
8bbcb06 qemu: Disable SHPC hotplug
cc4983e runtime: Remove unused qemuArchBase.appendBridges definition
e248de4 vendor: Update govmm
3bdcfaa kata-deploy: Add more info about the stable tag
41c590f kata-deploy: Improve README
debf3c9 kata-deploy: Remove qemu-virtiofs runtime class
43a72d7 release: update the kata-deploy yaml files accordingly
ea9b2f9 kata-deploy: Add "stable" info to the README
e541105 kata-deploy: Update the README
9acf4e5 kata-deploy: Add stable yaml files
a86babe kata-deploy: Point to the latest release
a156288 workflows: Add "stable" & "latest" tags to kata-deploy
077b77c runtime: tracing: Fix logger passed in newContainer
bb18cd4 virtcontainers: update VC SandboxConfig API add SandboxBindMounts field
58e77a3 sandbox: Allow the device to be accessed,such as /dev/null and /dev/urandom
0ca8c27 qemu: add v5.1.0 dir under tag_patches
1fe080f threat-model: Add missing threat-model document
305afc8 docs: documentation for running non-root VMM
21c8511 workflows,release: Upload the vendored cargo code
1cfe593 runtime: Run QEMU using a non-root user/group
fd98373 runtime: update .gitignore file cleare the vc shim config
9a6d56f runtime: fix empty cgroup path validation error
90e6388 ci: Call agent shutdown test only in the correspondent CI_JOB
9353cd7 runtime: Remove outdated TestStoreContainer
067c44d runtime: fix UT build failure
9a311a2 docs: fix invalid kernel dax doc url
e7c42fb runtime: unify generated config
4f7cc18 runtime: refactor commandline code directory
a606640 virtcontainers: update VC HypervisorConfig API add three lost fields
d865c80 virtcontainers: add unit tests for container.go
d00decc runtime: clh: Enable hugepages support
9d3cd98 agent/mount: Remove unused ensure_destination_exists()
64aa562 agent: Correct mount point creation
08d7aeb agent/mount: Split out regular file case from ensure_destination_exists()
9fa3bef agent: Remove unnecessary BareMount structure
4928285 agent: Simplify BareMount::mount by using nix::mount::mount
25ac352 versions: Allow newer Rust versions
851d5f8 tests: Correct heading in static checks test
64bb803 runtime/qemu: Move from query-cpus to query-cpus-fast
25670d3 packaging/qemu: Update qemu-exerimental version to v6.1.0
041a513 versions: Update qemu to v6.1.0
81de2d4 packaging: Correct error message in apply_patches.sh
4b7e4a4 runtime: Vendoring update
8d9d6e6 docs: Host cgroups documentation update
9bed2ad virtcontainers: Convert to the new cgroups package API
b42ed39 virtcontainers: cgroups: Add a containerd API based cgroups package
f17752b virtcontainers: container: Do not create and manage container host cgroups
dc7e9bc virtcontainers: sandbox: Host cgroups partitioning
f811026 virtcontainers: Unconditionally create the sandbox cgroup manager
d67a414 src/runtime/README.md: Fix URL of Licence
74d645c how-to: Add how-to-setup-swap-devices-in-guest-kernel.md
2174fee docs: Add swap annotations introduction
f785ff0 virtcontainers: clh: Revert the workaround incorrect default values
0e0e59d virtcontainers: clh: Re-generate the client code
f0b5331 versions: Upgrade to Cloud Hypervisor v18.0
13b8bb0 runtime: Fix README link
1fff9be qemu: remove default config for arm64.
71f915c sandbox: Add device permissions such as /dev/null to cgroup
62baa48 virtcontainers: fc: parse vcpuID correctly
1165213 actions: test make kata-tarball
626d659 actions: kata-deploy on PRs and use makefile
78d99f5 kata-deploy: Make verbose single builds
59486b8 kata-deploy: Add tarball suffix to makefile targets
96e1246 makefile: Include kata-deploy targets
2abc450 test: enable running tests under root user
924a68d osbuilder: Change to "=" operator to make script more portable
d422789 makefile: Fix error exit status code
bfcee91 osbuilder: fix inconsistent calculation of fs size
e2a9e78 virtcontainers: Remove NewStoreFeature
4996f9b snap: Test variable instead of executing "branch"
256c3b2 license: drop redundent license files
bcc9fa3 hotplugAddBlockDevice: Use ExecuteBlockdevAddWithDriverCache with swap
bd85da0 vendor: Update vendor/github.com/kata-containers/govmm
18c95b9 release: Kata Containers 2.3.0-alpha0
8f0f949 tracing: Move dynamically added attributes to Trace()
87de26b tracing: Modify Trace() to accept multiple tag maps
8058e97 tracing: Change runtime tracing tags to vars
0c7789f runtime: Add container field to logs
72e3538 shimv2: add information to method comment
8dadca9 shimv2: add logging to shimv2 api calls
2250360 docs: remove mentioning of qemu-lite
a9de761 runtime: drop qemu-lite support
8ae3edb runtime: fix default hypervisor path
39ffd8e runtime: delete types or const that no longer needed
ff37f5c runtime: Optimize the way slice created
a99fcc3 virtcontainers: simplify tests
932ee41 virtcontainers: clh: Workaround incorrect default values
bff38e4 virtcontainers: clh: Fix the unit test
d967d3c virtcontainers: clh: Use constructors to ensure proper default value
a6a2e52 virtcontainers: clh: Migrate to use the updated client APIs
46eb07e virtcontainers: clh: Re-generate the client code
80fba4d virtcontainers: clh: Upgrade to the openapi-generator v5.2.1
938981b build_image: Fix error soft link about initrd.img
b8717f3 ci: Temporarily skip agent shutdown test on s390x
87bbae1 fc: fix version parsing for fc >= 0.25
9de1129 osbuilder: Fix rootfs-builder when running in VMs
65a1e13 osbuilder: Allow running the tool several times
a421473 osbuilder: Fix Makefile
2304f93 docs: update the GoDoc url from kata 1.x to 2.x
2a61457 docs: update how-to README file for Firecracker config
8594f80 ci/openshift-ci: Pull centos from registry.centos.org
486baba docs: update containerd CRI plugin url

Compatibility with CRI-O

Kata Containers 2.3.0 is compatible with CRI-O

Compatibility with containerd

Kata Containers 2.3.0 is compatible with contaienrd v1.5.2

OCI Runtime Specification

Kata Containers 2.3.0 support the OCI Runtime Specification v1.0.0-rc5

Compatibility with Kubernetes

Kata Containers 2.3.0 is compatible with Kubernetes 1.22.0-00

Libseccomp Notices

The binaries inside the Kata Containers images provided with this release are
statically linked with the following GNU LGPL-2.1 licensed libseccomp library.

The uses the libseccomp v2.5.1 which is not modified from the upstream version.
However, in order to comply with the LGPL-2.1 (§6(a)), we attach the complete source code for the library.

If you want to use the which is not statically linked with the library, you can build
a custom that does not use the library from sources.
For the details, please check the developer guide.

Kata Linux Containers image

Agent version: 2.3.0

Default Image Guest OS:

description: |
Root filesystem disk image used to boot the guest virtual
machine.
url: "https://github.com/kata-containers/kata-containers/tools/osbuilder"
architecture:
aarch64:
name: "ubuntu"
version: "latest"
ppc64le:
name: "centos"
version: "latest"
s390x:
name: "ubuntu"
version: "latest"
x86_64:
name: "clearlinux"
version: "latest"
meta:
image-type: "clearlinux"

Default Initrd Guest OS:

description: |
Root filesystem initrd used to boot the guest virtual
machine.
url: "https://github.com/kata-containers/kata-containers/tools/osbuilder"
architecture:
aarch64:
name: "alpine"
version: "3.13.5"
ppc64le:
name: "alpine"
version: "3.13.5"
s390x:
name: "alpine"
version: "3.13.5"
x86_64:
name: "alpine"
version: "3.13.5"

Kata Linux Containers Kernel

Kata Containers 2.3.0 suggest to use the Linux kernel v5.10.25
See the kernel suggested Guest Kernel patches
See the kernel suggested Guest Kernel config

Installation

Follow the Kata installation instructions.

Issues & limitations

More information Limitations