New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Low memory utilisation with high number of memory-constrained workloads #1695
Comments
@fidencio Added the kata configuration to the description. |
It seems kata can only use half of Host memory.
so it seems overhead and cgroup limits are configured ok, and process is killed inside VM guest. VM by itself continue to work.
And being dropped out of console. Pod/VM restarts Host logs
The same |
Retested on bundled |
Also, it works on |
Summary
Trying deployments with increasing numbers of pods, I came across one scenario where Kata cannot utilize more than half the memory available in the cluster, whereas
runc
can take advantage of a much larger fraction.Workload
The workload is a special simple workload intended to consume a configurable amount of memory and CPU. The deployment starts with the following yaml file:
A script then scales the deployment. The cluster has 5 worker nodes with a total of 80G available memory.
Observations
Without the memory request and limit, I observed the following as the number of containers increased:
With the memory request and limit, but without the
runtimeClass
(i.e. running withrunc
), I observe a similar pattern:With kata, on the other hand, the memory utilization stays well below 50% of the memory of the cluster
Interestingly, the behavior does not seem to depend on the size of the limit. For example, with 512M request and 1G limit we get:
With 1G limit and 2G request we get:
Expectations
The expectation is that Kata would be able to use a larger fraction of the memory of the cluster.
Other relevant details
The
runtimeClass
contained the following Pod overhead during the testing:Kata configuration
[root@worker-0-0 core]# /usr/bin/kata-collect-data.sh
Show kata-collect-data.sh details
Meta details
Running
kata-collect-data.sh
version2.0.0 (commit 3e7173828955b61d9b2bf6e0f59712fb9a5c8b98)
at2021-04-15.13:55:34.798247454+0000
.Runtime
Runtime is
/bin/kata-runtime
.kata-env
/bin/kata-runtime kata-env
Runtime config files
Runtime config files
Runtime default config files
Runtime config file contents
cat "/etc/kata-containers/configuration.toml"
cat "/usr/share/kata-containers/defaults/configuration.toml"
Containerd shim v2
Containerd shim v2 is
/bin/containerd-shim-kata-v2
.containerd-shim-kata-v2 --version
KSM throttler
KSM throttler
version
systemd service
Image details
Image details
No image
Initrd details
Initrd details
unknown
Logfiles
Logfiles
Runtime logs
Runtime logs
No recent runtime problems found in system journal.
Throttler logs
Throttler logs
No recent throttler problems found in system journal.
Kata Containerd Shim v2 logs
Kata Containerd Shim v2
Recent problems found in system journal:
Container manager details
Container manager details
Kubernetes
Kubernetes
kubectl version
kubectl config view
systemctl show kubelet
crio
crio
crio --version
systemctl show crio
crio config
Podman
Podman
podman --version
podman system info
cat /etc/containers/registries.conf
cat /etc/containers/storage.conf
cat /etc/containers/policy.json
cat /usr/share/containers/containers.conf
cat /usr/share/containers/mounts.conf
cat /usr/share/containers/seccomp.json
Packages
Packages
No
dpkg
Have
rpm
rpm -qa|egrep "(cc-oci-runtime|cc-runtime|runv|kata-runtime|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"
Kata Monitor
Kata Monitor
kata-monitor
.kata-monitor --version
The text was updated successfully, but these errors were encountered: