yum install, transation freeze with containerd.io 1.5.11

[arpadmuller]

Description

I got got a freeze when I am trying to build centos 7 image.
My containerd.io version is:

containerd.io-1.5.11-3.1.fc34.x86_64.rpm

OS: Fedora 34
Kernel:Linux 5.16.15-101.fc34.x86_64

When I am trying to install packages (for example mc) inside the runnning container, yum stuck on transations.
One of my cpu thread at this time is working on 100%

Steps to reproduce the issue

Dockerfile:
FROM centos:7
RUN yum install -y mc

Step 1:
docker build -t test .

Describe the results you received and expected

This is where the docker stuck:

Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : gpm-libs-1.20.7-6.el7.x86_64 1/31
.......

What version of containerd are you using?

1.5.11

Any other relevant information

runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.5.3

uname -a:
Linux *** 5.16.15-101.fc34.x86_64 #1 SMP PREEMPT Thu Mar 17 05:50:00 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

[AkihiroSuda]

Unlikely to be an issue of containerd.
What is your Moby storage driver? (Shown in docker info)
With BuildKit or without?

[arpadmuller]

My store driver is ZFS
After I downgrade the containerd to 1.4.9, the problem disappeared.

Docker info:
`Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
scan: Docker Scan (Docker Inc., v0.17.0)

Server:
Containers: 3
Running: 1
Paused: 0
Stopped: 2
Images: 10
Server Version: 20.10.14
Storage Driver: zfs
Zpool: data
Zpool Health: ONLINE
Parent Dataset: data/docker
Space Used By Parent: 1096740864
Space Available: 198852698112
Parent Quota: no
Compression: lz4
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210f
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.16.16-100.fc34.x86_64
Operating System: Fedora 34 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.11GiB
Name: t480
ID: KITH:JKXD:Q54G:VTGQ:AX65:46R5:EQ7X:QSZO:VKDR:3253:KMB3:WNVW
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
`

[pnyheim]

Same problem here.
Fedora 35 with containerd.io 1.5.11 (also tried 1.5.10)
downgrading to 1.4.13 solved the issue for me

Steps I used to reproduce:

docker run -it centos:7 bash

and then inside container

yum install wget -y

the yum transaction takes almost 3 minutes to complete - whereas it usually is just a few seconds.

Strangely, I also tried the same with other images such as almalinux:8 and fedora:35 - but only experience the problem with centos:7

My storage driver is btrfs

[gjvoosten]

Yup, same for me on Fedora 35 with containerd.io 1.5.11; any yum install in a CentOS 7 container takes seemingly forever, keeping a CPU 100% busy all the time. Based on @pnyheim's comment I downgraded to 1.4.13 (and rebooted just to be sure; didn't even bother with 1.5.10), and then it was again superfast.

[aleitao]

Hi,
I think the change that is causing this issue is the value of LimitNOFILE:
cat /usr/lib/systemd/system/containerd.service | grep LimitNOFILE
changed between versions:
1.4.13: LimitNOFILE=1048576
1.5.11: LimitNOFILE=infinity

Changing the service file to the old value or alternatively creating a new file in /etc/systemd/system/containerd.service.d/override.conf with:
[Service]
LimitNOFILE=1048576

It may not be a perfect solution but it is a workaround that is working for me. (Personally i created the override.conf file to avoid messing with the service file that can be overwritten with an update)

fedora 35
containerd.io 1.5.11

[gjvoosten]

Changing the service file to the old value or alternatively creating a new file in /etc/systemd/system/containerd.service.d/override.conf with:
[Service]
LimitNOFILE=1048576

I can confirm that @aleitao's work-around works for me too. So contrary too @AkihiroSuda's remark, it does appear to be caused by changes in containerd.io.

[thaJeztah]

could someone check if there's a difference in output of ulimit -a (or ulimit -n) within the container if LimitNOFILE=1048576 is set or LimitNOFILE= infinity is set?

This change was made in #4475, but according to the systemd docs I referenced in that PR, unlimited should have the same effect as the previously hardcoded value, but possibly newer versions of systemd changed that?

Here's on a Ubuntu machine with containerd.io 1.5.11 installed;

containerd --version
containerd containerd.io 1.5.11 3df54a852345ae127d1fa3092b95168e4a88e2f8

systemctl cat containerd.service | grep LimitNOFILE
LimitNOFILE=infinity

docker run --rm centos:7 sh -c 'ulimit -n'
1048576

docker run --rm centos:7 sh -c 'ulimit -a'

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3838
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[thaJeztah]

FWIW, I think it's fine to revert that change, but it'd be good if we knew what changed (this could be a bug/regression in systemd, or a change in behaviour)

[aleitao]

containerd --version
containerd containerd.io 1.5.11 3df54a852345ae127d1fa3092b95168e4a88e2f8

Without overriding configuration:

systemctl cat containerd.service | grep LimitNOFILE
LimitNOFILE=infinity

docker run --rm centos:7 sh -c 'ulimit -n'
1073741816

docker run --rm centos:7 sh -c 'ulimit -a'
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63109
max locked memory       (kbytes, -l) 8192
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1073741816
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Overriding configuration in /etc/systemd/system/containerd.service.d/override.conf

systemctl cat containerd.service | grep LimitNOFILE                                                        
LimitNOFILE=infinity
LimitNOFILE=1048576

docker run --rm centos:7 sh -c 'ulimit -n'                                                                               
1048576

docker run --rm centos:7 sh -c 'ulimit -a'                                                                               
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63109
max locked memory       (kbytes, -l) 8192
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited