Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can't enable unified cgroup hierarchy on centos 7 #19760

Closed
unixisevil opened this issue May 29, 2021 · 6 comments
Closed

can't enable unified cgroup hierarchy on centos 7 #19760

unixisevil opened this issue May 29, 2021 · 6 comments

Comments

@unixisevil
Copy link

systemd version the issue has been seen with

repo: https://github.com/systemd-rhel/rhel-9 version: v247 commit: 4d484e1

Used distribution

CentOS Linux 7

 systemctl --version
systemd 247 (247)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -ZSTD -SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN -PCRE2 default-hierarchy=unified

Linux kernel version used (cat /proc/cmdline)

BOOT_IMAGE=/vmlinuz-5.12.7-1.el7.elrepo.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

can't see typical cgroup v2 interface file cgroup.*

on ubuntu 20.04 after enable cgroup v2: (ls /sys/fs/cgroup/cgroup.*)

/sys/fs/cgroup/cgroup.controllers      /sys/fs/cgroup/cgroup.procs            /sys/fs/cgroup/cgroup.threads
/sys/fs/cgroup/cgroup.max.depth        /sys/fs/cgroup/cgroup.stat
/sys/fs/cgroup/cgroup.max.descendants  /sys/fs/cgroup/cgroup.subtree_control

mount -l|grep cgroup:

cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

Unexpected behaviour you saw

i can see cgroup v1 directory structure

ls -l /sys/fs/cgroup/:

total 0
drwxr-xr-x.  2 root root 40 5月  29 03:28 blkio
drwxr-xr-x.  2 root root 40 5月  29 03:28 cpu,cpuacct
drwxr-xr-x.  2 root root 40 5月  29 03:28 cpuset
drwxr-xr-x.  2 root root 40 5月  29 03:28 devices
drwxr-xr-x.  2 root root 40 5月  29 03:28 freezer
drwxr-xr-x.  2 root root 40 5月  29 03:28 hugetlb
drwxr-xr-x.  2 root root 40 5月  29 03:28 memory
drwxr-xr-x.  2 root root 40 5月  29 03:28 net_cls,net_prio
drwxr-xr-x.  2 root root 40 5月  29 03:28 perf_event
drwxr-xr-x.  2 root root 40 5月  29 03:28 pids
drwxr-xr-x.  2 root root 40 5月  29 03:28 rdma
dr-xr-xr-x. 10 root root  0 5月  29 03:28 systemd

mount -l|grep cgroup :

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

Steps to reproduce the problem

manual upgrade on centos 7 from systemd-219 to systemd-247

Additional program output to the terminal or log subsystem illustrating the issue
full startup log

@poettering
Copy link
Member

Yout boot logs show that your initrd runs systemd 219. The initrd — which is where systemd is first started and thus cgroupfs mounted – decides which cgroup setup is used.

Consider rebuilding the initrd, so that it contains systemd 247.

@unixisevil
Copy link
Author

@poettering , yay , it works, i should upgrade systemd first, then kernel

@carlosabalde
Copy link

Sharing a few notes just in case they are helpful for somebody else:

  • The systemd >= 247 requirement described in https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-430 is unclear to me. Distributions like Ubuntu Bionic, Debian Buster or RHEL 8 ship older versions (237, 241 and 239 respectivelly) and they are working -apparently- just fine with Docker Desktop >= 4.3.0 (using --cgroupns=host, mapping -v /sys/fs/cgroup:/sys/fs/cgroup:rw as rw instead of ro, and continue using --privileged). Any specific reason to require 247?

  • CentOS 7 ships systemd 219. Using the backport described in https://maciej.lasyk.info/2016/Dec/16/systemd-231-latest-in-centos-7-thx-to-facebook/ is possible to upgrade to 234 in a quick and simple way. Once upgraded, the cgroupsv2 issue is fixed for CentOS 7 and Docker Desktop >= 4.3.0.

  • Same problem with Ubuntu Xenial (ships systemd 229), but I didn't find any backport making life easy.

@jacorvar
Copy link

Hi,

I'm still experiencing this issue. I updated as root both the kernel and systemd (as suggested here), in that order. Here are the current specs of my system:

$ hostnamectl 
   Static hostname: nodo02
         Icon name: computer-server
           Chassis: server
        Machine ID: 30fa5292f39f4ed8a79ddf233049e96b
           Boot ID: 5f19d5729ac44667839e980ead62db44
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 5.4.242-1.el7.elrepo.x86_64
      Architecture: x86-64

$ systemd 234
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

Then I added cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 to the contents of the GRUB_CMDLINE_LINUX variable and rebuilt the grub as follows:

$ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.242-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-5.4.242-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.90.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.90.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.15.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.15.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1062.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1062.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-30fa5292f39f4ed8a79ddf233049e96b
Found initrd image: /boot/initramfs-0-rescue-30fa5292f39f4ed8a79ddf233049e96b.img
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
logger: socket /dev/log: Connection refused
done

The logger: socket /dev/log: Connection refused does not appear any longer after doing the following:

$ rm /dev/log
$ ln -fs /run/systemd/journal/dev-log /dev/log

After reboot I can't find cgroup v2 enabled:

$ mount -l | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

I also got identical results with the following:

$ grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"

Any clues of what I might be missing?

Thanks

@leonardotrp
Copy link

Compartilhando algumas notas, caso sejam úteis para outra pessoa:

  • O systemdrequisito >= 247 descrito em https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-430 não está claro para mim. Distribuições como Ubuntu Bionic, Debian Buster ou RHEL 8 enviam versões mais antigas (237, 241 e 239 respectivamente) e estão funcionando - aparentemente - muito bem com Docker Desktop >= 4.3.0 (usando, mapeando como em vez --cgroupns=hostde -v /sys/fs/cgroup:/sys/fs/cgroup:rwe rwcontinue rousando --privileged) . Algum motivo específico para exigir 247?
  • O CentOS 7 vem com o systemd 219. Usando o backport descrito em https://maciej.lasyk.info/2016/Dec/16/systemd-231-latest-in-centos-7-thx-to-facebook/ é possível atualizar para 234 de forma rápida e simples. Após a atualização, o cgroupsv2problema foi corrigido para CentOS 7 e Docker Desktop >= 4.3.0.
  • O mesmo problema com o Ubuntu Xenial (fornecido com o systemd 229), mas não encontrei nenhum backport que facilitasse a vida.

@carlosabalde It worked!

Something changed in the latest versions of Docker that 'broke' a solution we had implemented in the past.

In the new solution below I even referenced your comment. Thank you very much!

FROM centos:7
ENV container docker

# https://github.com/systemd/systemd/issues/19760#issuecomment-992675467
RUN curl https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/repo/epel-7/jsynacek-systemd-backports-for-centos-7-epel-7.repo > /etc/yum.repos.d/jsynacek-systemd-centos-7.repo
RUN cat /etc/yum.repos.d/jsynacek-systemd-centos-7.repo
RUN yum -y update

@rafaelpirolla
Copy link

rafaelpirolla commented Jan 15, 2024

I fixed this in the past by adding cgroup_no_v1=all to /Applications/Docker.app/Contents/Resources/linuxkit/cmdline on MacOs... Latest version of Docker doesn't have this anymore...

Got systemd 219 (amazon linux 2) working just using a local directory on mac as a volume to /sys/fs/systemd:

docker run --rm -it --privileged --cgroupns host -v /tmp/systemd:/sys/fs/cgroup:rw my.systemd.image

I would need to study cgroup v1 and v2 more to say that this is safe. I'm just using it as a test container for ansible molecule so it's fine for my use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

6 participants