systemd 220+ doesn't work anymore under OpenVZ container

[drizzt]

systemd 220+ doesn't work anymore under OpenVZ container.
It returns the following error on "virtual console" and it hangs:

Failed to mount sysfs at /sys: Device or resource busy
Failed to mount proc at /proc: Device or resource busy
[!!!!!!] Failed to mount API filesystems, freezing.

Obviously I'm available for any tests or other logs.

[drizzt]

After a bisection I found that 27cc6f1 is the first bad commit

[haraldh]

What is mounted on /sys and /proc before systemd stumbles over it?

Maybe the mount options differ?

[haraldh]

Can you turn on debugging?

[haraldh]

btw, that code has been modified just recently... so please recheck with git master.

[drizzt]

I tried with v221 and I have the same problem.
I'll re-try with currently git master with LOG_DEBUG enabled and I'll add logs.

[drizzt]

Mounting sysfs to /sys of type sysfs with options n/a.
Failed to mount sysfs at /sys: Device or resource busy
Mounting proc to /proc of type proc with options n/a.
Failed to mount proc at /proc: Device or resource busy
Mounting devtmpfs to /dev of type devtmpfs with options mode=755.
Mounting tmpfs to /dev/shm of type tmpfs with options mode=1777.
Mounting devpts to /dev/pts of type devpts with options mode=620,gid=5.
Mounting tmpfs to /run of type tmpfs with options mode=755.
Mounting tmpfs to /sys/fs/cgroup of type tmpfs with options mode=755.
Mounting cgroup to /sys/fs/cgroup/systemd of type cgroup with options none,name=systemd,xattr.
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
Mounting cgroup to /sys/fs/cgroup/systemd of type cgroup with options none,name=systemd.
Mounting kdbusfs to /sys/fs/kdbus of type kdbusfs with options n/a.
Failed to mount kdbusfs at /sys/fs/kdbus: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

[drizzt]

With the last version that works (918315e), built with debugging, it doesn't try to mount /sys and /proc.
So it seems that path_is_mount_point was returning > 0 and so mount_one returns 0 correctly.
With the new implentation, path_is_mount_point returns 0 and so mount_one tries to mount /sys and /proc

Mounting devtmpfs to /dev of type devtmpfs with options mode=755.
Mounting tmpfs to /dev/shm of type tmpfs with options mode=1777.
Mounting devpts to /dev/pts of type devpts with options mode=620,gid=5.
Mounting tmpfs to /run of type tmpfs with options mode=755.
Mounting tmpfs to /sys/fs/cgroup of type tmpfs with options mode=755.
Mounting cgroup to /sys/fs/cgroup/systemd of type cgroup with options none,name=systemd,xattr.
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
Mounting cgroup to /sys/fs/cgroup/systemd of type cgroup with options none,name=systemd.
Mounting cgroup to /sys/fs/cgroup/cpu,cpuacct of type cgroup with options cpu,cpuacct.
Failed to mount cgroup at /sys/fs/cgroup/cpu,cpuacct: Invalid argument
Mounting cgroup to /sys/fs/cgroup/memory of type cgroup with options memory.
Mounting cgroup to /sys/fs/cgroup/net_cls,net_prio of type cgroup with options net_cls,net_prio.
Failed to mount cgroup at /sys/fs/cgroup/net_cls,net_prio: Invalid argument
Mounting cgroup to /sys/fs/cgroup/perf_event of type cgroup with options perf_event.
Failed to mount cgroup at /sys/fs/cgroup/perf_event: Invalid argument
Mounting cgroup to /sys/fs/cgroup/blkio of type cgroup with options blkio.
Mounting cgroup to /sys/fs/cgroup/cpuset of type cgroup with options cpuset.
Failed to mount cgroup at /sys/fs/cgroup/cpuset: Invalid argument
Mounting cgroup to /sys/fs/cgroup/freezer of type cgroup with options freezer.
Failed to mount cgroup at /sys/fs/cgroup/freezer: Invalid argument
Mounting cgroup to /sys/fs/cgroup/devices of type cgroup with options devices.
Failed to mount cgroup at /sys/fs/cgroup/devices: Invalid argument
systemd 219 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN)

[arvidjaar]

With the new implentation, path_is_mount_point returns 0 and so mount_one tries to mount /sys and /proc

Well, you are the only one able to debug it. Add printf's to path_is_mount_point to see at which point it makes wrong decision.

[BotoX]

+1 I am affected by this bug.

[Killed one of my Archlinux OpenVZ containers by upgrading the system to the latest version of systemd where this bug is still persistent. (That makes this bug the 3rd systemd related bug that killed one of my non-production servers.)]

[drizzt]

Sorry for the late reply, but I'm on vacation:

path_is_mount_point: /sys 1024
name_to_handle_at1: sys Operation not supported
name_to_handle_at2: sys No such file or directory
Mounting sysfs to /sys of type sysfs with options n/a.
Failed to mount sysfs at /sys: Device or resource busy

In fd_is_mount_point: First call of name_to_handle_at returns ENOTSUP/EOPNOTSUPP (I think it's because it's an old kernel), secondo call of name_to_handle_at2 returns ENOENT, so the function returns -ENOENT

[poettering]

Sounds like a kernel/OpenVZ bug to me.

The first name_to_handle_at() call is invoked with an fd of the root directory for the path "sys". The second name_to_handle_at() call is invoked on the same fd, but for the path "" and the AT_EMPTY_FLAG flag is set.

According to the man page ENOENT means: "pathname is an empty string, but AT_EMPTY_PATH was not specified in flags." Which doesn't apply here, because we do set AT_EMPTY_FLAG.

This works fine on other container managers (such as nspawn). I presume OpenVZ still needs some special kernel patches, no? Somehow they appear to break name_to_handle_at(). Please report this issue to OpenVZ hence.

I don't think there's anything to fix in systemd here. Closing hence.

[drizzt]

Opened upstream: https://bugzilla.openvz.org/show_bug.cgi?id=3280