New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-tmpfiles failing in updated lx zone #331

citrus-it opened this Issue Nov 23, 2018 · 2 comments


None yet
1 participant
Copy link

citrus-it commented Nov 23, 2018

A customer reported that after updating their ubuntu lx zone several services failed to start after reboot, including sshd.

The problem is that the systemd-tmpfiles service fails to start properly and so several temporary directories under /var/run are not created after /run is mounted on tmpfs.

root@ubuntu-14-04-b:~# systemctl status systemd-tmpfiles-setup.service
 systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; v
   Active: failed (Result: exit-code) since Thu 2018-11-22 12:00:51 UTC; 3min 47
     Docs: man:tmpfiles.d(5)
  Process: 28706 ExecStart=/bin/systemd-tmpfiles --create --remove --boot --excl
 Main PID: 28706 (code=exited, status=1/FAILURE)

The service log shows several instances of:

Failed to validate path /var/run/sshd: Too many levels of symbolic links

and strace shows:

openat(3, "var", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
openat(4, "run", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = -1 ELOOP (Too many levels of symbolic links)

/var/run is a symbolic link to /run

The linux man page for openat(2) states:

              If pathname is a symbolic link and the O_NOFOLLOW flag is also
              specified, then the call returns a file descriptor referring
              to the symbolic link.  This file descriptor can be used as the
              dirfd argument in calls to fchownat(2), fstatat(2), linkat(2),
              and readlinkat(2) with an empty pathname to have the calls
              operate on the symbolic link.

Since the openat call is opening the /var/run symlink with O_PATH|O_NOFOLLOW, it should succeed under lx. Newer systemd (after systemd/systemd@addc3e3) expects this.

@citrus-it citrus-it self-assigned this Nov 23, 2018

@citrus-it citrus-it added the bug label Nov 23, 2018


This comment has been minimized.

Copy link

citrus-it commented Nov 23, 2018

citrus-it@341b374 is a temporary workaround pending further investigation. This modifies lx_openat() so that a file descriptor is returned in this condition. Unlike true linux, the file descriptor points to the link target but has been modified as a result of the O_PATH flag so that it cannot be used for reading and writing. This is enough to satisfy systemd's chase_symlinks() check and resolve the immediate problem.

With this fix in place, if the returned file descriptor is used for the four functions mentioned in the Linux man page, then:

  • fchownat(2) succeeds (but affects the link target)
  • fstatat(2) succeeds (but reads the link target)
  • linkat(2) fails with ENOENT;
  • readlinkat(2) fails with ENOENT.

This comment has been minimized.

Copy link

citrus-it commented Nov 28, 2018

A hotfix for r151028 is available for this issue:

pkg apply-hot-fix

Since this updates a kernel driver a new boot environment is required. To control the name of that boot environment, add --bename=<name> before the URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment