"systemctl list-unit-files" doesn't work

[edolstra]

It fails as follows:

$ systemctl list-unit-files                         
Failed to issue method call: Too many levels of symbolic links

Strace shows that this is because /etc/systemd/system is a symlink, and systemd doesn't expect that:

open("/etc/systemd/system", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links)

(Note the O_NOFOLLOW.)

[shlevy]

Can we just patch systemd not to put the O_NOFOLLOW there? Or what's the point of it?

[wmertens]

So I think this is from http://cgit.freedesktop.org/systemd/systemd/tree/src/shared/install.c#n485 (in the find_symlinks function). There's a bunch of places where O_NOFOLLOW is done, it seems to me that removing it here could not fix things entirely.

It uses SYSTEM_CONFIG_UNIT_PATH to know where to look (via get_config_path), which the Makefile sets to $(pkgsysconfdir)/system. That could be set to /etc/static/ but then the whole configuration lives at /etc/static/systemd and not /etc/systemd which is unexpected.

Another option would be to dereference /etc/systemd/system, making it a real dir and copying the symlinks. Then it's a real directory and systemd should work unchanged.

One last approach is to symlink /etc/systemd elsewhere. Not sure which is preferable when system configs are rebuilt.

[MP2E]

I'd like to note that this is still happening on the latest git master as of 07/31/2014

EDIT: I can't seem to remove some systemd services because of this bug...

[CMCDragonkai]

Will this get fixed inside systemd or patched at NixOS?

[domenkozar]

See https://bugzilla.redhat.com/show_bug.cgi?id=955379#c14

[Profpatsch]

Still doesn’t work. Can this be patched? Is there an alternative?

[lucabrunox]

I think we can patch systemd for reading, should be safe. Lennart talks about enable/disable which creates/removes symlinks and may be unsafe in case of suspicious symlinks. But for listing, I think we're safe to patch it and even propose upstream.

[abbradar]

I think systemctl is-enabled doesn't work because of this too -- see the issue referenced above.

[musteresel]

Are there any news on this? Any suggestions on how to fix it? Would you accept a patch with a patch to apply during the build of systemd?

[Profpatsch]

@musteresel Yes, we’d be happy to review and accept a patch for that if you want to contribute it.

[DamienCassou]

I can't reproduce:

$ systemctl list-unit-files
UNIT FILE                                   STATE 
dev-hugepages.mount                         bad   
dev-mqueue.mount                            bad   
proc-sys-fs-binfmt_misc.mount               bad   
sys-fs-fuse-connections.mount               bad
[...]

$ systemctl --user list-unit-files
UNIT FILE            STATE 
emacs.service        bad   
offlineimap.service  static
ssh-agent.service    masked
systemd-exit.service static
basic.target         static
default.target       static
[...]

I just find it strange that nearly all units are marked as bad

[bjornfor]

It's apparently fixed in master, but not in release-15.09 (just checked it).

[CMCDragonkai]

What was the commit that fixed it?

[abbradar]

It's still not fixed (if I guess correctly that what I see is the same bug):

> systemctl is-enabled tlp
Failed to get unit file state for tlp.service: Too many levels of symbolic links

[mjhoy]

I also see all unit files listed as in a "bad" state. Is anyone working on this?

I tracked the code down to https://github.com/NixOS/systemd/blob/nixos-v228/src/shared/install.c#L629 and if I remove the O_NOFOLLOW there, the command seems to work, at least when i run

$ SYSTEMCTL_INSTALL_CLIENT_SIDE=1 ./systemctl list-unit-files

i have not tested it in a full install.

(edit: is-enabled appears to work as well)

[mjhoy]

and perhaps this systemd commit is helpful explaining why the NOFOLLOWs:

systemd/systemd@0ec0dea

[abbradar]

@mjhoy That's great! Maybe open a PR to NixOS/systemd?

[mjhoy]

@abbradar yeah... I feel I don't quite understand the design of both systemd and nixos well enough yet, but I will try to spend some time researching it. For instance poettering says explicitly "we'd never read install information through symlinks" in that commit I linked to above which seems to rule this out? At yet symlinks under /usr are ok? Also... I don't know how config_path (which is what I'm removing the NOFOLLOW from) fits into this logic.

[abbradar]

If I understand correctly, symlinks were not allowed because they are ought to be added by systemctl enable/disable, so they keep track of enabled units by checking if they are symlinks. But we don't use enable/disable at all on NixOS, so the should be file.

That being said, I don't know much about internals of systemd and judge only by this commit message, so these thoughts are very potentially wrong.

[mjhoy]

I'm now running systemd (v228, current on unstable) with this tiny change mjhoy/systemd@5e45c1d. It seems to fix both list-unit-files and is-enabled. When I tested it from within a container it did not work, but that could be because the container uses the old systemd (is this possible?)

That said I hardly do anything fancy with systemd, nor do I understand the system very well. If you would like to try the patch yourself you can plug this into configuration.nix:

systemd.package = (pkgs.lib.overrideDerivation pkgs.systemd (attrs: {
  src = pkgs.fetchFromGitHub {
    owner = "mjhoy";
    repo  = "systemd";
    rev   = "5e45c1da38ba5f9698ab1302332b8a7cf64e3b99";
    sha256 = "0f02dd4jzynycf4ixbqn4rhwxvy592msqavvhafkd5a3dw0wh9vb";
  };
}));

I can submit a PR to nixos/systemd if people think this makes sense. I fully expect that I'm missing something big, though, and this is a bad patch.

[Mathnerd314]

It looks fine to me; the code is a big blob with no documentation, and this is a minimal change for an edge case. I would submit it as a PR to the main systemd repository, to see if Lennart has an opinion on whether /etc/systemd/system can be a symlink.

[CMCDragonkai]

That's a really simple change. Definitely try a PR at systemd, and if they don't accept it, it surely can be made as a patch in NixOS.

[Profpatsch]

It does work for me on the current master.

[jgillich]

@Profpatsch Did you try systemctl is-enabled? Still doesn't work for me.

[Profpatsch]

@jgillich You are right, missed that one.

[rimmington]

@mjhoy Is there a PR for systemd I can watch?

[mjhoy]

@rimmington i sent a query in to their mailing list but didn't hear back from anyone. i'm not comfortable submitting a PR as i don't think i really understand the design. someone else who knows more should!

[groxxda]

Instead of patching systemd we could bindmount /etc/systemd/{system,user} to its respective targets. This fixes is-enabled and list-unit-files for me.

Any objections?
CC @edolstra @lethalman

[edolstra]

Please no. That sounds very hacky to me.

[rimmington]

@mjhoy I've opened a PR with systemd. I've kept the commit author as you; let me know if you're not happy/comfortable with that.

[mjhoy]

just FYI, looks like they merged this into master.

[abbradar]

Fixed on master.