nixos: make symlinks in `/etc` relative (except `/etc/static`) #54980

danbst · 2019-01-31T08:02:06Z

Motivation for this change

Main motivation is to fix extra-container, so it can run declarative NixOS containers on non-NixOS (erikarvstedt/extra-container#1 (comment)).

But seriously, there is a real problem with symlinking stuff currently. Let me show an example. Here's a simple container, where we override NixOS version.

  containers.custom = {
    autoStart = true;
    config = {
      system.nixos.version = "CUSTOM";
    };
  };

Now let's see, if that works:

### good
$ sudo nixos-container run custom -- cat /etc/os-release | grep VERSION=
VERSION="CUSTOM (Koi)"

### bad
$ sudo cat /var/lib/containers/custom/etc/os-release | grep VERSION=
VERSION="19.03.git.0652f62 (Koi)"

Indeed, in second case /var/lib/containers/custom/etc/os-release points to /etc/static/os-release of the host system, not container! This is wrong, but even worse happens on non-NixOS, where there is no /etc/static on the host - container can be created and started, but can't be rebooted.

This builds on top of #35364, however, I am not fixing the original problem (allow machinectl pull-tar to run NixOS containers). I'll make a separate PR, which builds on top of this (and #35364) and makes /etc relative symlink for container tarballs.

cc @edolstra
cc @florianjacob as original author
cc @Ekleog @7c6f434c @erikarvstedt as potential interested parties

Things done

This makes symlinks relative in host's /etc too. Rebuild-switch is handled correctly, and my machine survives multiple reboots. Also, simple installer NixOS test passes.

so that the links can be followed if the NixOS installation is not mounted as filesystem root. In particular, this makes /etc/os-release adhere to the standard: https://www.freedesktop.org/software/systemd/man/os-release.html Fixes NixOS#28833.

7c6f434c · 2019-01-31T08:58:23Z

Looks good to me.

Not immediately useful to me as 1) I want read-only /etc with symlinks to mutable stuff and atomic switching — and I have it — and this change is only useful as long as you have /etc/static because of not fully atomic /etc; 2) I don't want to follow some of the systemd rules — and I don't run systemd — so I will only use NixOS containers once it is clear how to run them with systemd only in the container (I am OK with systemd, DBus and PulseAudio as long as I have full control over scoping)

eadwu · 2019-02-22T00:29:46Z

Pretty sure it's this PR, but after upgrading this morning, I've been getting boot warnings on stage-2-init about symlinks, though it doesn't seem to break anything.

One such example:

Feb 22 08:09:38 nixos unknown: booting system configuration /nix/store/gsaml9lnkc0lg87253rkyhwjb3074jjj-nixos-system-nixos-19.03.git.ea44b55
Feb 22 08:09:38 nixos stage-2-init: running activation script...
Feb 22 08:09:38 nixos stage-2-init: setting up /etc...
Feb 22 08:09:38 nixos stage-2-init: /etc/fonts directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/dbus-1 directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/terminfo directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/cups directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/zoneinfo directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/kbd directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/systemd/system directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/systemd/user directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/systemd/system-generators directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.
Feb 22 08:09:38 nixos stage-2-init: /etc/udev/rules.d directory contains user files. Symlinking may fail. at /nix/store/amhpx347b7n6m2bq24siw0qq27w3np8n-setup-etc.pl line 114.

danbst · 2019-02-22T08:39:37Z

@eadwu I'll take a look

dtzWill · 2019-02-22T18:50:06Z

Same warnings here. Basically the check for isStatic is incomplete and doesn't handle things like:

/etc/plymouth/plugins -> ../static/plymouth/plugins

Some of this can be fixed but I don't know if there's a great answer for this that doesn't involve manually walking the filesystem or making some assumptions (perhaps enabled by constructing /etc to make this easier?).

Well nevermind, I was puttering late at night so maybe there's a good answer here :). But I think this should be fixed soon and carefully. And if this is going to 19.03 resolving these warnings should be a blocker so as to not confuse or mislead users. It looks like /etc is still populated appropriately, but would be good to have some confidence we're not going to nuke user files by misunderstanding their origin or something.

danbst · 2019-02-23T07:40:42Z

@dtzWill you were correct.

I've spent a few hours trying to make this coversion /etc/udev/rules.d -> ../static/udev/rules.d -> /etc/static/udev/rules.d, but failed. Perl explicitly warns that

/etc/udev/../static/udev/rules.d

isn't same as

/etc/static/udev/rules.d

and none of it's primitives resolve .. without taking symlink realpath. And realpath doesn't work, because /etc/static is itself tree of symlinks, so /etc/static realpath and /etc/static/udev/rules.d realpath are different.

I tend to reverting this PR, until I can figure out what is the most appropriate way to fix this. :(

This reverts commit 0b91fa4, reversing changes made to 183919a.

dtzWill · 2019-02-24T00:37:34Z

On Fri, 22 Feb 2019 23:41:14 -0800, Danylo Hlynskyi ***@***.***> wrote: @dtzWill you were correct.

:( I was hoping I'd missed something.

I've spent a few hours trying to make this coversion `/etc/udev/rules.d` -> `../static/udev/rules.d` -> `/etc/static/udev/rules.d`, but failed. Perl explicitly warns that ``` /etc/udev/../static/udev/rules.d ``` isn't same as ``` /etc/static/udev/rules.d ``` and none of it's primitives resolve `..` without taking symlink realpath. And realpath doesn't work, because `/etc/static` is itself tree of symlinks, so `/etc/static` realpath and `/etc/static/udev/rules.d` realpath are different. I tend to reverting this PR, until I can figure out what is the most appropriate way to fix this. :(

I've reverted it locally for this reason. If this can't be sorted out perhaps the original goal re:`/etc/os-release` can just be special-cased?

…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #54980 (comment) part: text/html

nixos-discourse · 2019-02-28T05:52:19Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/nixos-rebuild-switch-upgrade-error-directory-contains-user-files-symlinking-may-fail/2266/2

This reverts commit 0b91fa4, reversing changes made to 183919a.

layus · 2019-03-13T02:05:22Z

@danbst It turns out that checking if the symlink resolves to something inside /etc/static is indeed difficult. I think that you need to perform symlink resolution, just like the kernel does. The algorithm is not trivial to implement, but my rust (and python) versions are in http://github.com/layus/readlinks.

Reimplementing this in perl will be error prone. My own implementation is still buggy.

danbst · 2019-03-13T08:50:26Z

@layus there is also namei util, which I thought would be useful to reuse.

This has to be solved in some other way. Currently "static" files are checked by being symlink to /etc/static, but they can also be checked if those exist in some file etc-static-list. So etc-builder should build list of static files, which it can then use to remove unreferenced files.

florianjacob and others added 2 commits January 31, 2019 09:17

make back /etc/static absolute symlink

f47bfce

danbst requested a review from infinisil as a code owner January 31, 2019 08:02

danbst added 2 commits January 31, 2019 10:04

this is not related to this PR

fd807c6

🤦

3ae5420

GrahamcOfBorg added 6.topic: nixos 8.has: changelog 8.has: documentation 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jan 31, 2019

7c6f434c merged commit 0b91fa4 into NixOS:master Feb 21, 2019

dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Feb 23, 2019

Revert "Merge pull request NixOS#54980 from danbst/etc-relative"

c24725d

This reverts commit 0b91fa4, reversing changes made to 183919a.

cleverca22 mentioned this pull request Feb 25, 2019

Possible error following multiple relative symlinks from /etc #56278

Closed

danbst mentioned this pull request Feb 28, 2019

Revert "Merge pull request #54980 from danbst/etc-relative" #56507

Merged

danbst added a commit that referenced this pull request Feb 28, 2019

Revert "Merge pull request #54980 from danbst/etc-relative" (#56507)

79cc48c

This reverts commit 0b91fa4, reversing changes made to 183919a.

7c6f434c added a commit that referenced this pull request Feb 28, 2019

Revert "Merge pull request #54980 from danbst/etc-relative"

dddd1b6

This reverts commit 0b91fa4, reversing changes made to 183919a.

mtrsk mentioned this pull request Feb 28, 2019

Playonlinux complains about OpenGL and jq #56381

Open

eadwu mentioned this pull request Mar 6, 2019

stage-2-init: directory contains user files. Symlinking may fail. #56932

Closed

danbst mentioned this pull request Dec 4, 2019

/etc/os-release is an absolute symlink but should be a relative one for e.g. systemd-nspawn #28833

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos: make symlinks in `/etc` relative (except `/etc/static`) #54980

nixos: make symlinks in `/etc` relative (except `/etc/static`) #54980

danbst commented Jan 31, 2019

7c6f434c commented Jan 31, 2019

eadwu commented Feb 22, 2019 •

edited

Loading

danbst commented Feb 22, 2019

dtzWill commented Feb 22, 2019

danbst commented Feb 23, 2019

dtzWill commented Feb 24, 2019 via email

nixos-discourse commented Feb 28, 2019

layus commented Mar 13, 2019

danbst commented Mar 13, 2019

nixos: make symlinks in /etc relative (except /etc/static) #54980

nixos: make symlinks in /etc relative (except /etc/static) #54980

Conversation

danbst commented Jan 31, 2019

Motivation for this change

Things done

7c6f434c commented Jan 31, 2019

eadwu commented Feb 22, 2019 • edited Loading

danbst commented Feb 22, 2019

dtzWill commented Feb 22, 2019

danbst commented Feb 23, 2019

dtzWill commented Feb 24, 2019 via email

nixos-discourse commented Feb 28, 2019

layus commented Mar 13, 2019

danbst commented Mar 13, 2019

nixos: make symlinks in `/etc` relative (except `/etc/static`) #54980

nixos: make symlinks in `/etc` relative (except `/etc/static`) #54980

eadwu commented Feb 22, 2019 •

edited

Loading