From eb880051300257cc1e465cbb8042d6e65caf67d6 Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Tue, 13 Nov 2018 17:06:03 +0100 Subject: [PATCH 1/8] nixos/systemd: Add a regression test for #50273 --- nixos/tests/systemd.nix | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/nixos/tests/systemd.nix b/nixos/tests/systemd.nix index 65aa553b3148..4d470126abee 100644 --- a/nixos/tests/systemd.nix +++ b/nixos/tests/systemd.nix @@ -56,6 +56,11 @@ import ./make-test.nix { $machine->succeed('test -z $(ls -1 /var/log/journal)'); }; + # Regression test for https://github.com/NixOS/nixpkgs/issues/50273 + subtest "DynamicUser actually allocates a user", sub { + $machine->succeed('systemd-run --pty --property=Type=oneshot --property=DynamicUser=yes --property=User=iamatest whoami | grep iamatest'); + }; + # Regression test for https://github.com/NixOS/nixpkgs/issues/35268 subtest "file system with x-initrd.mount is not unmounted", sub { $machine->shutdown; From e71241793671c11498c388bde8a3de2d89fde25e Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Tue, 13 Nov 2018 16:28:37 +0100 Subject: [PATCH 2/8] nixos/nscd: Disable caching of group and passwd Systemd provides an option for allocating DynamicUsers which we want to use in NixOS to harden service configuration. However, we discovered that the user wasn't allocated properly for services. After some digging this turned out to be, of course, a cache inconsistency problem. When a DynamicUser creation is performed, Systemd check beforehand whether the requested user already exists statically. If it does, it bails out. If it doesn't, systemd continues with allocating the user. However, by checking whether the user exists, nscd will store the fact that the user does not exist in it's negative cache. When the service tries to lookup what user is associated to its uid (By calling whoami, for example), it will try to consult libnss_systemd.so However this will read from the cache and tell report that the user doesn't exist, and thus will return that there is no user associated with the uid. It will continue to do so for the cache duration time. If the service doesn't immediately looks up its username, this bug is not triggered, as the cache will be invalidated around this time. However, if the service is quick enough, it might end up in a situation where it's incorrectly reported that the user doesn't exist. Preferably, we would not be using nscd at all. But we need to use it because glibc reads nss modules from /etc/nsswitch.conf by looking relative to the global LD_LIBRARY_PATH. Because LD_LIBRARY_PATH is not set globally (as that would lead to impurities and ABI issues), glibc will fail to find any nss modules. Instead, as a hack, we start up nscd with LD_LIBRARY_PATH set for only that service. Glibc will forward all nss syscalls to nscd, which will then respect the LD_LIBRARY_PATH and only read from locations specified in the NixOS config. we can load nss modules in a pure fashion. However, I think by accident, we just copied over the default settings of nscd, which actually caches user and group lookups. We already disable this when sssd is enabled, as this interferes with the correct working of libnss_sss.so as it already does its own caching of LDAP requests. (See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/usingnscd-sssd) Because nscd caching is now also interferring with libnss_systemd.so and probably also with other nsss modules, lets just pre-emptively disable caching for now for all options related to users and groups, but keep it for caching hosts ans services lookups. Note that we can not just put in /etc/nscd.conf: enable-cache passwd no As this will actually cause glibc to _not_ forward the call to nscd at all, and thus never reach the nss modules. Instead we set the negative and positive cache ttls to 0 seconds as a workaround. This way, Glibc will always forward requests to nscd, but results will never be cached. Fixes #50273 --- nixos/modules/services/system/nscd.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nixos/modules/services/system/nscd.conf b/nixos/modules/services/system/nscd.conf index 6d0dcacf9778..27599a08e7e9 100644 --- a/nixos/modules/services/system/nscd.conf +++ b/nixos/modules/services/system/nscd.conf @@ -4,16 +4,16 @@ paranoia no debug-level 0 enable-cache passwd yes -positive-time-to-live passwd 600 -negative-time-to-live passwd 20 +positive-time-to-live passwd 0 +negative-time-to-live passwd 0 suggested-size passwd 211 check-files passwd yes persistent passwd no shared passwd yes enable-cache group yes -positive-time-to-live group 3600 -negative-time-to-live group 60 +positive-time-to-live group 0 +negative-time-to-live group 0 suggested-size group 211 check-files group yes persistent group no From 99d32799520366422b169438cd990fb133d847d4 Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Wed, 14 Nov 2018 12:19:58 +0100 Subject: [PATCH 3/8] nixos/nscd: Disable negative caching of hosts Hopefully fixes #50290 --- nixos/modules/services/system/nscd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nixos/modules/services/system/nscd.conf b/nixos/modules/services/system/nscd.conf index 27599a08e7e9..e860e153965a 100644 --- a/nixos/modules/services/system/nscd.conf +++ b/nixos/modules/services/system/nscd.conf @@ -21,7 +21,7 @@ shared group yes enable-cache hosts yes positive-time-to-live hosts 600 -negative-time-to-live hosts 5 +negative-time-to-live hosts 0 suggested-size hosts 211 check-files hosts yes persistent hosts no From 335b41b3fbf1191e9310dc2717a68df2b0759b76 Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Wed, 14 Nov 2018 13:03:13 +0100 Subject: [PATCH 4/8] nixos/nscd: Add release note entry about nscd changes --- nixos/doc/manual/release-notes/rl-1903.xml | 60 ++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/nixos/doc/manual/release-notes/rl-1903.xml b/nixos/doc/manual/release-notes/rl-1903.xml index bade93c0984e..975c566411c0 100644 --- a/nixos/doc/manual/release-notes/rl-1903.xml +++ b/nixos/doc/manual/release-notes/rl-1903.xml @@ -245,6 +245,66 @@ options. + + + The nscd now disables all caching of + passwd and group databases by + default. This was interferring with the correct functioning of the + libnss_systemd.so module which is used by + systemd to manage uids and usernames in the presence + of DynamicUser= in systemd services. + The was already the default behaviour in presence of + services.sssd.enable = true because nscd caching + would interfere sssd in unpredictable ways as well.Because we're using nscd + not for caching, but for convincing glibc to find NSS modules in the + nix store instead of an absolute path, we have decided to disable + caching globally now, as it's usually not the behaviour the user wants + and can lead to surprising behaviour. + Furthermore, negative caching of host lookups is also disabled now by + default. This should fix the issue of dns lookups failing in the + presence of an unreliable network. + + + If the old behaviour is desired, this can be restored by setting + the services.nscd.config option + with the desired caching parameters. + + services.nscd.config = + '' + server-user nscd + threads 1 + paranoia no + debug-level 0 + + enable-cache passwd yes + positive-time-to-live passwd 600 + negative-time-to-live passwd 20 + suggested-size passwd 211 + check-files passwd yes + persistent passwd no + shared passwd yes + + enable-cache group yes + positive-time-to-live group 3600 + negative-time-to-live group 60 + suggested-size group 211 + check-files group yes + persistent group no + shared group yes + + enable-cache hosts yes + positive-time-to-live hosts 600 + negative-time-to-live hosts 5 + suggested-size hosts 211 + check-files hosts yes + persistent hosts no + shared hosts yes + ''; + + See #50316 + for details. + + GitLab Shell previously used the nix store paths for the From de76c16f9c277c0a104ca7f9fa8d8be46ada6c9a Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Fri, 16 Nov 2018 15:03:00 +0100 Subject: [PATCH 5/8] nixos/nscd: Merge nscd and sssd-nscd config --- nixos/modules/services/misc/nscd-sssd.conf | 36 ---------------------- nixos/modules/services/misc/sssd.nix | 1 - nixos/modules/services/system/nscd.conf | 8 +++++ 3 files changed, 8 insertions(+), 37 deletions(-) delete mode 100644 nixos/modules/services/misc/nscd-sssd.conf diff --git a/nixos/modules/services/misc/nscd-sssd.conf b/nixos/modules/services/misc/nscd-sssd.conf deleted file mode 100644 index 92380f3e4ba4..000000000000 --- a/nixos/modules/services/misc/nscd-sssd.conf +++ /dev/null @@ -1,36 +0,0 @@ -server-user nscd -threads 1 -paranoia no -debug-level 0 - -enable-cache passwd yes -positive-time-to-live passwd 0 -negative-time-to-live passwd 0 -suggested-size passwd 211 -check-files passwd yes -persistent passwd no -shared passwd yes - -enable-cache group yes -positive-time-to-live group 0 -negative-time-to-live group 0 -suggested-size group 211 -check-files group yes -persistent group no -shared group yes - -enable-cache hosts yes -positive-time-to-live hosts 600 -negative-time-to-live hosts 5 -suggested-size hosts 211 -check-files hosts yes -persistent hosts no -shared hosts yes - -enable-cache services yes -positive-time-to-live services 0 -negative-time-to-live services 0 -suggested-size services 211 -check-files services yes -persistent services no -shared services yes diff --git a/nixos/modules/services/misc/sssd.nix b/nixos/modules/services/misc/sssd.nix index e818f4a4804d..fe472a6c68e5 100644 --- a/nixos/modules/services/misc/sssd.nix +++ b/nixos/modules/services/misc/sssd.nix @@ -75,7 +75,6 @@ in { }; system.nssModules = optional cfg.enable pkgs.sssd; - services.nscd.config = builtins.readFile ./nscd-sssd.conf; services.dbus.packages = [ pkgs.sssd ]; }) diff --git a/nixos/modules/services/system/nscd.conf b/nixos/modules/services/system/nscd.conf index e860e153965a..304be0942dc4 100644 --- a/nixos/modules/services/system/nscd.conf +++ b/nixos/modules/services/system/nscd.conf @@ -26,3 +26,11 @@ suggested-size hosts 211 check-files hosts yes persistent hosts no shared hosts yes + +enable-cache services yes +positive-time-to-live services 0 +negative-time-to-live services 0 +suggested-size services 211 +check-files services yes +persistent services no +shared services yes From a74619c1ae5348a5b7cd0dc3b6ca6e0166086098 Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Fri, 16 Nov 2018 15:08:01 +0100 Subject: [PATCH 6/8] nixos/nscd: also add netgroup to the config It was the last database that wasn't listed. --- nixos/modules/services/system/nscd.conf | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/nixos/modules/services/system/nscd.conf b/nixos/modules/services/system/nscd.conf index 304be0942dc4..ce583816e156 100644 --- a/nixos/modules/services/system/nscd.conf +++ b/nixos/modules/services/system/nscd.conf @@ -19,6 +19,14 @@ check-files group yes persistent group no shared group yes +enable-cache netgroup yes +positive-time-to-live netgroup 0 +negative-time-to-live netgroup 0 +suggested-size netgroup 211 +check-files netgroup yes +persistent netgroup no +shared netgroup yes + enable-cache hosts yes positive-time-to-live hosts 600 negative-time-to-live hosts 0 From ef6ed03e2f7e757a46469077d8ef66cecccb919d Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Wed, 12 Dec 2018 14:49:19 +0100 Subject: [PATCH 7/8] nixos/nscd: Address doc feedback --- nixos/doc/manual/release-notes/rl-1903.xml | 25 +++++++++++----------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/nixos/doc/manual/release-notes/rl-1903.xml b/nixos/doc/manual/release-notes/rl-1903.xml index 975c566411c0..9405bf063d51 100644 --- a/nixos/doc/manual/release-notes/rl-1903.xml +++ b/nixos/doc/manual/release-notes/rl-1903.xml @@ -247,22 +247,21 @@ - The nscd now disables all caching of + The nscd service now disables all caching of passwd and group databases by default. This was interferring with the correct functioning of the libnss_systemd.so module which is used by - systemd to manage uids and usernames in the presence - of DynamicUser= in systemd services. - The was already the default behaviour in presence of - services.sssd.enable = true because nscd caching - would interfere sssd in unpredictable ways as well.Because we're using nscd - not for caching, but for convincing glibc to find NSS modules in the - nix store instead of an absolute path, we have decided to disable - caching globally now, as it's usually not the behaviour the user wants - and can lead to surprising behaviour. - Furthermore, negative caching of host lookups is also disabled now by - default. This should fix the issue of dns lookups failing in the - presence of an unreliable network. + systemd to manage uids and usernames in the presence of + DynamicUser= in systemd services. This was already the + default behaviour in presence of services.sssd.enable = + true because nscd caching would interfere with + sssd in unpredictable ways as well. Because we're + using nscd not for caching, but for convincing glibc to find NSS modules + in the nix store instead of an absolute path, we have decided to disable + caching globally now, as it's usually not the behaviour the user wants and + can lead to surprising behaviour. Furthermore, negative caching of host + lookups is also disabled now by default. This should fix the issue of dns + lookups failing in the presence of an unreliable network. If the old behaviour is desired, this can be restored by setting From 1d5f4cbb784904e0a6420cc62b0b24ca0873abc5 Mon Sep 17 00:00:00 2001 From: Arian van Putten Date: Wed, 12 Dec 2018 15:34:05 +0100 Subject: [PATCH 8/8] nixos/nscd: Add a descriptive comment to the nscd configuration --- nixos/modules/services/system/nscd.conf | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/nixos/modules/services/system/nscd.conf b/nixos/modules/services/system/nscd.conf index ce583816e156..603a5d01acce 100644 --- a/nixos/modules/services/system/nscd.conf +++ b/nixos/modules/services/system/nscd.conf @@ -1,3 +1,11 @@ +# We basically use nscd as a proxy for forwarding nss requests to appropriate +# nss modules, as we run nscd with LD_LIBRARY_PATH set to the directory +# containing all such modules +# Note that we can not use `enable-cache no` As this will actually cause nscd +# to just reject the nss requests it receives, which then causes glibc to +# fallback to trying to handle the request by itself. Which won't work as glibc +# is not aware of the path in which the nss modules live. As a workaround, we +# have `enable-cache yes` with an explicit ttl of 0 server-user nscd threads 1 paranoia no