nix-collect-garbage --delete-older-than Nd does not delete system profile generations older than N days

[exarkun]

Describe the bug

According to nix-collect-garbage --help:

--delete-older-than period, where period is a value such as 30d, which deletes all generations older than the specified number of days in all profiles in /nix/var/nix/profiles (except for the generations that were active at that point in time)

However:

[root@payments:~]# ls -l /nix/var/nix/profiles/
total 68
drwxr-xr-x 4 root root 4096 Jul 19 13:09 per-user
lrwxrwxrwx 1 root root   14 Aug  9 12:47 system -> system-28-link
lrwxrwxrwx 1 root root   87 Jul 15 14:56 system-13-link -> /nix/store/qah43gvnh04kbxbfdaajc8x5qyfraam3-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:14 system-14-link -> /nix/store/hfvdb3sfpazqfz7lcjhpa6v5b465mqk2-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:22 system-15-link -> /nix/store/k7byv3787jjmgn902rrsdnr5613fmgnj-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:25 system-16-link -> /nix/store/f0gps8cqagg7xks6z6n1jx75djxr2i2m-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 14:15 system-17-link -> /nix/store/1szwz2m2amw6y3w6rhk6nyckwca1p6vd-nixos-system-payments-21.05.961.1f91fd10406
lrwxrwxrwx 1 root root   87 Jul 19 14:45 system-18-link -> /nix/store/vlk71zjcvmdg4hy1dp6hbjgzllp8hjh6-nixos-system-payments-21.05.961.1f91fd10406
lrwxrwxrwx 1 root root   87 Jul 19 15:27 system-19-link -> /nix/store/74w1w4xslgrh7j2knkp6l05gja1wixmv-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 15:47 system-20-link -> /nix/store/kldvhw8lbi65mwb3vp0x94ggfkfpdz9r-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 15:49 system-21-link -> /nix/store/74w1w4xslgrh7j2knkp6l05gja1wixmv-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 19:47 system-22-link -> /nix/store/j3vl0f1mv8z16qj449akc97dyrkqynrf-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 14:31 system-23-link -> /nix/store/an796jqxhhi92hyjqjx876ahfcdb67rs-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 14:51 system-24-link -> /nix/store/0dlcmiqcjg293frk6wwky5zaxniwkynx-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 15:26 system-25-link -> /nix/store/j3vl0f1mv8z16qj449akc97dyrkqynrf-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 29 12:14 system-26-link -> /nix/store/17mc5jfwndv7l81c1qsd3z7n6rvn61np-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 29 14:28 system-27-link -> /nix/store/p4dq8knaax3mc8iwgyziz9s8hh7nlv45-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Aug  9 12:47 system-28-link -> /nix/store/p2v9a2b85qk7f1c6w9qaqygrbyrv8wcs-nixos-system-payments-21.05.804.5de44c15758

[root@payments:~]# date --date 'today -25 days'
Sun Jul 18 02:01:43 PM UTC 2021

[root@payments:~]# nix-collect-garbage --delete-older-than 25d
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
removing old generations of profile /nix/var/nix/profiles/system
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
0 store paths deleted, 0.00 MiB freed

[root@payments:~]# ls -l /nix/var/nix/profiles/
total 68
drwxr-xr-x 4 root root 4096 Jul 19 13:09 per-user
lrwxrwxrwx 1 root root   14 Aug  9 12:47 system -> system-28-link
lrwxrwxrwx 1 root root   87 Jul 15 14:56 system-13-link -> /nix/store/qah43gvnh04kbxbfdaajc8x5qyfraam3-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:14 system-14-link -> /nix/store/hfvdb3sfpazqfz7lcjhpa6v5b465mqk2-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:22 system-15-link -> /nix/store/k7byv3787jjmgn902rrsdnr5613fmgnj-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 13:25 system-16-link -> /nix/store/f0gps8cqagg7xks6z6n1jx75djxr2i2m-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 14:15 system-17-link -> /nix/store/1szwz2m2amw6y3w6rhk6nyckwca1p6vd-nixos-system-payments-21.05.961.1f91fd10406
lrwxrwxrwx 1 root root   87 Jul 19 14:45 system-18-link -> /nix/store/vlk71zjcvmdg4hy1dp6hbjgzllp8hjh6-nixos-system-payments-21.05.961.1f91fd10406
lrwxrwxrwx 1 root root   87 Jul 19 15:27 system-19-link -> /nix/store/74w1w4xslgrh7j2knkp6l05gja1wixmv-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 15:47 system-20-link -> /nix/store/kldvhw8lbi65mwb3vp0x94ggfkfpdz9r-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 15:49 system-21-link -> /nix/store/74w1w4xslgrh7j2knkp6l05gja1wixmv-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 19 19:47 system-22-link -> /nix/store/j3vl0f1mv8z16qj449akc97dyrkqynrf-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 14:31 system-23-link -> /nix/store/an796jqxhhi92hyjqjx876ahfcdb67rs-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 14:51 system-24-link -> /nix/store/0dlcmiqcjg293frk6wwky5zaxniwkynx-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 28 15:26 system-25-link -> /nix/store/j3vl0f1mv8z16qj449akc97dyrkqynrf-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 29 12:14 system-26-link -> /nix/store/17mc5jfwndv7l81c1qsd3z7n6rvn61np-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Jul 29 14:28 system-27-link -> /nix/store/p4dq8knaax3mc8iwgyziz9s8hh7nlv45-nixos-system-payments-21.05.804.5de44c15758
lrwxrwxrwx 1 root root   87 Aug  9 12:47 system-28-link -> /nix/store/p2v9a2b85qk7f1c6w9qaqygrbyrv8wcs-nixos-system-payments-21.05.804.5de44c15758

Despite system-13-link being more than 25 days old, nix-collect-garbage has not deleted it.

Steps To Reproduce

1. Have a system with at least one generation not in use
2. As root, run nix-collect-garbage --delete-older-than Nd where N is smaller than the age of a generation not in use, in days
3. See that the old generation has not been deleted

Expected behavior

All generations that are older than the given number of days are removed from the profile directory and all nix store objects that become unreachable as a result are deleted from the nix store.

nix-env --version output

nix-env (Nix) 2.3.12

Additional context

None

[stale]

I marked this as stale due to inactivity. → More info

[Atemu]

I'm running into the same issue on a more recent version of Nix:

$ ls -l /nix/var/nix/profiles/
total 44
lrwxrwxrwx 1 root root 43 Jul 12  2023 default -> /nix/var/nix/profiles/per-user/root/profile
drwxr-xr-x 1 root root 18 May  6  2023 per-user
lrwxrwxrwx 1 root root 14 Apr 19 05:42 system -> system-33-link
lrwxrwxrwx 1 root root 86 Oct 25 20:27 system-25-link -> /nix/store/gq5q81wwkgjanrv47g2i2zy9wiwmki7z-nixos-system-PLATON-23.11.git.6e6f46c88801
lrwxrwxrwx 1 root root 86 Mar 29 12:32 system-26-link -> /nix/store/38prfflrixiczv34jxnvl26q17i0hrix-nixos-system-PLATON-24.05.git.51af882ac3a2
lrwxrwxrwx 1 root root 86 Mar 29 19:43 system-27-link -> /nix/store/8cqaryx7kpv0vdpflp76fxa1wzgbk468-nixos-system-PLATON-24.05.git.51af882ac3a2
lrwxrwxrwx 1 root root 86 Mar 30 00:31 system-28-link -> /nix/store/75wcyf2gk44501dkzmp5bkzbhq04mhda-nixos-system-PLATON-24.05.git.51af882ac3a2
lrwxrwxrwx 1 root root 86 Apr  2 19:42 system-29-link -> /nix/store/x6xkrm6ial9nw31jjq6vccqypd7ya4mc-nixos-system-PLATON-24.05.git.51af882ac3a2
lrwxrwxrwx 1 root root 86 Apr  4 08:31 system-30-link -> /nix/store/qv10jmw6ygamwgyyjg9pj3031p58fwn0-nixos-system-PLATON-24.05.git.ae92c8291baa
lrwxrwxrwx 1 root root 86 Apr  5 03:15 system-31-link -> /nix/store/zsxikbi44abw2yrmrclzgqmzq2qp0daz-nixos-system-PLATON-24.05.git.ae92c8291baa
lrwxrwxrwx 1 root root 86 Apr  7 11:23 system-32-link -> /nix/store/dkcqxzjpm9yv7yzrlhb5vizrp7sk4g6q-nixos-system-PLATON-24.05.git.ae92c8291baa
lrwxrwxrwx 1 root root 86 Apr 19 05:42 system-33-link -> /nix/store/4nh79r7w9n0ifnvg1biqx5c036xab2qr-nixos-system-PLATON-24.05.git.28f737ff73ce
drwxr-xr-x 1 root root 30 Apr 19 07:10 system-profiles

$ date --date 'today -25 days'
Mon Mar 25 06:13:24 CET 2024

$ sudo nix-collect-garbage --delete-older-than 30d -vvvvvv |& | grep -v -E '^got additional root'
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
locking path '/nix/var/nix/profiles/per-user/root/channels'
lock acquired on '/nix/var/nix/profiles/per-user/root/channels.lock'
lock released on '/nix/var/nix/profiles/per-user/root/channels.lock'
removing old generations of profile /nix/var/nix/profiles/per-user/atemu/profile
locking path '/nix/var/nix/profiles/per-user/atemu/profile'
lock acquired on '/nix/var/nix/profiles/per-user/atemu/profile.lock'
lock released on '/nix/var/nix/profiles/per-user/atemu/profile.lock'
removing old generations of profile /nix/var/nix/profiles/system-profiles/test
locking path '/nix/var/nix/profiles/system-profiles/test'
lock acquired on '/nix/var/nix/profiles/system-profiles/test.lock'
lock released on '/nix/var/nix/profiles/system-profiles/test.lock'
removing old generations of profile /nix/var/nix/profiles/system
locking path '/nix/var/nix/profiles/system'
lock acquired on '/nix/var/nix/profiles/system.lock'
lock released on '/nix/var/nix/profiles/system.lock'
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
locking path '/nix/var/nix/profiles/per-user/root/channels'
lock acquired on '/nix/var/nix/profiles/per-user/root/channels.lock'
lock released on '/nix/var/nix/profiles/per-user/root/channels.lock'
finding garbage collector roots...
deleting garbage...
cannot delete '/nix/store/g0xz4n15vr8q5klyl7iqavxfn9qxwv8l-nixos' because it's a root
cannot delete '/nix/store/q74idm55v5km2pp9yh5qhzc4cw639kp4-cantarell-fonts-0.303.1' because it's a root
cannot delete '/nix/store/41pi9f016l2n5s0fiiy13rdx9xmcs01x-nixos-system-PLATON-23.11.git.be0782147524M' because it's a root
cannot delete '/nix/store/gq5q81wwkgjanrv47g2i2zy9wiwmki7z-nixos-system-PLATON-23.11.git.6e6f46c88801' because it's a root
cannot delete '/nix/store/c0xz1xlbba1a7m0g7v19gff7c1lkg0cs-lorri-keep-env-hack-ghc-shell-for-packages-0' because it's a root
cannot delete '/nix/store/x6xkrm6ial9nw31jjq6vccqypd7ya4mc-nixos-system-PLATON-24.05.git.51af882ac3a2' because it's a root
cannot delete '/nix/store/38prfflrixiczv34jxnvl26q17i0hrix-nixos-system-PLATON-24.05.git.51af882ac3a2' because it's a root
cannot delete '/nix/store/sfhy4ci6n9jfznji6wgbw2f73wbg5ir5-ghc-shell-for-scotty-0.20.1-0-env' because it's a root
cannot delete '/nix/store/ypkhxink7miy8gw2mi88jlacl213cg4d-flake-registry.json' because it's a root
cannot delete '/nix/store/8cqaryx7kpv0vdpflp76fxa1wzgbk468-nixos-system-PLATON-24.05.git.51af882ac3a2' because it's a root
cannot delete '/nix/store/75wcyf2gk44501dkzmp5bkzbhq04mhda-nixos-system-PLATON-24.05.git.51af882ac3a2' because it's a root
cannot delete '/nix/store/i4acm4vqgcqnxm7abi6in80ya4kc5kzn-system-path' because it's a root
cannot delete '/nix/store/dkcqxzjpm9yv7yzrlhb5vizrp7sk4g6q-nixos-system-PLATON-24.05.git.ae92c8291baa' because it's a root
cannot delete '/nix/store/gs839h6lmscnzsi00nk4zwa1n81a7pzn-lorri-keep-env-hack-ghc-shell-for-packages-0' because it's a root
cannot delete '/nix/store/6k01yn5mp3xdp9br4hf7kyiabj8sp58z-gitlib-3.1.3' because it's a root
cannot delete '/nix/store/4nh79r7w9n0ifnvg1biqx5c036xab2qr-nixos-system-PLATON-24.05.git.28f737ff73ce' because it's a root
cannot delete '/nix/store/qv10jmw6ygamwgyyjg9pj3031p58fwn0-nixos-system-PLATON-24.05.git.ae92c8291baa' because it's a root
cannot delete '/nix/store/6g81pmziwkahlcix3b6dbldrr87r0lg3-user-environment' because it's a root
cannot delete '/nix/store/jx8bfvvzlqakz37kgsm0hwijrm2ny742-user-environment' because it's a root
cannot delete '/nix/store/a97ran5c6b95m1lg1c5lbx67ijw1i23c-user-environment' because it's a root
cannot delete '/nix/store/57904rflsfh46y7fxi62rzc7sidc9vmq-user-environment' because it's a root
cannot delete '/nix/store/f445xsr08kpwhqs93gwmhj8bcs01r4cz-user-environment' because it's a root
cannot delete '/nix/store/6lyhsb152hd3lpgf605na4afrz32k1ya-user-environment' because it's a root
cannot delete '/nix/store/j1jzl8mwvqdiyswg141698hasix2ihan-user-environment' because it's a root
cannot delete '/nix/store/bdyfdhwc6ggz01hghjczlrg9qv2kj797-user-environment' because it's a root
cannot delete '/nix/store/zsxikbi44abw2yrmrclzgqmzq2qp0daz-nixos-system-PLATON-24.05.git.ae92c8291baa' because it's a root
deleting unused links...
note: currently hard linking saves 13180.82 MiB
GC roots server shutting down
0 store paths deleted, 0.00 MiB freed

$ nix-env --version
nix-env (Nix) 2.18.2

cc @thufschmitt