Cannot delete a store path when sudo-ing nix-store --delete $storePath

[picnoir]

Describe the bug

It is currently impossible to use nix-store --delete $storePath when running nix through sudo.

@thufschmitt investigated a bit the issue and realized sudo adds the sudo cmd in a SUDO_COMMAND env variable. See https://manpage.me/index.cgi?apropos=0&q=sudo&sektion=0&manpath=Debian+8.1.0&arch=default&format=html#ENVIRONMENT

Nix is looking at the live processes env variables and creates GC roots for the store paths it'll find. See https://github.com/NixOS/nix/blob/master/src/libstore/gc.cc#L392. It'll find the store path we're trying to delete in the SUDO_COMMAND env variable and will create a temporary GC-Root for it, preventing us to delete the store path.

Steps To Reproduce

~ » hello=$(nix-build -E '(import <nixpkgs> {}).hello' --no-out-link)

~ » sudo nix-store --delete $hello
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1' since it is still alive. To find out why, use: nix-store --query --roots

~ » sudo nix-store --query --roots $hello                                     
/proc/10017/environ -> /nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1
/proc/10034/environ -> /nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1

~ » sudo nix-store --query --roots $hello                                     
/proc/9985/environ -> /nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1
/proc/10002/environ -> /nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1

# Looks like nix is creating some temp gc roots each times it's called
# as root, making effectively impossible to GC a path.

# No issue when run as a user
~ » nix-store --delete $hello
finding garbage collector roots...
deleting '/nix/store/y4mxrg8c6l09lb2szl69vwl4f6441i5k-hello-2.12.1'
deleting unused links...
note: currently hard linking saves 5662.99 MiB
1 store paths deleted, 0.19 MiB freed

Expected behavior

sudo nix-store --delete $hello should delete the store path as expected

nix-env --version output

~ » nix-store --version
(Nix) 2.11.0

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-use-nix-store-delete/22024/10

[rapenne-s]

I'm not sure if it's exactly the same bug, but when trying to delete a store path as a normal user, I get this error:

error: Cannot delete path '/nix/store/29qxfyv1gr5qxkvx0y3bs2chhg8a9f46-nixos-22.11.20220924.fde244a-x86_64-linux.iso' since it is still alive. To find out why, use: nix-store --query --roots

And when running nix-store --query --roots, there is no result.

If I use sudo before the nix-store command, I get the same result as OP

[ncfavier]

Probably related to #6141 and #6135

[picnoir]

I think I also hit the bug you are describing @rapenne-s . I however did not manage to reproduce it later on :/

Would you happen to have a minimal snippet to reproduce it? Maybe we could try to pinpoint what's wrong.

It might be related to this issue to some extent, but I don't think it's the same bug.

[rapenne-s]

Create the files iso.nix and flake.nix and run nix build .#nixosConfigurations.isoimage.config.system.build.isoImage.

Then delete the local result symlink, and try to run nix-store --delete on the iso file that was created.

iso.nix

{ config, pkgs, ... }:
{

  isoImage.squashfsCompression = "zstd -Xcompression-level 5";
  boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "cifs" ];
  networking.dhcpcd.enable = true;
  networking.hostName = "biggy"; # Define your hostname.
  networking.wireless.enable = false;
  time.timeZone = "Europe/Paris";
}

flake.nix

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
  };

  outputs = {
    self,
    nixpkgs,
    ...
  } @ inputs: let
    system = "x86_64-linux";
    pkgs = import nixpkgs {
      inherit system;
      config = {allowUnfree = true;};
    };
    lib = nixpkgs.lib;
  in {
    nixosConfigurations.isoimage = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./iso.nix
        "${nixpkgs}/nixos/modules/installer/cd-dvd/installation-cd-base.nix"
      ];
    };
  };
}

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-36/22120/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cant-find-out-what-takes-so-much-space-in-my-store/22562/3

[tpwrules]

This can be worked around using a command like sudo bash -c "SUDO_COMMAND= `which nix-store` --delete /nix/store/<blah>" to remove the problematic variable from the environment, but ideally Nix should be taught to ignore that environment variable somehow.

However, I am not aware of a case where it is necessary for sudo to be involved with nix-store --delete unless somehow the store permissions have become mangled.

[fricklerhandwerk]

Discussed on the Nix team meeting 2023-01-16:

• @roberth: if Nix unsets SUDO_COMMAND, that should fix the problem?

  • @thufschmitt: no, sudo itself will still have the environment.
    • we could blacklist the environment variable
      • @edolstra: that sounds fragile
    • is there a reason why the GC checks the environment for garbage collection roots, except --delete wouldn't work
    • @tomberek: it checks arg[0] instead of argv to prevent deleting the running program or its dependencies
      • @fricklerhandwerk: how does it work actually?
        • @thufschmitt: there is a file /proc/self that contains the environment variables, and the GC scans that for store paths
          • https://github.com/NixOS/nix/blob/master/src/libstore/gc.cc#L406

• @edolstra: could pass the store path through stdin, that would avoid the problem

  • @thufschmitt: users would still run into the problem
  • @Ericson2314: can add a warning
  • @fricklerhandwerk: and documentation
  • related: feat: read installable paths from stdin #7594

• @tomberek: is there a similar problem with doas?

  • @thufschmitt: don't know

• @roberth: possible overengineerd solution: use Unix socket to determine the UID is what the client claims it to be

  • decision: review feat: read installable paths from stdin #7594 and prioritise a PR that adds an error message

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-01-13-nix-team-meeting-minutes-23/24644/1

[rayhem]

I'm a newbie Nix user but I'm seeing this exact same behavior with some leftover Mathematica files:

./vfbhssrgjrrjacmihg1id5bad5d8rf6w-nixos-23.11.2451.2d2ea8eab9e4/nixos/pkgs/applications/science/math/mathematica
./5qv53mcdjir23y39bymjc81s4ghhfvc0-shared-mime-info-2.3/share/mime/application/mathematica.xml
./c20vhihv4vwmxr77avl9sxis30f1xy6m-system-path/share/mime/application/mathematica.xml
./n9s4dprgz6savrqqld90i2ckym4sg5z8-system-path/share/mime/application/mathematica.xml
./zqi4hz5fgrzv3dw4hqs71v7d7cian753-breeze-icons-5.112.0/share/icons/breeze/apps/48/mathematica.svg
./zqi4hz5fgrzv3dw4hqs71v7d7cian753-breeze-icons-5.112.0/share/icons/breeze-dark/apps/48/mathematica.svg
./9pwr6fbci9yap9r2jy6rrj28vs4if3k5-system-path/share/mime/application/mathematica.xml
./8xy0g5acca10hl6n6nw2s08jvm4qaf80-system-path/share/mime/application/mathematica.xml
./zxajh23dciqh2mha5b6pbib755h2vi74-system-path/share/mime/application/mathematica.xml
./8azz8jpkm8zxb9is2p4cqywmm1qb0igi-system-path/share/mime/application/mathematica.xml
./jy52b0vmd0m7d264m19w9rmaz7y8lsnq-system-path/share/mime/application/mathematica.xml
./2icy4j1kl7zwjy5pgaxmx4fj3j2g2fkw-system-path/share/mime/application/mathematica.xml
./s14kbaldaybkk5qbwwymhyikkzqxidpm-system-path/share/mime/application/mathematica.xml
./rj53r55l36998v7vl8dm9lww0shvfyar-system-path/share/mime/application/mathematica.xml
./ycjv1s494xcr1a4r2fh0ayc78yxfi4pn-system-path/share/mime/application/mathematica.xml
./zlfmivf53p3q4l8h9nqv9v83zzx78c2s-system-path/share/mime/application/mathematica.xml
./dc3x9clpgnnrlfvcz7z3k3yywnk1k3sr-system-path/share/mime/application/mathematica.xml
./65kxy7fs99fckdjv1wwkc6jxiz9q4nx6-system-path/share/mime/application/mathematica.xml
./dv4npqfj8k7hr0h6s0pbah2736ld44zi-Mathematica_13.3.0_BNDL_LINUX.sh.drv
./dcgf5f1lp0f65ycybgcy7svfi770cp3q-mathematica-13.3.0.drv

Ideally I'd like to remove the remnant 13.3.0 files.

[justinTM]

this worked for me: nix-collect-garbage -d

removed 6 gigs of shit woohoo!