Nix can't receive derivations

[grahamc]

Describe the bug

Switching from Nix stable to nixUnstable (3.0pre20201020_e0ca98c) in 20.09 has broken nix-copy-closure with .drv files:

$ nix-copy-closure --to root@4cf2cdd7.packethost.net /nix/store/l6zwpfnvbha57zhkqxh8inj1shpzym7h-netboot.drv
copying 1848 paths...
copying path '/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv' to 'ssh://root@4cf2cdd7.packethost.net'...
copying path '/nix/store/0bkk8kp4qyfygr11j47b7l2qqqswqpqd-config.sub?id=e78c96e5288993aaea3ec44e5c6ee755c668da79.drv' to 'ssh://root@4cf2cdd7.packethost.net'...
error: --- InvalidPath --- nix-store
path '/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv' is not a valid store path
error (ignored): unexpected end-of-file
error (ignored): unexpected end-of-file
error (ignored): unexpected end-of-file
error (ignored): unexpected end-of-file
error (ignored): interrupted by the user
error: unexpected end-of-file

Sending with the same version of Nix locally, I get the same error:

nix-copy-closure --to root@4cf2cdd7.packethost.net /nix/store/l6zwpfnvbha57zhkqxh8inj1shpzym7h-netboot.drv
copying 1848 paths...
copying path '/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv' to 'ssh://root@4cf2cdd7.packethost.net'...
copying path '/nix/store/0bkk8kp4qyfygr11j47b7l2qqqswqpqd-config.sub?id=e78c96e5288993aaea3ec44e5c6ee755c668da79.drv' to 'ssh://root@4cf2cdd7.packethost.net'...
error: --- InvalidPath --- nix-store
path '/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv' is not a valid store path
error (ignored): error: --- EndOfFile ------------------------------------------------------------------------------------------------------------ nix-copy-closure
unexpected end-of-file
error (ignored): error: --- Interrupted ---------------------------------------------------------------------------------------------------------- nix-copy-closure
interrupted by the user
error (ignored): error: --- EndOfFile ------------------------------------------------------------------------------------------------------------ nix-copy-closure
unexpected end-of-file
error (ignored): error: --- Error ---------------------------------------------------------------------------------------------------------------- nix-copy-closure
unexpected end-of-file
^Cerror (ignored): error: --- Error ---------------------------------------------------------------------------------------------------------------- nix-copy-closure
cannot connect to 'root@4cf2cdd7.packethost.net'
error: --- EndOfFile ------------------------------------------------------------------------------------------------------------ nix-copy-closure
unexpected end-of-file

Since I use nix-copy-closure extensively in the building and deploying of large closures, this has broken a lot of my workflows -- including the workflow to fix the aarch64 builders needed to advance the NixOS channels.

On the receiving side, /nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv is created with the correct contents but never gets registered.

[thufschmitt]

I could easily reproduce it (see below). At a glance it looks like nix copy doesn't have the bug.

Quite astonishingly, the standard Nix test suite doesn't have anything for nix-copy-closure, which is indeed quite broken. The failure can be reproduced by running make install && make tests/nix-copy-ssh.sh.test with the following patch applied:

diff --git a/tests/nix-copy-ssh.sh b/tests/nix-copy-ssh.sh
index eb801548d..6743abce1 100644
--- a/tests/nix-copy-ssh.sh
+++ b/tests/nix-copy-ssh.sh
@@ -15,6 +15,9 @@ nix copy --to "ssh://localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fs
 
 clearStore
 
+nix-copy-closure --to "localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fstore=$NIX_STORE_DIR%26real=$remoteRoot$NIX_STORE_DIR" $outPath
+[ -f $remoteRoot$outPath/foobar ]
+
 nix copy --no-check-sigs --from "ssh://localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fstore=$NIX_STORE_DIR%26real=$remoteRoot$NIX_STORE_DIR" $outPath
 
 [ -f $outPath/foobar ]

[grahamc]

Maybe this test could be extended to explicitly test copying .drv files, too?

[thufschmitt]

Maybe this test could be extended to explicitly test copying .drv files, too?

It sure could! (More generally I think it would be wonderful if someone™ could step-up to rework and extend the test infrastructure of Nix which I find quite weak and doesn't really have a good coverage… but that's a whole other story)

[grahamc]

Yes, I think @andir and @gilligan worked on that some but found it hard to have their PRs merge.

[edolstra]

There is a test for nix-copy-closure in tests/nix-copy-closure.nix and it's currently succeeding (https://hydra.nixos.org/build/129476570).

[gilligan]

i don’t think I’ll invest time into unit tests again.

I have accepted that my professional beliefs in regard to conducting and requiring tests for any code added or modified to a project of this scope are just too far off from the desired status quo - so i stepped back. I don’t want to cause more friction.

[grahamc]

There is a test for nix-copy-closure in tests/nix-copy-closure.nix and it's currently succeeding (https://hydra.nixos.org/build/129476570).

I wonder what is going on in this case, then? I think @LnL7 was able to reproduce.

[SuperSandro2000]

I am getting error: unexpected end-of-file a lot with nixpkgs-review to one of my builders which runs nix version 3.0pre20201020_e0ca98c. When building KDE packages I get it almost everytime.

[grahamc]

I don't think all "end of file" errors are related to this issue. In fact I highly doubt it is. Maybe check your builder's journal logs for nix-daemon? This bug is about nix-copy-closure with .drv files failing with the drv being "not a valid store path".

[SuperSandro2000]

Yeah you are right I get a different error:

unexpected Nix daemon error: error: --- SysError --- nix-daemon
writing to file: Broken pipe

[thufschmitt]

I had a quick look at this. Couldn't get to the bottom of it, but on a modified version of the nix-copy-ssh test I got it to fail with the same error because in LocalStore::addToStore a call to Store::toRealPath(info.path) returns an erroneous path (in my test it returned /tmp/nix-test/default/store2/tmp/nix-test/default/store/qlwxnh1v8sk1lk5m0kln7gv0yphd5fqm-dependencies-input-1.drv while it should have been /tmp/nix-test/default/store2/qlwxnh1v8sk1lk5m0kln7gv0yphd5fqm-dependencies-input-1.drv), causing the path to be written in a wrong location.
Then a subsequent call to readDerivation causes a failure.

That might be a different issue because these command also try all the possible combinations of remote and local stores, but it looks rather similar.

To reproduce this error, run make install && make tests/nix-copy-ssh.sh.test with the following patch

diff --git a/tests/nix-copy-ssh.sh b/tests/nix-copy-ssh.sh
index eb801548d..ae6d4fda9 100644
--- a/tests/nix-copy-ssh.sh
+++ b/tests/nix-copy-ssh.sh
@@ -4,17 +4,18 @@ clearStore
 clearCache
 
 remoteRoot=$TEST_ROOT/store2
-chmod -R u+w "$remoteRoot" || true
-rm -rf "$remoteRoot"
+clearRemote() {
+    chmod -R u+w "$remoteRoot" || true
+    rm -rf "$remoteRoot"
+}
+clearRemote
 
-outPath=$(nix-build --no-out-link dependencies.nix)
 
-nix copy --to "ssh://localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fstore=$NIX_STORE_DIR%26real=$remoteRoot$NIX_STORE_DIR" $outPath
+# The uri of the remote store as it's accepted by `nix-copy-closure` (without
+# the `ssh://` scheme)
+remoteStoreNoScheme="localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fstore=$NIX_STORE_DIR%26real=$remoteRoot$NIX_STORE_DIR"
+# Same, but with the scheme for `nix copy`
+remoteStore="ssh://$remoteStoreNoScheme"
 
-[ -f $remoteRoot$outPath/foobar ]
-
-clearStore
-
-nix copy --no-check-sigs --from "ssh://localhost?store=$NIX_STORE_DIR&remote-store=$remoteRoot%3fstore=$NIX_STORE_DIR%26real=$remoteRoot$NIX_STORE_DIR" $outPath
-
-[ -f $outPath/foobar ]
+drvPath=$(nix-instantiate dependencies.nix)
+nix-copy-closure --to "$remoteStoreNoScheme" "$drvPath"

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-4/9862/1