Deprecate legacy ssh store / `nix-store --serve`

[Ericson2314]

Issues like #3372 and #2770 show that things slip through the cracks when we have two sorts of remote stores.

I think after the next release they will finally be at feature parity, we should then deprecate the legacy one so we only have one thing to maintain.

Finally, we can then add the protocol version parameter to our serialization infra. This resolves the tension between structure / code reuse and and being able to evolve the protocol that has hamstrung #4588, for example.

1. Ensure support up is to par
   • Respect builders-use-substitutes flag
   • Add support in hydra (No support for ssh-ng:// protocol hydra#688)
   • Ensure --keep-going can makes it over the wire (nix build --keep-going is broken with --store ssh-ng://... --eval-store auto #5600)
     2 Deprecate / switch defaults
   • Add support ssh-ng:///ssh-legacy:// so we can change the default prior to removing support
   • Use ssh-ng by default for remote builders Use ssh-ng by default #5010
2. Actually remove support
   • Remove LegacySSHStore
   • Remove nix-store --serve

[rickynils]

@Ericson2314 Currently, no command line Nix options are used when using ssh-ng:// (see #5600 (comment)). I think that resolving it should be on the list.

[Ericson2314]

Thanks, added.

[asymmetric]

@Ericson2314 #4180 ticks the first box

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/a-proposal-for-replacing-the-nix-worker-protocol/20926/12

[nrdxp]

@Ericson2314 Currently, no command line Nix options are used when using ssh-ng:// (see #5600 (comment)). I think that resolving it should be on the list.

I'd like to ask specifically about signing. Right now it is already possible to capture an entire remote build and upload cycle in a single command: nix copy --eval-store auto --store ssh-ng://my.builder.com --to s3://myBucket. This is pretty great for building a package on some remote builder and then copying it to your intended cache all in one go, since if the artifact doesn't already exist on the builder targeted by --store then the build will initiate.

The only issue I have run into is signing. Because the nix.conf setting nix-secret-key-files doesn't propagate, I have to run a nix store sign --store s3://myBucket manually on the artifact, which is fine for s3 backed caches, but this doesn't work for ssh stores. It'd be nice if there was either a flag to nix copy to specify a singing key explicitly, or if nix-secret-key-files was respected by the remote builder.

[jmgilman]

@Ericson2314 Currently, no command line Nix options are used when using ssh-ng:// (see #5600 (comment)). I think that resolving it should be on the list.

I'd like to ask specifically about signing. Right now it is already possible to capture an entire remote build and upload cycle in a single command: nix copy --eval-store auto --store ssh-ng://my.builder.com --to s3://myBucket. This is pretty great for building a package on some remote builder and then copying it to your intended cache all in one go, since if the artifact doesn't already exist on the builder targeted by --store then the build will initiate.

The only issue I have run into is signing. Because the nix.conf setting nix-secret-key-files doesn't propagate, I have to run a nix store sign --store s3://myBucket manually on the artifact, which is fine for s3 backed caches, but this doesn't work for ssh stores. It'd be nice if there was either a flag to nix copy to specify a singing key explicitly, or if nix-secret-key-files was respected by the remote builder.

Should this be a separate issue?

[nrdxp]

Should this be a separate issue?

It might have to be, because after writing this post I came to the realization that nix store sign -r on an s3 cache has dismal performance, probably because it is trying to determine dependencies by making an absurd amount of requests. The best solution would be to either forward the key to the build machine temporarily, or even sign in transit, which might be out of scope for this issue.

[Ericson2314]

On a more general note, we should probably have a brand new rethought protocol to where we think about these higher level issues. For me getting rid of legacy ssh isn't because ssh-ng is good enough (it is not!), but because we want to "free up our complexity budget" first --- supporting 3 simultaneous protocols would be disastrous!

[fogti]

On that note, if we don't directly pull in something like protocol buffers or such for a newer protocol, can we anyways please describe the protocol structure in something that isn't C++, and allows automatic generation of the C++ serialization/deserialization code from that (write the transformation in perl or python for portability, we depend on that anyways), to make it easier to interact with it from other languages (e.g. Go and Rust, as commonly suggested)

[nrdxp]

Just wanted to point out that there is actually already an "in transit" signing option, at least for s3. I learned about it just yesterday from this comment: #6960 (comment)

Essentially there is an undocumented parameter on an s3 uri to specify a Nix secret key, I tested it yesterday and it works even when the caller is neither the builder or the cache (--from & --to), so I guess no issue is needed, but we probably want to document that somewhere, as its super useful.

[rickynils]

@nrdxp

Right now it is already possible to capture an entire remote build and upload cycle in a single command: nix copy --eval-store auto --store ssh-ng://my.builder.com --to s3://myBucket. This is pretty great for building a package on some remote builder and then copying it to your intended cache all in one go, since if the artifact doesn't already exist on the builder targeted by --store then the build will initiate.

That is a very useful command! You can set --to auto to get the build result copied to the local store. However, this suffers from #6950, which makes the performance simply awful :(