-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Substitutions from different store dirs #3689
Substitutions from different store dirs #3689
Conversation
Substituters can substitute from one store dir to another with a little bit of help. The store api just needs to have a CA so it can recompute the store path based on the new store dir. We can only do this for fixed output derivations with no references, though.
this needs to only continue if the path replacement fails.
This can be used to add flat hashes to the nix store.
these rewrites should be transparent, but they are important to know about when debugging
We can’t use custom name here because different names will have different store paths. This is a limitation of the Store API’s reliance on store paths. We might be able to get around the above in the future by using a dummy name for certain fixed output paths.
This puts what we are already doing into a shared method. It just needs a path name and a ca and produces a store path.
Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.
@edolstra We really like this as a new future. One downside of replacing hashed mirrors with it, however, is that we still use the same "name" when rewriting the path even though it is immaterial for fixed output derivations. This means one has to be quite cautious about renaming fixed output derivations, which isn't really the case today. I might still consider the old #3673 in addition to this as a stop-gap until we add another mitigation, such as making a |
There are some legitimate issues with #3673 I think though. We don’t have any good disk caching because we don’t have narinfo and we also will end up requesting a lot more things from tarballs.nixos.org than we currently do. It might make sense to keep hashed-mirrors for people that still use them. |
src/libstore/build.cc
Outdated
@@ -4526,8 +4545,15 @@ void SubstitutionGoal::tryToRun() | |||
Activity act(*logger, actSubstitute, Logger::Fields{worker.store.printStorePath(storePath), sub->getUri()}); | |||
PushActivity pact(act.id); | |||
|
|||
auto subPath = storePath; | |||
if (ca) { | |||
subPath = sub->makeFixedOutputPathFromCA(storePath.name(), *ca); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we already computed subPath
in SubstitutionGoal::tryNext()
, we could avoid computing it again by storing subPath
in SubstitutionGoal
.
I think error in CI has been fixed on master since. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/67 |
|
@Mic92 So it is my goal to handle more obscure situations like:
This basic approach scales to those sorts of things, and combined with CA derivations will get us decently close to making arbitrary store dirs work in practice. I think that is the right foundation on the Nix sides to then pursue the various tricks discussed in NixOS/rfcs#17 on the Nixpkgs side and get all the way there for practical purposes. |
I thought a bit about changing the length of the store path i.e. exporting to different directories. The hard part are string references elf binaries, but it may be possible to solve this as well: https://github.com/yalue/elf32_string_replace There might be other binary formats that could also break but I think with plain text substitutions + elf binaries most programs would already run. |
@Mic92 Cool. Basically it's my view that we need the basic Nix support to get us motivated, but once we are motivated we can persue all the tricks in the book to beat back mass rebuilds in Nixpkgs / user code:
And further generic Nix mechanisms to help:
I think we can really pull all stops and increase our productivity by orders of magnitude. Just need to get these initial steps out of the way so the "crowd" is motivated enough that the rest can be crowdsourced. edit: clarified should should happen in external code vs Nix. |
Sorry, but that's crazy talk. There is no way we're going to teach Nix how to rewrite ELF binaries, or any file format. Nix is a generic package manager so it doesn't (and shouldn't) have any knowledge of particular file formats. And I don't think it's even possible for ELF binaries. A tool like Also, how would you distinguish between |
That would be external tools, not part of nix, like patchelf is not part of nix itself. It's possible to compile binaries to be relocatable. |
Sorry!! Yes I absolutely did mean for any "tricks" to be in Nixpkgs/user code and not Nix, which I agree should stay a generic tool (c.f. CMake vs Ninja). Edited my post to make that clear----keep the kitchen sink out of Nix :)! |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/69 |
It looks like you're deleting hashed mirrors, but is there a way to have a dead simple FTP server serving source tarballs for mirroring into enterprise networks? We rely really heavily on the hashed mirrors in our production setup. Is it possible to make a simple nix binary cache that can only hold fixed-output derivations? |
@bhipple A previous version of this made a substituter out of the hashed mirrors code, so existing hashed mirrors still work. @edolstra didn't want that since regular binaries caches subsume it, but I think it would be great to resurrect that as a store plugin or something. |
@bhipple https://github.com/NixOS/nix/pull/3673/files is the earlier version with the |
Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \NixOS#3689. All of these are available on cache.nixos.org.
Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \NixOS#3689. All of these are available on cache.nixos.org.
When deploying a Hydra instance with current Nix master, most builds would not run because of errors like this: queue monitor: error: --- Error --- hydra-queue-runner error: --- UsageError --- nix-daemon not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv The last error message is from parseContentAddress, which expects a colon-separated string, however what we got here is a store path. Looking at the worker protocol, the following message sent to the Nix daemon caused the error above: 0x1E -> wopQuerySubstitutablePathInfos 0x01 -> Number of paths 0x16 -> Length of string "/nix/store/...-somedrv" 0x00 -> Length of string "" Looking at writeStorePathCAMap, the store path is indeed the first field that's transmitted. However, readStorePathCAMap expects it to be the *second* field *on my machine*, since expression evaluation order is a classic form of unspecified behaviour[1] in C++. This has been introduced in #3689, specifically in commit 66a62b3. [1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions Signed-off-by: aszlig <aszlig@nix.build>
Substituters can substitute from one store dir to another with a
little bit of help. The store api just needs to have a CA so it can
recompute the store path based on the new store dir. We can only do
this for fixed output derivations with no references, though.