-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make fetchGit do full checkout if ref = "*" #2409
Conversation
@NixOS/nix-core Ping! Could you provide some feedback on this? :-) This feature is used extensively as part of a fully sandboxable stack-to-nix Nix generator. Currently, all users have to install patched Nix to their user profile (which works because patch is eval time only). The use case is to be able to fetch from language-specific lock files that do not include refs, at runtime, inside the sandbox. |
I'm not sure it is a great idea, because even '*' isn't all the refs. For example, github has lots of refs that wouldn't be fetched but are valid refs, and this list could change over time. For that reason, I feel the best way to go is specifying the correct ref from the start. I understand this is complicated because other tools are making invalid assumptions about Git as well. The problem and question is do we propagate this invalid assumption to Nix? In particular this question is hard because Nix's goal is for very long-term reproducibility, which is at odds with this request. |
Without this feature, sandboxable Nix generators for Stack and any other lock file format that doesn't include refs are not possible (even in upcoming Stack freeze file). So my motivation is primarily to improve on that front, allowing to build projects via Nix without any intermediary non-sandboxed step (e.g. what Some affected lock file formats: I will check other tools and add them to the list.
I think this feature leads to higher, not lower reproducibility, because ref is incidental to rev. Meaning, it won't matter on which ref is the rev that we're interested in, and so if we were trying to fetch a rev from an ephemeral feature branch, And fundamentally, any remote location fetch is not long-term reproducible, it always depends on other parties. I think the correct question (in Nix context) to ask here is: will it always produce the exact same result? The answer here seems to be yes. |
It won't because the remote list of branches isn't certain to be stable, but your point about it remaining true for any branch name seems legitimate. Perhaps the default should be instead of having to specify I think some input from @NixOS/nix-core would be helpful. |
It is not stable, but it only affects cache, not Nix store.
It seems that this is an optimization problem (and of course everyone is encouraged to specify refs when possible!) rather than a reproducibility problem. This patch mimics
I agree; maybe it should fetch
❤️ |
@yegortimoshenko I think having this functionality make sense, but I'd like to ensure we're matching the same logic these other tools use and nothing more. Does stack do a git clone, or fetch all refs, or what? Same question for mix, npm, etc. Once we're settled on that we can figure out the interface and everything. |
Here's the code for what stack is doing: https://github.com/commercialhaskell/stack/blob/master/subs/pantry/src/Pantry/Repo.hs#L90
npm seems to be resolving github deps to http urls. I don't know what happens if you explicitly give it git urls, but it's rare enough that we haven't run into it yet. |
|
Just to clarify, |
OK, so let's switch this to fetch all heads? As for interface... I don't really like |
It already does fetch all heads: nix/src/libexpr/primops/fetchGit.cc Line 125 in 4b279a0
Do you mean: fetchGit {
url = https://github.com/foo/bar;
} Or rather: fetchGit {
url = https://github.com/foo/bar;
ref = "";
} |
Ah! Your description said all And I meant the former example there, though I think with the way the primop is implemented both would work the same. |
OK! Unlike |
I don't mean to be rude, but I'd love to see this merged. Could I help it along somehow? |
I stated my opinion here: #2431 (comment) |
We're using this in serokell/stack-to-nix (and serokell/mix-to-nix), translating git repo fetches from language-specific lock files. These files includes the hashes and url, but nothing more. External repo's are not guaranteed to even have a ref for a commit. Furthermore, for internal repos, we may need ssh keys. In the public github case, we can use The only alternative here is probably |
I will update this patch following @shlevy's feedback until |
This seems to be less needed with nix 2.2. It's possible github enabled fetching unadvertised refs, and a recent nix change allows the |
To be more precise, since this commit the example from this PR is translated to On a separate note, since maintaining a patched Nix is a little annoying, here is a Nix plugin that replaces the |
What change? It's certainly not intended to allow fetching |
Oh, sorry, the link in my comment was broken. I’ll edit it. Here is the correct one: 4aee93d#diff-7e39b3dccfe1a07ed3f4af7f4d8da628R127 |
Thanks. I guess we should really validate |
I was bit again by this today. I post though, because it seems with slight modification the perfect solution for our current reality is already in this thread.
To be explicit, to me that means given:
Where as now you get this error:
Instead, do a full clone and provide the warning:
Then if someone forgets it, the worst problem is "it's slow" but the reason why is printed right in front of the user. That lets get the best of all worlds I think.
I agree, but breaking compatibility here is too high of a cost with regards to barrier to entry. I think the best thing we can do is allow it since it's so pervasive, and noisily prod users towards the correct solution while dangling "it'll be faster if you just add ref" as a carrot 🙂
In summary, I think this solution lets us meet the software world and it's invalid assumptions where they are, but smooths their transition and nudges them towards more correctness and it's benefits. And it gives people like me trying to evangelize nix to improve reproducibility in the software industry a fighting chance in actually being able to convince people to use it 😄 |
@Ma27 |
@yorickvP on
|
For public github repo's, fetching with the https .tar.gz archive download should always work. You can try this as a workaround. It's sad to hear that nix 2.4 breaks ref="*" again. |
I'm well-aware of that. I described my use-case (which requires
Ahh, I assumed that you meant the unstable versions by mentioning "recent nix versions". It actually works on On Nix 2.4 ( |
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409
Sometimes one might want to fetch a rev without knowing the ref. In this case, full repo clone is usually required. This commit adds a special case to ref handling, so that when
ref = "*"
,fetchGit
fetches all refs prior to fetching the rev.Usage example: