Add `allRefs = true` argument to `fetchGit` to explicitly perform a full checkout #3814

Ma27 · 2020-07-14T19:21:22Z

Sometimes it's needed to fetch the revision of a git repository without knowing the specific ref. In those cases it's needed to perform a full checkout which is now possible by declaring builtins.fetchGit { allRefs = true; /* url, rev, etc */ }.

Refs #2431 #2409

For further details, please read the descriptions of the two commits.

cc @edolstra

bjornfor · 2020-07-14T20:09:40Z

Git clone fetches all branches by default, that's why I think fetchGit should do the same, by simply omitting the ref argument.

Ma27 · 2020-07-14T20:11:41Z

Well, we're not cloning a git repo though, only the specified ref (or master as fallback) is fetched using git fetch. That's the entire problem behind this :)

Ma27 · 2020-07-14T20:18:08Z

Also, I think it's fine to fetch master only by default: unless you need it, you shouldn't have to fetch a full repo.

src/libfetchers/git.cc

bjornfor · 2020-07-15T07:55:25Z

Well, we're not cloning a git repo though, only the specified ref (or master as fallback) is fetched using git fetch. That's the entire problem behind this :)

I know, but instead of falling back to master we could fall back to "all". IMHO that's more natural, because git clone URL can be translated more directly tofetchGit { URL } (less new things to learn).

Ma27 · 2020-07-15T17:39:22Z

@bjornfor I agree with @edolstra's reasoning in #2431 (comment). This should just be an escape-hatch if you really need it.

Ma27 · 2020-07-17T18:44:04Z

Rebased onto latest master.

Ma27 · 2020-07-26T00:23:57Z

@edolstra would you be fine with the overall approach taken here to work around the usability-issues of fetchGit?

Ma27 · 2020-07-29T15:00:40Z

Rebased onto master now.

Ma27 · 2020-08-10T15:10:57Z

cc @edolstra would you mind taking a look at this? :)

Ma27 · 2020-09-08T17:41:20Z

@edolstra @domenkozar rebased onto latest master. Would you mind taking a look? :)

Ma27 · 2020-09-27T20:21:22Z

@edolstra is there anything missing in this PR? :)

src/libfetchers/git.cc

edolstra · 2020-09-29T11:49:04Z

For backwards compatibility, instead of adding an allRefs argument, we could also bring back ref = "*".

BTW, when using this, what ref gets stored in lock files?

Ma27 · 2020-09-29T15:33:38Z

For backwards compatibility, instead of adding an allRefs argument, we could also bring back ref = "*".

Unfortunately I don't remember the exact details why I decided to create a new attribute, but I think that this was to avoid doing even more magic with ref (we already check if it starts with refs/ etc.).

However I just realized that we'd have to use a string anyways to get this working as flake input (unless we alter the logic in libexpr, see below in this comment). So @edolstra it's up to you what you'd prefer, I guess I'd be fine with both variants.

BTW, when using this, what ref gets stored in lock files?

None as I just realized that using this as flake-input doesn't even evaluate (only tested with fetchGit here, sorry!).

The reason is that flake-attrs can't seem to be a boolean:

nix/src/libexpr/flake/flake.cc

Lines 118 to 122 in 5999978

    
           if (attr.value->type == tString) 
        
               attrs.emplace(attr.name, attr.value->string.s); 
        
           else 
        
               throw TypeError("flake input attribute '%s' is %s while a string is expected", 
        
                   attr.name, showType(*attr.value));

Ma27 · 2020-10-02T09:11:24Z

ping @edolstra how shall we proceed here? :)

Ma27 · 2020-10-09T09:14:59Z

ping @edolstra

edolstra · 2020-10-09T09:39:57Z

The reason is that flake-attrs can't seem to be a boolean:

We can just handle booleans there. The only reason they're not implemented is because there was no need at the time.

Ma27 · 2020-10-09T09:41:31Z

...which means that the current approach is fine?

Ma27 · 2020-10-28T10:05:59Z

cc @edolstra anything else tbd here now? :)

Ma27 · 2020-11-08T18:55:58Z

@edolstra is the current change fine now?

Ma27 · 2020-11-15T23:23:16Z

cc @edolstra anything tbd here? :)

Ma27 · 2020-11-29T17:29:08Z

cc @edolstra can we merge this?

src/libexpr/primops/fetchTree.cc

edolstra · 2020-12-21T22:01:01Z

src/libfetchers/git.cc

+
+        auto result = runProgram(checkCommitOpts);
+        if (WEXITSTATUS(result.first) == 128
+            && result.second.find("bad file") != std::string::npos


Checking for strings like "bad file" in stderr seems pretty fragile... Do we actually need this test? Presumably a wrong rev will fail in the subsequent call to git archive.

I'm afraid no: if git cat-file errors, the exit code is always 128, the reason for the error is on stderr.

…revision can't be checked out A common pitfall when using e.g. `builtins.fetchGit` is the `fatal: not a tree object`-error when trying to fetch a revision of a git-repository that isn't on the `master` branch and no `ref` is specified. In order to make clear what's the problem, I added a simple check whether the revision in question exists and if it doesn't a more meaningful error-message is displayed: ``` nix-repl> builtins.fetchGit { url = "https://github.com/owner/myrepo"; rev = "<commit not on master>"; } moderror: --- Error -------------------------------------------------------------------- nix Cannot find Git revision 'bf1cc5c648e6aed7360448a3745bb2fe4fbbf0e9' in ref 'master' of repository 'https://gitlab.com/Ma27/nvim.nix'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit. ``` Closes NixOS#2431

Sometimes it's necessary to fetch a git repository at a revision and it's unknown which ref contains the revision in question. An example would be a Cargo.lock which only provides the URL and the revision when using a git repository as build input. However it's considered a bad practice to perform a full checkout of a repository since this may take a lot of time and can eat up a lot of disk space. This patch makes a full checkout explicit by adding an `allRefs` argument to `builtins.fetchGit` which fetches all refs if explicitly set to true. Closes NixOS#2409

Ma27 · 2020-12-22T11:19:17Z

@edolstra rebased, fixed the docs and also added a test :)

edolstra · 2020-12-22T15:57:37Z

Thanks, merged!

AleXoundOS · 2022-06-09T23:19:27Z

I know, but instead of falling back to master we could fall back to "all".

Originally I thought so too. But in practice this could leave missing commits in master branch unnoticed, until a feature branch gets deleted on remote.

domenkozar reviewed Jul 15, 2020

View reviewed changes

src/libfetchers/git.cc Outdated Show resolved Hide resolved

Ma27 force-pushed the git-rev-error branch 2 times, most recently from f49c707 to 9cdd68e Compare July 17, 2020 18:43

Ma27 force-pushed the git-rev-error branch from 9cdd68e to 9ea7349 Compare July 29, 2020 15:00

Ma27 force-pushed the git-rev-error branch from 9ea7349 to 2dd3944 Compare September 8, 2020 17:40

domenkozar requested a review from edolstra September 17, 2020 21:03

edolstra reviewed Sep 29, 2020

View reviewed changes

src/libfetchers/git.cc Outdated Show resolved Hide resolved

Ma27 force-pushed the git-rev-error branch from c6c0c2d to 9129c9e Compare September 29, 2020 15:28

edolstra reviewed Dec 21, 2020

View reviewed changes

src/libexpr/primops/fetchTree.cc Outdated Show resolved Hide resolved

edolstra reviewed Dec 21, 2020

View reviewed changes

Ma27 added 4 commits December 22, 2020 10:46

Document allRefs argument of builtins.fetchTree

e54971d

tests/fetchGit: test behavior of allRefs = true;

897ae23

Ma27 force-pushed the git-rev-error branch from 9129c9e to 897ae23 Compare December 22, 2020 11:18

edolstra merged commit f4a9fb6 into NixOS:master Dec 22, 2020

Ma27 deleted the git-rev-error branch December 22, 2020 17:31

cstrahan mentioned this pull request Apr 5, 2021

Private repositories nix-community/gomod2nix#3

Open

funketh mentioned this pull request Aug 3, 2021

Support all of the package locations that Stack (Pantry) does input-output-hk/haskell.nix#1188

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `allRefs = true` argument to `fetchGit` to explicitly perform a full checkout #3814

Add `allRefs = true` argument to `fetchGit` to explicitly perform a full checkout #3814

Ma27 commented Jul 14, 2020

bjornfor commented Jul 14, 2020

Ma27 commented Jul 14, 2020

Ma27 commented Jul 14, 2020

bjornfor commented Jul 15, 2020

Ma27 commented Jul 15, 2020

Ma27 commented Jul 17, 2020

Ma27 commented Jul 26, 2020

Ma27 commented Jul 29, 2020

Ma27 commented Aug 10, 2020

Ma27 commented Sep 8, 2020

Ma27 commented Sep 27, 2020

edolstra commented Sep 29, 2020

Ma27 commented Sep 29, 2020

Ma27 commented Oct 2, 2020

Ma27 commented Oct 9, 2020

edolstra commented Oct 9, 2020

Ma27 commented Oct 9, 2020

Ma27 commented Oct 28, 2020

Ma27 commented Nov 8, 2020

Ma27 commented Nov 15, 2020

Ma27 commented Nov 29, 2020

edolstra Dec 21, 2020

Ma27 Dec 22, 2020

Ma27 commented Dec 22, 2020

edolstra commented Dec 22, 2020

AleXoundOS commented Jun 9, 2022

Add allRefs = true argument to fetchGit to explicitly perform a full checkout #3814

Add allRefs = true argument to fetchGit to explicitly perform a full checkout #3814

Conversation

Ma27 commented Jul 14, 2020

bjornfor commented Jul 14, 2020

Ma27 commented Jul 14, 2020

Ma27 commented Jul 14, 2020

bjornfor commented Jul 15, 2020

Ma27 commented Jul 15, 2020

Ma27 commented Jul 17, 2020

Ma27 commented Jul 26, 2020

Ma27 commented Jul 29, 2020

Ma27 commented Aug 10, 2020

Ma27 commented Sep 8, 2020

Ma27 commented Sep 27, 2020

edolstra commented Sep 29, 2020

Ma27 commented Sep 29, 2020

Ma27 commented Oct 2, 2020

Ma27 commented Oct 9, 2020

edolstra commented Oct 9, 2020

Ma27 commented Oct 9, 2020

Ma27 commented Oct 28, 2020

Ma27 commented Nov 8, 2020

Ma27 commented Nov 15, 2020

Ma27 commented Nov 29, 2020

edolstra Dec 21, 2020

Choose a reason for hiding this comment

Ma27 Dec 22, 2020

Choose a reason for hiding this comment

Ma27 commented Dec 22, 2020

edolstra commented Dec 22, 2020

AleXoundOS commented Jun 9, 2022

Add `allRefs = true` argument to `fetchGit` to explicitly perform a full checkout #3814

Add `allRefs = true` argument to `fetchGit` to explicitly perform a full checkout #3814