fetchgit: remove only .git folder

[domenkozar]

See commit for explanation

cc @rbvermaa

[mention-bot]

By analyzing the blame information on this pull request, we identified @timbertson, @bjornfor and @nbp to be potential reviewers

[dezgeg]

Doesn't this potentially change some hashes?

[domenkozar]

Yes, but there is no way around really.

On Mon, May 16, 2016, 16:06 Tuomas Tynkkynen notifications@github.com
wrote:

Doesn't this potentially change some hashes?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#15469 (comment)

[dezgeg]

Well, shouldn't they be changed via some scripts at the same time or builds will start failing left and right?

[domenkozar]

There are 1200 invocations, we should probably change those that fail now indeed.

[dezgeg]

Here's something hacky to try to detect them (I get only ~800 hits, though): https://gist.github.com/dezgeg/94b7895fb79bbe3c32cc9903956690cc

Run like: nix-instantiate --eval --strict --json ./find-fetchgits.nix --show-trace

[domenkozar]

@dezgeg nice! I'll give it a run.

[domenkozar]

@dezgeg this doesn't build the derivations. Just evaluating won't be enough

[dezgeg]

Yes, but you get the attr path for each thing you need to call nix-build --hash to calculate the new hash.

[domenkozar]

It's not useful because --hash just uses the substitute (which uses fixed hash which might be broken):

$ nix-build -A w3m.src --hash 
/nix/store/ih4fpapyy982jig4kj24x0rd677jmd24-w3m-e0b6e02

[domenkozar]

We could use that information to run nix-prefetch-git before/after the merge.

[dezgeg]

Well, that's not very useful indeed... But nix-build --check does work.

[dezgeg]

I wouldn't use nix-prefetch-git for this, it's a mess to get all the keepDotGit, includeSubmodules etc. in sync.

[domenkozar]

gnulib should fail right?

$ nix-build -A gnulib.src --hash 
/nix/store/7m5gabs2qqzal8f08klrqqbrcp00qm19-gnulib-92b60e6

[dezgeg]

It does:

» nix-build -A gnulib.src --hash --check
checking path(s) ‘/nix/store/7m5gabs2qqzal8f08klrqqbrcp00qm19-gnulib-92b60e6’
exporting http://git.savannah.gnu.org/r/gnulib.git (rev 92b60e61666f008385d9b7f7443da17c7a44d1b1) into /nix/store/0n0a49xpjzvjsfipzr4azwyrx8l3qwfc-gnulib-92b60e6
Initialized empty Git repository in /nix/store/0n0a49xpjzvjsfipzr4azwyrx8l3qwfc-gnulib-92b60e6/.git/
remote: Counting objects: 166900, done.        
remote: Compressing objects: 100% (24346/24346), done.        
remote: Total 166900 (delta 142542), reused 166827 (delta 142496)        
Receiving objects: 100% (166900/166900), 31.47 MiB | 2.00 MiB/s, done.
Resolving deltas: 100% (142542/142542), done.
From http://git.savannah.gnu.org/r/gnulib
 * [new branch]      EMACS_21_1_RC -> origin/EMACS_21_1_RC
<snip>
Switched to a new branch 'fetchgit'
removing `.git'...
warning: rewriting hashes in ‘/nix/store/7m5gabs2qqzal8f08klrqqbrcp00qm19-gnulib-92b60e6’; cross fingers
output path ‘/nix/store/7m5gabs2qqzal8f08klrqqbrcp00qm19-gnulib-92b60e6’ has r:sha256 hash ‘0sa1dndvaxhw0zyc19al5cmpgzlwnnznjz89lw1b4vj3xn7avjnr’ when ‘0xpxq8vqdl0niib961dnsrgjq6kbpyap6nnydzp15dvzfhzgz189’ was expected
error: build of ‘/nix/store/wdn8nk25l0dczjnwlr67f2yslzvij719-gnulib-92b60e6.drv’ failed

[domenkozar]

Ah!

[dezgeg]

BTW, the nix file I posted doesn't locate packages that aren't included in the channel. I suppose temporarily sprinkling recurseIntoAttrs a bit in all-packages.nix would help. I think in addition to the ones in ccd1029, melpa/emacs packages are not included.

[zimbatm]

maybe add -type d too ?

[googl]

-type d -delete would be more beneficial than piping to xargs and rm -rf, yes?

find "$dir" -name .git -type d -delete

[groxxda]

Building upon @dezgeg s expression I produced a list of the hashes
https://gist.github.com/groxxda/0f5ff6e5f905534acd9690c42a0d9b17

I added to pkgs/build-support/fetchgit/default.nix:
meta.passthru._debugGitHashes = { inherit url rev md5 sha256 leaveDotGit fetchSubmodules deepClone branchName name; };

and altered find-fetchgits.nix to check for this variable and output a nix-prefetch-git command based upon them: https://gist.github.com/groxxda/c205a41f5e70edac3c1bd731ab943d36

then some shell hacking to fetch everything and generate the output.

If it's of use, I'll post the shell snippets so someone else can run everything and we can compare the hashes. A simple sed s/old/new/ on the nixpkgs repo should then be enough to update the hashes.

[domenkozar]

@groxxda let's do it :)

[groxxda]

https://gist.github.com/groxxda/669e350b5494a94e093d0262221bf3e3 ugh that's some ugly code ☺️ hope it works for you @domenkozar

We still miss a good amount of invocations as dezgeg pointed out.. Also no vim plugins picked up 😞

I opened an issue for the packages that have a src that can't be fetched: #15558

[copumpkin]

@groxxda that's a nice script. Also seems like it could be nice to figure out which fetchgit invocations could be replaced by fetchFromGitHub (I expect many of them)

[groxxda]

👍 replacing fetchgit with fetchFromGitHub - now would be the perfect time
Getting the replace right would be a bit harder though and we still lack a nix-prefetch-fromgithub so I don't know an easy way to automatically get the hashes (maybe generate the fetchFromGitHub derivation with a fake hash, build it and parse the error to get the real hash?)

[vcunat]

@groxxda: there's this new command nix-prefetch-url -A hello.src that works on anything.

[groxxda]

Thanks @vcunat
I took a look at replacing the github fetchgits but theres another problem: fetchgit defaults to fetchSubmodules = true, and only 3 fetchgits set this to false. Since git-archive (and thus the github archive api) only pack the main repository without submodules we may run into problems with a mass-replace. We would have to check the repository if it contains submodules and only if it doesn't we can replace with fetchFromGitHub..

Edit: just noticed #15559 addresses this issue 👏

[domenkozar]

There might be another change needed to fetchgit (remove the current naming of produce .drv so it matches new builtins.fetchgit).

Should we revert this and do both changes at the same time together with hash updates?

[groxxda]

Is the builtin meant to replace pkgs.fetchgit some day? Then it will have to support submodules as well I guess.
In my opinion naming after url & rev is good, even if it triggers unneeded building when repositories are renamed. So the primop should be adjusted accordingly.

[domenkozar]

No, builtins.fetchgit (in current Nix master branch) fetches from git without checking hash. Useful for CI/Hydra and such. pkgs.fetchgit requires a hash, but it would be really useful if those those functions would yield the same hash for same git revision

[vcunat]

Could we not use the builtin to fetch without checking into $out... and finishing the derivation makes nix check the recursive hash? (I'm just guessing.)

[garbas]

vimplugins hashes fixed

[dezgeg]

I pushed https://git.io/voeqV which is a superset of https://gist.githubusercontent.com/groxxda/0f5ff6e5f905534acd9690c42a0d9b17/raw/0e289d419c133542ceca9ca0ed97300b0dbb77fa/broken-git-hashes.json. I don't think that still catched them all, though.