Tracking issue for RFC 133: Git hashing and Git-hashing-based remote stores

[Ericson2314]

Info

• RFC 133: Git hashing and Git-hashing-based remote stores
• PR thread
• Milestone

Steps

Preperatory

• Shuffle ParseSink code in preparation for git hashing support #9024
• Factor out MemorySourceAccessor, implement missing features #9283
• Content addressing and adding to store cleanup #9325

Feature itself

• Git object hashing in libutil #9294

• Git object hashing in libstore #8918

  • At the store layer (manually added files, derivation outputs)

• Git hashing for Git fetching #9025
  -builtins.fetchGit and friends.

• Use StorePathDescriptor to fetching from CA-only stores #3754

  • Nix-agnostic content-addressing "stores"
  • That PR might be changed a lot.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixcon-governance-workshop/32705/9

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/65

[roberth]

Idea:

1. Store Object =~ Name + File System Object:
2. Git hashing can not hash an executable blob
3. Store Object =~ single-item Git directory (which includes the Name)

I know you don't like Name, but maybe this is helpful.

File under storeDir bad, name as directory?

Another idea is maybe paths should have been $storeDir/$hash/$name/$fhsLikeStuffOrWhatever. It'd prevent things like cp ${fooScript} $out/bin producing a command containing a hash, but it's not the only way to do it, and probably a little bit slower than the current store path format, and not worth the effort to make such a change. That specific problem is solved more easily by making Nixpkgs produce directories, and making writeScript return an outPath of "${drv.outPath}/bin/${name}".

[edef1c]

I don't feel confident that we should have more SHA1 in the store layer; if anything, I would like to move towards prohibiting SHA1 as a FOD hash. We can get the same UX benefits[1] by checking in SHA1 → NAR mappings into a lockfile somewhat automatically: Go, Cargo, and a number of other language package managers do this. If we don't, we're going to need an answer to what happens when we encounter collisions, and it's going to have to be an answer that doesn't throw out Nix's reproducibility guarantees. At minimum, we need to consider using SHA1DC[2] as the hash function, rather than SHA1 itself.

[1]: This at least captures, and strongly binds, the actual store path contents. Ideally, we have a phase separation between "I am capturing a new binding", and "I am using an existing binding, no malleability here" (ie, lockfile-only), but that may be somewhat of a lost cause at this point.
[2]: A counter-cryptanalytic hash function, equivalent to SHA1 for most inputs, but distinct for inputs that are likely to be intentional collisions (either adding more rounds for collision resistance, resulting in different outputs, or rejecting the input outright)

PS: I'm aware that there is an accepted RFC specifying this behaviour. I think that was a mistake, and collision resistance not being a listed concern in the final RFC or the comment thread reflects poorly on our level of rigour and community expertise.

[Ericson2314]

I think I could be OK just stabilizing sha256 git hashes, and hoping the rest of the world belatedly catches up. (Or stabilizing both, but banning SHA-1 alone in Nixpkgs.) I do want something "deeply used" in the store layer, though.

[Ericson2314]

#13543 Implements SHA-256 Git Hashing.