[PoC] fetchurl: use Software Heritage as a content addressable mirror

[veprbl]

This is a proof of concept implementation of the idea from #53653.

Can be tested like this:

with import ./. {};

fetchurl {
  # simulate a dead URL (we never packaged version 1.2.9)
  url = "gopher://example.com/not_a_thing/zlib-1.2.9";

  # base32 hash as commonly used in nixpkgs
  sha256 = "08dabvnaqis3jlhyzwvh0dmz4lsq5zsmdbyjjm4fgl8yycp31avk";
}

trying https://archive.softwareheritage.org/api/1/content/sha256:73ab302ef31ed1e74895d2af56f52f5853f26b0370f3ef21954347acec5eaa21/raw/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  593k  100  593k    0     0   144k      0  0:00:04  0:00:04 --:--:--  144k
/nix/store/3ynhg01hji29wsv1g177f5sqyx6zfr91-zlib-1.2.9

To make this really work we would need to figure out how to convert hashes without "nix-hash". Also this would require some kind of negotiation with Software Heritage, and, perhaps, we would need to set up some kind of caching proxy.

Obviously, do not merge this.

[asymmetric]

How does Guix do this?

[nlewo]

@asymmetric According to https://issues.guix.info/issue/33432, it seems they fallback to Software Heritage for their git fetcher. They can use the commit id as an identifier.
@veprbl Why using nix-hash is an issue (I was not able to find an another tool to do this actually)?

[veprbl]

@nlewo This explains a lot. When I originally made this PR I tested it with zlib and it so happened that they had that tarball. Later I tried to locate couple other tarballs on SH to no avail. It seems like they prefer to mirror whole repositories instead, and that would be a reason for guix to implement this for git fetcher.

The problem with using nix in fetchurl is that produces a circular dependency. It should not be impossible to implement the hash conversion in some other way (e.g. write it in bash).

[nlewo]

@veprbl it seems to be a pain to translate hash in bash! There are several hash types (md5, sha256, sha512) and the base32 encoding is not trivial (https://github.com/NixOS/nix/blob/master/src/libutil/hash.cc#L173)!
So, instead we could add a builtin such as hashToBase16 that takes a hash and returns its base16 representation.
But to start, I would suggest to break the cycle by ignoring the Software Heritage fallback for Nix itself. We could for instance add an extra arg enableSoftwareHeritageFallback ? true to fetchurl and override it to false for the Nix package. For all other packages, we could then use nix-hash for the hash translation.

[veprbl]

This should just work now (at least on darwin). Conversion is only implemented for sha256 hashes for now.

[domenkozar]

https://www.tweag.io/blog/2020-06-18-software-heritage/