Implement BLAKE2b checksums

[lukateras]

SHA256 is rather slow at checksumming large files (tangentially related to #619), would be nice to have something that is both fast and future-proof.

Use case would be to nix-hash large (> 4GB) files to check if already present in Nix store. This takes a while with SHA256, and would probably take at least 2x less time with BLAKE2b.

[lukateras]

I would like to send a patch if this is approved (no new dependencies).

[edolstra]

Well, we already have SHA-512, which is much faster than SHA-256 (at least in the current implementation).

[lukateras]

BLAKE2b is about 1.5x faster than SHA-512 (similar to margin between SHA-256 and SHA-512), while also being way simpler (compare https://github.com/openssl/openssl/blob/master/crypto/sha/sha512.c at 694 SLOC with https://github.com/openssl/openssl/blob/master/crypto/blake2/blake2b.c at 241 SLOC). See https://blake2.net/.

It was also introduced in coreutils, see b2sum(1) and http://lists.gnu.org/archive/html/coreutils/2015-05/msg00048.html.

(For anyone who wants to study the function, here's cleaned up source code: https://github.com/yegortimoshenko/sylpheed/blob/d7a91f82e5785992e3716529b63d8530560a5479/src/blake2b.c)

[xzfc]

Another promising candidate is BLAKE3 (which appeared recently in Jan 2020). It is 6x faster than sha512 on my machine on some random 91M file from /nix/store.

Benchmark

$ du -sh /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
91M	/nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi

$ hyperfine 'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi'
Benchmark #1: b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
  Time (mean ± σ):      45.8 ms ±   4.4 ms    [User: 142.5 ms, System: 10.4 ms]
  Range (min … max):    41.4 ms …  58.9 ms    62 runs
 
Benchmark #2: sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
  Time (mean ± σ):     274.6 ms ±  19.2 ms    [User: 258.6 ms, System: 15.8 ms]
  Range (min … max):   254.9 ms … 309.2 ms    11 runs
 
Summary
  'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' ran
    5.99 ± 0.72 times faster than 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi'

$ b3sum --version
b3sum 0.1.1

$ sha512sum --version
sha512sum (GNU coreutils) 8.31
Packaged by https://NixOS.org
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Ulrich Drepper, Scott Miller, and David Madore.

$ grep 'model name' /proc/cpuinfo | head -n1
model name	: Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz

[tnias]

Probably duplicate of #1133.

Gentoo uses blake2b since end of 2017. (source: wiki page on manifest files and mailing list annoucement)

Archlinux has blake2 support. (source: arch wiki PKGBUILD and b2sums in PKGBUILD(5))

[stale]

I marked this as stale due to inactivity. → More info

[figsoda]

blake3 seems like a good candidate

[stale]

I marked this as stale due to inactivity. → More info