BLAKE3 digest function

[tylerwilliams]

This PR adds the BLAKE3 digest function to Bazel. This new digest type is already supported in the remote API.

Rather than re-implementing the BLAKE3 algorithm in Java, JNI is used to call the C language implementation. The advantage of using the C implementation is that native instructions like AVX2/NEON can be used on x86_64/arm, which greatly speeds up hashing, especially for large files.

Performance depends on the build, but is generally similar to or faster than SHA256 (the default). Faster hashing is especially useful in situations where Bazel's digest is unavailable but results already exist on disk: this is common during CI runs, like on Github Actions.

Here's an example of slow iOS build that's improved with this change:

tylerw@blokus integration % hyperfine -w 1 --prepare 'blake3bazel --digest_function=SHA256 shutdown' 'blake3bazel --digest_function=SHA256 build //iOSApp' --prepare 'blake3bazel --digest_function=BLAKE3 shutdown' 'blake3bazel --digest_function=BLAKE3 build //iOSApp'
Benchmark 1: blake3bazel --digest_function=SHA256 build //iOSApp
  Time (mean ± σ):     31.935 s ±  0.339 s    [User: 0.018 s, System: 0.023 s]
  Range (min … max):   31.333 s … 32.480 s    10 runs

Benchmark 2: blake3bazel --digest_function=BLAKE3 build //iOSApp
  Time (mean ± σ):      9.284 s ±  0.434 s    [User: 0.019 s, System: 0.023 s]
  Range (min … max):    9.025 s … 10.484 s    10 runs

Summary
  'blake3bazel --digest_function=BLAKE3 build //iOSApp' ran
    3.44 ± 0.17 times faster than 'blake3bazel --digest_function=SHA256 build //iOSApp'
tylerw@blokus integration %

Before:

After:

Here's an example of a linux build on a modern machine that's similar / slightly faster:

tylerw@lunchbox:~/buildbuddy$ cd ../bazel
tylerw@lunchbox:~/bazel$ hyperfine -w 1 --prepare 'blake3bazel --digest_function=SHA256 shutdown' 'blake3bazel --digest_function=SHA256 build //src:bazel-bin --java_runtime_version=remotejdk_11' --prepare 'blake3bazel --digest_function=BLAKE3 shutdown' 'blake3bazel --digest_function=BLAKE3 build //src:bazel-bin --java_runtime_version=remotejdk_11'
Benchmark 1: blake3bazel --digest_function=SHA256 build //src:bazel-bin --java_runtime_version=remotejdk_11
  Time (mean ± σ):      3.015 s ±  0.044 s    [User: 0.004 s, System: 0.003 s]
  Range (min … max):    2.939 s …  3.078 s    10 runs

Benchmark 2: blake3bazel --digest_function=BLAKE3 build //src:bazel-bin --java_runtime_version=remotejdk_11
  Time (mean ± σ):      2.984 s ±  0.043 s    [User: 0.006 s, System: 0.001 s]
  Range (min … max):    2.935 s …  3.066 s    10 runs

Summary
  'blake3bazel --digest_function=BLAKE3 build //src:bazel-bin --java_runtime_version=remotejdk_11' ran
    1.01 ± 0.02 times faster than 'blake3bazel --digest_function=SHA256 build //src:bazel-bin --java_runtime_version=remotejdk_11'
tylerw@lunchbox:~/bazel$

The change is already somewhat large, so I wanted to start with the minimum useful bits here, but the performance can likely be improved further by parallelizing and or optimizing how data transits the JNI interface.

N.B. If you are applying this change as a patch to do your own benchmarking, make sure you build bazel with -c opt :)

[chiragramani]

Thank you @tylerwilliams for putting this up! this will be a huge performance win in all our iOS builds as well!

[chiragramani]

The addresses #18614 really well. performance win is huge!

Repro in the above issue
On the latest run, the contribution of actuallyCompleteAction
Before: 14.8 seconds

After: 3s 177ms

On Uber's Rider app,
Linking stage actuallyCompleteAction contribution: 6s 676ms -> 1s 296ms
Bundling stage actuallyCompleteAction contribution: 7s 297ms -> 1s 866m

really happy to see the biggest bottlenecks being addressed by this PR! 🙌🏼

[coeuvre]

Nice improvement! I am happy to review this.

To make it easier to review, test and import, I suggest we split this large PR into multiple small ones (and we have to if one PR contains both non-thirdparty and thirdparty changes):

1. PR that includes update to REAPI.
2. PR that import sBLAZE3 c implementation.
3. PR that introduces BLAZE3 hash function to Bazel's vfs.
4. and PR that integrate BLAZE3 with remote execution.

What do you think?

[tylerwilliams]

Nice improvement! I am happy to review this.

To make it easier to review, test and import, I suggest we split this large PR into multiple small ones (and we have to if one PR contains both non-thirdparty and thirdparty changes):

1. PR that includes update to REAPI.
2. PR that import sBLAZE3 c implementation.
3. PR that introduces BLAZE3 hash function to Bazel's vfs.
4. and PR that integrate BLAZE3 with remote execution.

What do you think?

Thank you for offering to review this, that sounds great, and your proposed splits seem good too.

I sent the first two smaller CLs here:

1. Add BLAKE3 digest function to remote_execution.proto #18681
2. Add BLAKE3 source code to third_party #18682

[tylerwilliams]

The last two CLs related to this work are:

• Simplify blake3 hasher and improve performance #19041
• Ensure namedSetOfFiles URIs specify blob type correctly #19044

I will close this out after they are in.