Checksum utilities, particularly sha1sum and sha256sum, slow compared to GNU coreutils

[jabl]

I tested the common checksumming utilities, and they seem significantly slower than the GNU ones. Note that the GNU coreutils tests were run on a bare metal Ubuntu 24.04 machine, and the uutils tests on a Ubuntu 26.04 VM running on the same HW, so some differences are expected, but this level would suggest significant optimization opportunities.

Results for GNU coreutils 9.4:

File: /tmp/checksum-bench-zi1p2vm0 (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: cksum (24.257 ms)

Utility         Best time      Ratio
---------- -------------- ----------
cksum           24.257 ms      1.00x
sha1sum        138.245 ms      5.70x
b2sum          176.584 ms      7.28x
md5sum         188.270 ms      7.76x
sha512sum      201.897 ms      8.32x
sha256sum      281.013 ms     11.58x

Running the same with uutils 8.0:

File: /tmp/checksum-bench-371wmgzt (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: cksum (33.813 ms)

Utility         Best time      Ratio
---------- -------------- ----------
cksum           33.813 ms      1.00x
b2sum          126.431 ms      3.74x
md5sum         237.955 ms      7.04x
sha512sum      301.289 ms      8.91x
sha1sum        482.523 ms     14.27x
sha256sum      963.138 ms     28.48x

One can see that uutils b2sum is actually faster than the coreutils one, most utilities are somewhat slower, but sha1sum and sha256sum in particular are a lot slower for the uutils versions.

[sylvestre]

what is the file size that you are using to test?
any args are provided? thanks

[oech3]

Do we use our logic for checksum instead of relying on external crates?
There is nothing we can do for external crates.

[jabl]

what is the file size that you are using to test? any args are provided? thanks

As mentioned in the output of the test wrapper, 100 MiB. No further command-line arguments given to the utilities except the path to the 100MiB test file.

[RenjiSann]

Do we use our logic for checksum instead of relying on external crates? There is nothing we can do for external crates.

Maybe we could gain some performance if there are some SIMD options we are not leveraging at the moment.
I'll give it a look

[xtqqczze]

Were the utilities compiled from source? There are significant checksum performance improvements when building with RUSTFLAGS="-C target-cpu=x86-64-v3", as discussed in #12003.

[oech3]

It is not a solution at here. There are many repositories doing same things for GNU coreutils.

[xtqqczze]

Ubuntu 26.04 now uses cksum from uutils.

@jabl Are you using the Ubuntu package build for these benchmarks? If so, is it the amd64v3 variant?

[jabl]

Ubuntu 26.04 now uses cksum from uutils.

@jabl Are you using the Ubuntu package build for these benchmarks? If so, is it the amd64v3 variant?

Yes, this was done with an out-of-the-box Ubuntu 26.04 install, with the uutils provided by Ubuntu. Quite sure it was just the standard amd64, nothing special was done to enable a amd64v3 variant.

[xtqqczze]

@jabl If you are running on compatible hardware you can upgrade to amd64v3 packages in Ubuntu 26.04:
https://documentation.ubuntu.com/release-notes/26.04/summary-for-lts-users/#architecture-variants-and-amd64v3

[scx1332]

ubuntu 24.04:

time ./coreutils-0.8.0-i686-unknown-linux-gnu/coreutils sha256sum random_file.bin
f07842e25aaae475cb66c3cc27738e69679030d724e0bea1d5d578057fb9dd8e random_file.bin

real 0m0.644s
user 0m0.561s
sys 0m0.082s
ubuntu@geode:~/sha256test$ time sha256sum random_file.bin
f07842e25aaae475cb66c3cc27738e69679030d724e0bea1d5d578057fb9dd8e random_file.bin

real 0m0.646s
user 0m0.567s
sys 0m0.079s

Time almost the same. I suggest you to compare speeds on the same machine. How can you compare results between bare metal and VM?