tests::bytes test fails on sparc64

[glaubitz]

This is reproducible on sparc64 on Linux:

glaubitz@cfarm202:~/rustc-hash$ cargo build --release
    Updating crates.io index
     Locking 14 packages to latest compatible versions
      Adding rand v0.8.6 (available: v0.10.1)
   Compiling rustc-hash v2.1.2 (/home/glaubitz/rustc-hash)
    Finished `release` profile [optimized] target(s) in 1.10s
glaubitz@cfarm202:~/rustc-hash$ cargo test
   Compiling rustc-hash v2.1.2 (/home/glaubitz/rustc-hash)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 4.89s
     Running unittests src/lib.rs (target/debug/deps/rustc_hash-d513507c1662bed2)

running 7 tests
test seeded_state::tests::cloned_seeded_states_are_equal ... ok
test seeded_state::tests::different_states_are_different ... ok
test seeded_state::tests::same_seed_produces_same_hasher ... ok
test tests::unsigned ... ok
test tests::signed ... ok
test tests::bytes ... FAILED
test tests::with_seed_actually_different ... ok

failures:

---- tests::bytes stdout ----

thread 'tests::bytes' (199963) panicked at src/lib.rs:463:9:
assertion `left == right` failed
  left: 6513278015018369270
 right: 17606491139363777937
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::bytes

test result: FAILED. 6 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--lib`
glaubitz@cfarm202:~/rustc-hash$

Access to this machine is possible through the GCC Compile Farm.

[Noratrieb]

Interesting, we've had endianness issues before but we now run Mips in CI through Miri, and it passes: https://github.com/rust-lang/rustc-hash/actions/runs/23682425453/job/68996488544#step:4:45

Can you reproduce the issue when running the tests through Miri with the same target? (on any host)

[glaubitz]

It also passes on ppc64 and s390x, for example, which are both big-endian. Maybe it assumes that sparc64 is little-endian for some reason?

[Noratrieb]

Ah, the byte hashing code deliberately behaves differently on Sparc (and 32-bit) because Sparc lacks 64-bit wide mul:

rustc-hash/src/lib.rs

Line 225 in fdb275c

// The following code path is only fast if 64-bit to 128-bit widening

In that case the tests should be adjusted to handle that, feel free to make a PR

[glaubitz]

I'm not really familiar with the code base. Could you suggest an appropriate change?

[glaubitz]

I'm not 100% sure whether I understand what should be adjusted. Should there be just other expected test results for sparc64? Shouldn't the hash function return the same result for all 64-bit targets?

[Noratrieb]

Shouldn't the hash function return the same result for all 64-bit targets

I don't think thats really an intended property of the hash, it makes no guarantees about its output (cc @orlp).

I am not fully sure what the best way to resolve this is. We could of course just add a cfg to the tests and hardcode more numbers. I'm not sure how I feel about that though. Maybe it makes more sense to just remove the hardcoded tests.

[orlp]

The problem is a mismatch between how the actual implementation switches between the 64-bit and 32-bit implementation:

rustc-hash/src/lib.rs

Lines 233 to 241 in fdb275c

	if cfg!(any(
	all(
	target_pointer_width = "64",
	not(any(target_arch = "sparc64", target_arch = "wasm64")),
	),
	target_arch = "aarch64",
	target_arch = "x86_64",
	all(target_family = "wasm", target_feature = "wide-arithmetic"),
	)) {

and how the test switches:

rustc-hash/src/lib.rs

Line 375 in fdb275c

const B32: bool = cfg!(target_pointer_width = "32");

Shouldn't the hash function return the same result for all 64-bit targets

No, that's not intended. The intention is to provide the fastest hash on all platforms. If the 64-bit widening multiply is unavailable a different implementation is chosen. Just because a platform has a 64-bit usize does not mean it has the 64 x 64 -> 128 multiply.

[glaubitz]

OK, so I have tried to adapt the conditional for the tests but that failed. However, dropping sparc64 from the conditional helps:

glaubitz@cfarm202:~/rustc-hash$ uname -a
Linux cfarm202 6.17.0-rc5+ #1 SMP Fri Sep 12 20:37:32 UTC 2025 sparc64 GNU/Linux
glaubitz@cfarm202:~/rustc-hash$ git diff
diff --git a/src/lib.rs b/src/lib.rs
index b3270c3..92986b5 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -233,7 +233,7 @@ fn multiply_mix(x: u64, y: u64) -> u64 {
     if cfg!(any(
         all(
             target_pointer_width = "64",
-            not(any(target_arch = "sparc64", target_arch = "wasm64")),
+            not(any(target_arch = "wasm64")),
         ),
         target_arch = "aarch64",
         target_arch = "x86_64",
glaubitz@cfarm202:~/rustc-hash$ cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running unittests src/lib.rs (target/debug/deps/rustc_hash-d513507c1662bed2)

running 7 tests
test seeded_state::tests::cloned_seeded_states_are_equal ... ok
test seeded_state::tests::same_seed_produces_same_hasher ... ok
test seeded_state::tests::different_states_are_different ... ok
test tests::bytes ... ok
test tests::signed ... ok
test tests::unsigned ... ok
test tests::with_seed_actually_different ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests rustc_hash

running 3 tests
test src/lib.rs - FxBuildHasher (line 338) ... ok
test src/lib.rs - (line 5) ... ok
test src/seeded_state.rs - seeded_state::FxSeededState (line 13) ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.90s

glaubitz@cfarm202:~/rustc-hash$

[glaubitz]

SPARC actually supports 64-bit to 128-bit widening multiplication starting with SPARC T4. Maybe we should just remove the conditional for sparc64, so that newer SPARC machines can make use of umulxhi through the help of gmp.

Looking at the generated assembly on sparc64, it will just call __multi3 which may implement the multiplication with umulxhi when the code is generated with -mcpu=niagara4 on at least GCC.

[karcherm]

I'm sometimes coaching @glaubitz in Open Source development. My 2cts:

We do not just have two functions and a bad switch, but we instead have three hash functions, because there are two 32/64 bit distinctions:

One is in

rustc-hash/src/lib.rs

Lines 76 to 79 in fdb275c

	#[cfg(target_pointer_width = "64")]
	const K: usize = 0xf1357aea2e62a9c5;
	#[cfg(target_pointer_width = "32")]
	const K: usize = 0x93d765dd;

where a different multiplication constant is defined for 32-bit hashes on 32-bit systems and 64-bit hashes on 64-bit systems. The constant k is used in

rustc-hash/src/lib.rs

Lines 121 to 126 in fdb275c

	impl FxHasher {
	#[inline]
	fn add_to_hash(&mut self, i: usize) {
	self.hash = self.hash.wrapping_add(i).wrapping_mul(K);
	}
	}

. This is the core implementation of FxHasher, which uses an usize value as accumulator.

The other one is the one quoted by @orlp in

rustc-hash/src/lib.rs

Lines 233 to 241 in fdb275c

	if cfg!(any(
	all(
	target_pointer_width = "64",
	not(any(target_arch = "sparc64", target_arch = "wasm64")),
	),
	target_arch = "aarch64",
	target_arch = "x86_64",
	all(target_family = "wasm", target_feature = "wide-arithmetic"),
	)) {

, which is a part of multiply_mix which is used in hash_bytes. hash_bytes always returns a 64-bit hash code for a byte sequence, but uses one of two different algorithms. hash_bytes is used a pre-hasher in FxHasher if a byte sequence is to be added to the hash state, as seen in

rustc-hash/src/lib.rs

Lines 130 to 133 in fdb275c

	fn write(&mut self, bytes: &[u8]) {
	// Compress the byte string to a single u64 and add to our hash.
	self.write_u64(hash_bytes(bytes));
	}

. The 64-Bit output is added using write_64, which is also split on accumulator size:

rustc-hash/src/lib.rs

Lines 151 to 155 in fdb275c

	fn write_u64(&mut self, i: u64) {
	self.add_to_hash(i as usize);
	#[cfg(target_pointer_width = "32")]
	self.add_to_hash((i >> 32) as usize);
	}

.

So we have:

• A 32-bit FxHasher. That one always uses the byte hasher without a 64x64 to 128 multiplication
• A "typical" 64-bit FxHasher. That one uses the byte hasher including the 64x64 to 128 multiplication
• A "light" 64-bit FxHasher. This one is identical to the "typical" 64-bit FxHasher (and clearly different from the 32-bit Fxhasher), but it uses the byte hasher that is typically only used in conjunction with teh 32-bit FxHasher which avoids the 64x64 to 128 multiplication.

@Noratrieb is spot on that we would need to add a third set of expected values for all test cases that include byte sequences. Furthermore, the it seems to make sense to ask the question whether it makes sense to test a hash implementation with the requirement: "Needs to be really fast, and provide reasonable low collision rates; the specific hash values not matter" against * specific reference values*. I did not dig deep enough into rustc to know whether this hasher is used for the hash codes in the Rust symbol mangling, but if it is, that hash code is part of the ABI on a platform, and I think it does make sense to protect agains accidental ABI breaks.