Skip to content

mbo/hash: known-answer tests + SMHasher3 include-order fix; verify_digest_test bzl rule - #244

Merged
helly25 merged 2 commits into
mainfrom
hash-kat-smhasher-fix
Jul 10, 2026
Merged

mbo/hash: known-answer tests + SMHasher3 include-order fix; verify_digest_test bzl rule#244
helly25 merged 2 commits into
mainfrom
hash-kat-smhasher-fix

Conversation

@helly25

@helly25 helly25 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

The SMHasher3 build fix and the known-answer tests for the in-house hashes, in one commit (follows #243).

SMHasher3 include-order fix

smhasher3/mbohash.cpp included Hashlib.h before Platform.h. Platform.h defines seed_t and the FLAG_IMPL_*/FLAG_HASH_* enums the REGISTER_HASH macros rely on, so the reversed order put every $.field/flag out of scope and the reproducible build failed. This was the real cause behind the "build pin is broken" note from #243 (a misdiagnosis - the pin 6ab4343 is correct); that measurements/README "Open items" note is corrected here.

The include order is now pinned behind a // clang-format off guard, because SortIncludes alphabetizes Hashlib.h first and silently reverts it. Re-verified in a fresh container build (build_smhasher3.sh -> gcc:13): mumbo-64, jumbo-128, dumbo-64 all PASS.

Known-answer tests for the in-house mumbo/jumbo and dumbo

hash_test_vectors_gen emits hash_test_vectors.inc as one field-named constexpr struct array (reads like textproto, stays a compile-time header, no runtime parser). Regenerate intentionally on an algorithm change; the values are not stable across library versions.

Integrity is enforced in three independent layers:

  • InHouseVectors (hash_test) re-derives every vector from the live algorithm, and per-algorithm CountKatAlgo(...) >= 20 asserts prove every in-house algorithm is covered with enough cases (all dispatch tiers, seeded and unseeded) - not an arbitrary total.
  • hash_test_vectors_up_to_date_test (diff_test) compares the committed .inc against a freshly generated one, so a forgotten regeneration after an algorithm or generator change fails.
  • hash_test_vectors_digest_test (verify_digest_test, below) checks the committed hash_test_vectors.inc.sha256 sidecar - an independently verifiable artifact anyone can re-check with stock sha256sum -c.

New rule: //mbo/digest:digest.bzl verify_digest_test

Given an algorithm and files mapped to a saved digest - either digests (a checksum sidecar file, preferred and externally verifiable) or checksums (an inline hex digest in the BUILD file) - it re-checks each file with //mbo/digest:digest --check, so a file may only change when its saved digest is updated in the same commit. Self-tested in mbo/digest against a fixture covering both forms.

Follow-up

With the plugin building reproducibly again, the authoritative smhasher.json can be produced by run_measurements.py on a clean main checkout after this merges (the last "Open items" entry).

helly25 added 2 commits July 10, 2026 08:04
…gest_test bzl rule

SMHasher3 plugin: include Platform.h before Hashlib.h (Platform.h defines
seed_t and the FLAG_IMPL_*/FLAG_HASH_* enums the REGISTER_HASH macros need;
this is the order SMHasher3's own hashes use). Pinned behind a clang-format
guard, since SortIncludes alphabetizes "Hashlib.h" first and silently reverts
it. Re-verified in a container build: mumbo-64, jumbo-128, dumbo-64 all PASS.

Known-answer vectors: hash_test_vectors_gen emits hash_test_vectors.inc as one
field-named constexpr struct array (reads like textproto, stays a compile-time
header, no runtime parser). hash_test's InHouseVectors re-derives every vector
from the live algorithm, and per-algorithm CountKatAlgo(...) >= 20 asserts prove
every in-house algorithm is covered with enough cases (all dispatch tiers,
seeded and unseeded) instead of asserting an arbitrary total.

Integrity in three independent layers:
- diff_test: committed .inc vs a freshly generated one (fails on a forgotten
  regeneration after an algorithm or generator change).
- verify_digest_test: the committed hash_test_vectors.inc.sha256 sidecar, an
  independently verifiable artifact (stock `sha256sum -c` or `digest --check`).
- InHouseVectors: the runtime value check above.

New reusable rule //mbo/digest:digest.bzl verify_digest_test: given an algorithm
and files mapped to a saved digest, either `digests` (a checksum sidecar file,
preferred and externally verifiable) or `checksums` (an inline hex digest), it
re-checks each file with `//mbo/digest:digest --check`, so a file may only change
when its saved digest is updated in the same commit. Self-tested in mbo/digest
against a fixture covering both forms.

Also corrects the measurements/README "Open items" note (the "SMHasher3 build pin
is broken" misdiagnosis): the pin 6ab4343 is correct, and the include-order fix
here is what unblocks a committed authoritative smhasher.json.
GCC compiles hash_test with -Werror=comment; a '//' line ending in '\' (the
regen command wrapped across lines) is a multi-line-comment warning and failed
CI. The generator now emits the command unwrapped (clang-format re-wraps it at a
space, '>'-terminated, not a backslash), and its own header comment is fixed the
same way. Regenerated hash_test_vectors.inc + .sha256. Verified with
clang++ -Wcomment -Werror; macOS clang did not flag it originally.
@helly25
helly25 requested a review from Fab-Cat July 10, 2026 07:47
@helly25
helly25 enabled auto-merge (squash) July 10, 2026 08:11
@helly25
helly25 merged commit 3e7ce37 into main Jul 10, 2026
39 checks passed
@helly25
helly25 deleted the hash-kat-smhasher-fix branch July 10, 2026 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants