Speed up and harden cache fingerprinting: xxh3_128 + vectorized DataFrame hashing + collision fixes

[Dev-iL]

Follow-up to: #1616

What this does

Cache fingerprinting (hamilton/caching/fingerprinting.py) maps a Python value to a data_version string used in cache keys. This PR makes it faster and more correct, without weakening the collision-prevention guarantees added in #1616.

1. Swap the hash algorithm — md5/sha224 → xxhash.xxh3_128. All hashing now routes through a single _hash_bytes helper wrapping xxhash.xxh3_128(data).digest(), reusing the existing _compact_hash base64url encoding.

2. Vectorize the DataFrame paths (the real bottleneck — see benchmark):

• pandas: hash the hash_pandas_object(obj).values uint64 buffer in one shot instead of round-tripping through .to_dict() and an ordered per-row hash_mapping. Column names + dtypes (schema) are folded in; the path stays order-sensitive (the old docstring claiming row order "doesn't matter" was incorrect and is now fixed).
• polars: hash the hash_rows().to_numpy() buffer in one shot instead of .to_list() into a per-element hash_sequence loop. The schema_hash + row_hash combine from Include metadata in numpy/polars cache fingerprints to prevent collisions #1616 is preserved.

3. Close confirmed collisions. Primitives and bytes now carry a type tag, so 1, "1", b"1", 1.0, and "1.0" hash distinctly; pandas frames with identical values but different column names or dtypes no longer collide.

Any fingerprint change invalidates existing caches exactly once (cache miss → recompute, never a wrong result), which is what makes it safe to land the collision fixes alongside the algorithm swap.

Benchmark results

scripts/benchmark_fingerprinting.py fingerprints a 500,000-row, 3-column DataFrame, comparing the old per-row approach against the new vectorized path (best of 3 runs):

Path	Time
Old (per-row to_dict() loop)	~3,200–3,600 ms
New (vectorized buffer hash)	~210–260 ms
Speedup	~14–15×

The structural "no per-row Python loop" assertion is the hard correctness gate; the benchmark is corroborating evidence with a generous ≥5× floor to avoid timing flakiness.

Why xxh3_128 is a sound replacement for the longer sha224

The previous code mixed two digests: md5 (128-bit) for primitives/bytes and sha224 (224-bit) for sequences, mappings, and sets. Replacing the wider sha224 with a 128-bit digest is safe here for three reasons:

• Collision resistance is about digest width, not cryptographic strength. For a fingerprint, the only property that matters is the probability that two distinct inputs map to the same digest. For a well-distributed n-bit hash that's governed by the birthday bound (~2^(n/2)). These fingerprints are never a security boundary — there is no adversary choosing inputs to force a collision; inputs are ordinary pipeline values. So sha224's extra resistance to deliberate collision attacks (its reason for being longer) buys nothing in this use case.
• 128 bits is astronomically sufficient for cache keys. The birthday bound for a 128-bit digest is ~2^64 (≈1.8×10¹⁹) distinct values before a ~50% collision chance. A cache will never hold anywhere near that many fingerprints; the realistic collision probability is effectively zero. sha224's 2^112 headroom is far past the point of any practical difference for this workload.
• xxh3_128 is purpose-built for this. It is a fast, non-cryptographic hash with strong dispersion (passes the SMHasher quality suite), and at 128 bits it matches the width md5 was already trusted for in the same module — so the swap strengthens the former md5 paths' guarantees to par and keeps the former sha224 paths comfortably collision-safe, while removing the cryptographic-hashing overhead we were paying for no benefit.

Net: we trade unused cryptographic headroom for a large throughput win, with collision safety that remains far beyond what any cache will ever exercise.

Dependency & licensing

• Adds xxhash>=0.8.0 to core runtime dependencies (xxh3_128 was introduced in 0.8.0). Fingerprinting is imported eagerly via the caching adapter, so this is a hard dependency, not an optional extra.
• xxhash (the python-xxhash package) is BSD-2-Clause; its copyright and licence text are appended to LICENSE in the same style as the existing third-party (MIT databackend) entry.

Testing

• Pinned literal-digest tests recomputed against the new algorithm (run, not hand-written).
• New must-differ tests (cross-type primitives; pandas different column-names; pandas/polars different dtypes) and must-match tests (identical frames; list == tuple sequence equality).
• Full caching suite passes (115 non-polars tests). Polars-dependent tests are exercised on CI — they can't run in every local environment (the polars wheel crashes on hosts lacking certain CPU features), so they're guarded with pytest.importorskip.

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[jernejfrank]

Looks good, the speedup is amazing! I just have the one concern if invalidating existing caches is breaking in case some users relied on it for cachign some heavy computations.

[Dev-iL]

Looks good, the speedup is amazing! I just have the one concern if invalidating existing caches is breaking in case some users relied on it for cachign some heavy computations.

Valid concern! Several reasons why I think it's alright:

1. This is a followup to @skrawcz's PR that already invalidated a lot of caches, and since we didn't have a release in between - there's no significant penalty from introducing this change presently.
2. Some hashes genuinely need recomputing since they should point to distinct objects (where at the moment their cache is the same).
3. The next release will be the first under "apache", so users might expect (and accept) such changes as "the price of progress".

[jernejfrank]

Good point on #1616 , make sense to add this now!

[skrawcz]

@Dev-iL one thought -- should we make this optional?

[skrawcz]

I have a goal of basically having very minimal dependencies if possible

[Dev-iL]

See #1619 (comment)

[skrawcz]

I think we should change this so:

1. we can function without xxhash
2. if it's available we use that version, else we use the slower hashlib one

thoughts?

[Dev-iL]

• I can't really argue if this is how you/PPMC see the future of the library.
• A precedence for your suggestion exists in pandas.
• Has this concern been raised by users? As a user, I find it unintuitive having to specify an extra to gain performance.

I guess my only argument is - can this be postponed until we carve out the -core package?

[elijahbenizzy]

I'd prefer to keep it light -- this is the consistent pattern. But can you highlight the trade-offs/user impacts? We should have removed pandas and numpy the fact that they're there is tech debt hard to see adding another... hamilton[caching] is fine imo

[Dev-iL]

I'd prefer to keep it light -- this is the consistent pattern.

Here's a benchmark I ran in a recent discussion with @skrawcz:

Environment: clean Python 3.14.3 venv (uv venv), latest versions installed fresh. Each library imported in its own fresh interpreter subprocess; timing measured with time.perf_counter() around the import statement only (interpreter startup excluded). 20 runs per library.

Library	Version	Min (ms)	Median (ms)	Mean (ms)	Max (ms)
xxhash	3.7.0	0.52	0.64	0.86	4.94
numpy	2.4.6	89.88	99.20	102.47	125.50
polars	1.41.2	130.39	136.18	139.31	161.25
pandas	3.0.3	283.24	299.08	301.85	334.94

(sorted fastest → slowest by median)

Notes

• polars caveat: this CPU lacks AVX2/FMA, so the default polars 1.41.2 wheel crashed on import with SIGILL (illegal instruction). The number above is from the polars[rtcompat] runtime build (same 1.41.2 version) — the variant Polars recommends for older CPUs. On an AVX2-capable machine the default wheel would be used and its import time may differ slightly.
• pandas pulls in numpy (plus python-dateutil/six) as a dependency, which is why its import is roughly the cost of numpy plus pandas' own modules. The figures reflect each library's full import cost as a user would experience it, which is why they're measured one-per-process rather than sharing a warm interpreter.
• xxhash is a thin C-extension binding, hence the sub-millisecond import.

As you can see, xxhash is as light as it gets.

can you highlight the trade-offs/user impacts?

As mentioned in the PR description the hashing time is ~15x shorter with xxhash. I'd be happy to add more benchmarks if there's a specific use case you have in mind.

hamilton[caching] is fine imo

Stefan proposed this extra name too. One thing I don't understand is: does [caching] mean "fast caching" or "caching is possible"? If "slow caching" would be enabled without the extra, I'd say it is counterintuitive. Otherwise, are we ready to move the caching feature to an extra? I'm fine with that.