Skip to content

Neural-inference cache (declined)

Choose a tag to compare

@BertilBraun BertilBraun released this 06 Sep 16:28
· 1301 commits to master since this release

An upper-bound measurement of neural-inference repetition in chess self-play, used to decline
building an inference cache. Preserved here because it is not merged to master.

Decision

A neural-inference cache does not deserve implementation priority for this workload.

The measurement

The decision-relevant experiment reproduced the actual mixed search policy — 800-search full requests
at a deterministic 25% Bernoulli rate, 150-search fast requests otherwise, 60% retained root visits
before each full search, matching SelfPlayWorker.run_batch. 512 distinct random legal starting
positions over six measured moves, two matched seeds with reversed run order, production search
constants and root noise.

Parallel searches Inference positions Potentially avoidable Upper-bound repeat rate Same-batch repeats Searches/s
1 1,824,486 64,741 3.5485% 0 29,420.6
2 1,826,618 64,470 3.5295% 1 38,222.0 (+29.9%)

~3.5% gross opportunity, before paying any cache cost — finite capacity, lookup,
synchronization, output storage and transfer. Same-batch repetition was effectively zero: all repeats
were prior-batch. The result is insensitive to parallel-search count.

Why the earlier, more exciting numbers were thrown out

An earlier harness reported 37.6% repetition at high budget and 98.5% at low budget. Those are
rejected methodology evidence and must not drive the decision: that harness used up to 64
parallel searches, cycled a fixed 50-opening suite, used evaluation-like search settings, and
repeatedly restarted completed games from the same openings. It did not represent project self-play.

The correction is the main value of this branch — an appealing result that did not survive being
measured against the real workload.

Incidental finding that did survive

parallel_searches: 2 improved aggregate search throughput +29.9% over 1 in two matched seeds
with reversed run order (+33.2% and +26.6% individually), with identical instrumentation in both
arms. It created no meaningful cache opportunity, but it is a throughput result in its own right.

Full data, environment and checksums:
documentation/benchmarks/inference-cache-hit-rate-20260818/