v1.9.0
A weight-memory substrate (mmap + madvise + a span-residency cache) lifted into a
new leaf package, plus the payoff it unlocks: an int8 ANN index that can be queried
larger than RAM. The mechanism is the one goinfer proved demand-paging a 35B-A3B
MoE's experts; only the generic, model-free core moves here. All new surface is
Experimental (outside the 1.0 guarantee).
Added
-
mmap— new leaf package: read-only mapping + residency control. The
read-onlyMAP_PRIVATEmapping primitive thatannandembedeach kept a
private byte-identical copy of (to avoid anann→embededge) now lives once, in a
zero-dependency leaf both import:MapReadOnly/Unmap— the mapping itself.Advise(span, willNeed)—MADV_WILLNEED/MADV_DONTNEEDresidency hints.
Firm cap on Linux; on darwinWILLNEEDis an advisory prefetch and eviction is
an OS-discretion no-op; best-effort no-op on the BSDs/Windows.SpanCache[K]— a demand-signal-agnostic LRU of page-aligned spans bounded by a
byte budget:Addregisters a member's spans,Touchfaults it in and releases
the LRU tail to stay under budget. No model logic — the caller owns the demand
signal. Gated by an eviction unit test and a model-free property test that a
MADV_DONTNEED-released read-only mapping re-faults byte-identical.PageAlignedInterior,AvailableRAM,AutoBudgethelpers.
Leaf invariant: stdlib-only, except
golang.org/x/sys/unixon darwin only
(the stdlib has nomadvisewrapper there) — so it stays invisible to the core
dependency invariant on Linux, and cgo-free everywhere.!unixkeeps the existing
heap-read fallback. -
linalg.WeightMat.MappedSpan(base, end). Returns the page-aligned interior of
a weight's int8/int4 backing bytes iff they alias the given mapping, else nil
(f32/heap-backed → skip). The bridge from aWeightMattommap.SpanCache. -
ann.LoadFlatI8MmapPaged(path, budget)— query an int8 index larger than RAM.
The int8 code block is split into blocks that page in and out of the mapping
throughmmap.SpanCacheunderbudget(≤ 0 auto-selects ~half of available RAM);
cold blocks re-fault from the read-only mapping. Per-block scoring is
byte-identical to the resident whole-corpus scan (deterministic query quant +
independent row dots), so paging changes residency only, never results — gated by
a paged-equals-resident test that also asserts eviction count > 0. The default
LoadFlatI8Mmapis unchanged (paging is opt-in).FlatI8.PageStats()exposes the
cache's hit/miss/eviction counts. A paged index serializes concurrentQuery
calls (the pager is stateful) — the cap traded against cross-query parallelism.
Changed
annandembednow call themmapleaf instead of their local
mmapReadOnly/munmapcopies, which are deleted. Pure refactor — the existing
OpenSafetensorsMmap/OpenGGUFMmap/LoadFlatI8Mmapbehavior and tests are
unchanged.