supercontext — benchmarks for leCore retrieval, reproducible

Everything here runs against public endpoints and public datasets. No secrets, no private deps. Every number in the tweet came out of this code.

Built in one night, off three separate challenges:

• @shawmakesmagic — "Can you run this on mteb and see how it benchmarks in comparison?"
• @CottenIO — "Make sure to test adversarially with near matches/collisions in mind."
• Moose (leCore) — "you are ignoring leCore" (he was right, twice)

---

What's here

file	what it measures
suites/supercontext/	NIAH past the model's context window — HIT / MISS / UNRUNNABLE
mteb_lecore_hybrid.py	leCore on MTEB retrieval — BM25 / dense VSA / RRF hybrid
mteb_lecore.py	dense-VSA-only encoder (the first, worse attempt — kept for the record)
cost_bench.py	cost to index + time to first answer, leCore vs FAISS + embeddings
results/	raw JSON from every run quoted below

---

1. supercontext — past the window

context-bench measures retrieval inside a window the model can attend to, so its ceiling is the model's ceiling. Past that there is no axis: the control does not score badly, it stops existing.

So this suite scores three outcomes, never two:

HIT         answered, fact recovered
MISS        answered, fact not recovered   <- quality failure
UNRUNNABLE  the endpoint refused the body  <- capability ceiling

Folding UNRUNNABLE into MISS would let a memory system claim it "beat" a model that was never allowed to compete. The first thing this caught was our own runner doing exactly that — OpenRouter returns HTTP 200 with an error body for oversized prompts, and we were scoring it as MISS. A 200 is not a run.

gemini-2.5-flash, needle at start/middle/end, 3 seeds/cell:

tier	control	lecore	lecore tokens
60k	9/9 HIT	9/9 HIT	13,123
200k	9/9 HIT	9/9 HIT	25,101
500k	8/9 HIT, 1 UNRUNNABLE	9/9 HIT	29,833

control : HIT 26 | MISS 0 | UNRUNNABLE 1 | 6,676,718 prompt tokens
lecore  : HIT 27 | MISS 0 | UNRUNNABLE 0 |    68,057 prompt tokens

Past the window (n=6/cell):

tier	control	lecore
900k	5/6 HIT, 1 UNRUNNABLE	6/6 HIT
1M	0/6 — 6 UNRUNNABLE	6/6 HIT
2M	0/6 — 6 UNRUNNABLE	6/6 HIT
5M	0/6 — 6 UNRUNNABLE	6/6 HIT

control : HIT 5  | UNRUNNABLE 19 | 4,836,442 prompt tokens
lecore  : HIT 24 | UNRUNNABLE  0 |    80,300 prompt tokens

The token count is flat: 60k corpus → ~1,458 tokens read; 5M corpus → 3,401. Corpus grows 80×, what the model reads does not move.

OPENAI_BASE_URL=http://127.0.0.1:8899/v1 python suites/supercontext/run.py \
  --arm lecore --tiers 1m,2m,5m --positions start,middle,end
python suites/supercontext/sweep.py "60k,200k,500k" 3

---

2. MTEB — the honest answer to "how does it compare"

BM25-tier. Not embedding-tier.

task	leCore BM25+expand	published BM25
SciFact	0.6705	~0.665
ArguAna	0.4314	~0.315
NFCorpus	0.3167	~0.325
SCIDOCS	0.1577	~0.158

And the finding that cost us the most: the dense VSA arm is dead weight.

SciFact config	nDCG@10
dense VSA only (what our sidecar shipped)	0.4160
hybrid, w_dense 0.30	0.6057
hybrid, w_dense 0.15	0.6307
hybrid, w_dense 0.05	0.6596
BM25 alone	0.6679
BM25 + expand=True	0.6705

Monotone: every amount of our hand-rolled VSA encoder made retrieval worse. We were shipping the weakest of the three configurations.

leCore ships no text embedder by design — CAPABILITIES.md says the dense artifact is "the document side only (509 modules × 128d)" and free text "returns an honest None". Our item_vector() was an invention filling a gap leCore deliberately left open.

MTEB_TASKS=SciFact LECORE_MODE=bm25 LECORE_EXPAND=1 python mteb_lecore_hybrid.py

---

3. Cost to first answer — where it actually wins

Latency-only comparisons skip the part where a FAISS index does not exist until something embeds the corpus.

   tokens    docs |  leCore idx    q ms |  FAISS idx    q ms |  TTFA leCore  TTFA FAISS
   50,000     595 |       0.04s    0.2 |      6.67s    5.6 |       0.04s      16.05s
  200,000    2373 |       0.68s    0.7 |     24.62s   59.6 |       0.68s      29.82s
1,000,000   11847 |      25.78s    4.5 |    132.33s   30.8 |      25.79s     137.13s

5.3× faster to first answer at 1M. 6.8× faster per query. No model download, no GPU, no index build.

Only the harness is parallelised (corpus generation, chunking). BM25 fits single-threaded because that is how leCore ships it; the embedder gets its own batching because that is how it ships. Speeding up either side would make the comparison a lie about what a user gets.

SIZES=50000,200000,1000000 python cost_bench.py

---

What this does NOT show

• The adversarial near-match test has now been run, and we lose it. @CottenIO asked for it twice and he was right. Our filler shared no vocabulary with the query, so the needle was the only chunk containing the query terms — easy mode, and a dumb exact-substring counter scores precision@1 = 1.00 on it, identical to leCore. Plant decoys carrying the full query vocabulary and precision@1 collapses 1.00 → 0.19 at two decoys. At production top_k=8 the answer still reaches the model up to ~10 competing passages (recall@8 0.91) and breaks between 10 and 25 (0.22). See adversarial_bench.py and results/adversarial_k8.json. The honest consequence: the lexical stage cannot be the last stage — this needs a reranker over the top-k, and the retrieval claim must be stated as recall@k on low-confusability corpora, never precision@1.
• The control never degraded. gemini held 9/9 in every cell it could run. This is cost and reach, not accuracy rescue.
• Query-latency numbers disagree with @CottenIO's (he measured 518ms, we measure 0.2–4.5ms). We believe he benchmarked the deployed sidecar — HTTP hop plus the VSA encoder now shown to be dead weight — not leCore's BM25. Unresolved.
• n=9 per cell at 60k–500k, n=2–6 past it. Single runs, temperature 0.
• elizaOS's own harness could not be run — bun install fails on @elizaos/cloud-shared, @elizaos/plugin-remote-manifest, @elizaos/registry, none on public npm. The suite is written to run without the bridge, against any OpenAI-compatible endpoint. Numbers are ours, scored by our runner, using their methodology.

Bugs this found in our own stack

1. Non-overlapping chunks deleted facts. A 57-char needle straddling a 1200-char boundary appeared in zero of 1,584 chunks. leCore's chunk_text docstring had already called it: "a fact split across two chunks is retrievable from neither." 500k/start: 0/3 → 3/3.
2. The retrieval query was the whole tail — ~1,900 chars of filler plus the question, so filler chunks won at cosine 0.909 and the needle never made top-8.
3. A 200 is not a run (see above).
4. chunk_text's own giant-paragraph fallback is p[:max_chars] in a loop — fixed windows, the thing its docstring rejects. Measured over 86 offsets: blob input loses the needle 8/86 (9%), paragraphed input 0/86. Reported upstream.

MIT.