Corpus Hub, next: one line in, answer out. Ship indexed packs, not JSON.

[xerj-org]

What we learned

We measured the hub the way an agent meets it: start from llms.txt, nothing installed. It works, every agent found and used it. But it costs about twice what a normal agent pays to git clone and grep, because the hub ships a JSON pointer and the laptop does the cloning and the indexing. Nine to twelve steps where grep needs two. Numbers at the end.

Target experience

One line. No corpus names, no install steps, no indexing.

xerj "what is the default idle timeout in quinn's transport config"

quinn  pin 2026-10-12 (1 day old)  Apache-2.0/MIT  quinn-proto/src/config.rs:412
pub(crate) const DEFAULT_IDLE_TIMEOUT: Duration = Duration::from_secs(30);
answered from: hub pack, cached locally

First time: answered through the hosted hub right away, while the pack for that corpus downloads in the background. Every time after: local, under a second, works offline. The corpus is never named in the question and always named in the answer.

For agents without a shell the same thing is one MCP tool call.

What ships

Not JSON manifests. Pre-indexed packs: the engine's own segment bundles with ranking, plus a definitions table, built once in CI per corpus, signed, pulled on demand, never re-indexed on the laptop. The registry stays a git branch of manifests, because that is what makes pins reviewable; the packs are built from it.

Development plan

Each step says what to build and when it is done. Do them in order.

Step 1. Hosted hub node and remote MCP

• Run one node with every live corpus indexed. The reference-coding node is the pattern.
• Endpoints, all read-only:
  • GET /api/v1/catalogue.json
  • GET /api/v1/def/{slug}/{symbol}
  • GET /api/v1/code/{slug}?q=&k=&max_tokens=
  • GET /api/v1/search?q=&k= across all corpora, results grouped by corpus
• Every answer carries: corpus, pin_sha, pin_date, age_days, licence, file, line, text, value, score, truncated.
• The slug must come from the catalogue; the client never names an index.
• Cache identical queries; rate limit per key with an anonymous budget.
• Remote MCP server with three tools: xerj_hub_find, xerj_hub_def, xerj_hub_code.
• Done when: one curl returns file, line, value and pin for an exact symbol in any live corpus; the MCP tool list shows the three tools; p95 under 500 ms on the reference machine.

Step 2. Packs

• Builder, run in CI per corpus when its upstream commit changes: corpus add at the pin, corpus index into a builder node, export the index as the engine's bundle format (segments/*.bundle, snapshot.json, meta/ with schema and mapping), extract defs.jsonl, files.jsonl, sections.jsonl, write pack.json (slug, pin_sha, pin_date, builder_version, format_version 1, byte and record counts), SHA256SUMS, ed25519 signature.
• Publish as a GitHub Release asset pack-<slug>-<pin8>.tar.zst with the signature beside it. Same path pack-publish uses today.
• Client: xerj corpus pull <slug>: read the catalogue, resolve the latest pin, download, verify signature and checksums before opening anything, hydrate into the data directory through the bundle hydrate path, register the corpus so queries see it.
• Done when: on a fresh machine xerj corpus pull lz4 then one def query gives the same answer as the hosted node, with no clone and no corpus index; a pack with one changed byte is refused.

Step 3. Definitions table

• At build time, one line per definition: symbol, kind, lang, file, line, text, value (value for constants and macros). For prose corpora one line per section: heading, file, line_start, line_end.
• Backs the def endpoint and the local --def mode. A symbol found in several corpora returns all of them, each with its corpus and file.
• Done when: the 50-question benchmark suite scores 50/50 in def mode, one call per question, 80 output tokens or fewer per answer.

Step 4. Catalogue

• Generated from the registry on every corpus-hub merge. Per corpus: slug, domain, topics, ask_for, pin_sha, pin_date, licence_class, pack_bytes, pack_url, sig_url, refresh_cadence, quality_grade, dependency_names[], term_sketch (Bloom filter over identifiers, base64). Signed.
• Client caches it at ~/.xerj/hub/catalogue.json, refreshes with an ETag at most once a day and on xerj hub sync.
• Done when: the catalogue is under 2 MB for 515 corpora, a stale copy refreshes, a bad signature is refused.

Step 5. The router and xerj "<question>"

• Order of evidence: local user indexes first; then project context (Cargo.lock, package.json, go.mod, requirements.txt, pyproject.toml → dependency names → dependency_names in the catalogue); then hosted federated search when online and allowed; then the local term sketches.
• High confidence: answer. Ambiguous: show the top three candidates with one line each and ask. Never answer silently from a corpus that might be wrong.
• Fetch policy: pull automatically up to 50 MB; above that answer through the hosted route and print xerj hub pull <slug> with the size; adapt-with-attribution pulls silently, approach-only pulls with a warning line, excluded never pulls; disk budget 4 GB, least recently used pruned.
• Hosted now, local next: answer through the hosted route while the pack downloads in a detached process whose PID and log live under ~/.xerj/hub/.
• Flags: --corpus, --offline, --no-fetch, --hosted=on|off|auto, --def, --how, --files, --max-tokens, --json.
• Output is the block shown above. It always includes corpus, pin, age, licence and where it was answered from.
• Done when: on a labelled routing set of at least 100 questions over 20 corpora, top-1 is 90% or better with project context and 80% without; an agent transcript shows one turn per question.

Step 6. Lockfile and freshness

• xerj.lock in the project root records corpora and pins; xerj hub update shows the diff; every answer records the pin it used.
• A question containing "current" or "latest" against a pin older than the corpus's cadence gets a warning line.
• CI re-pins live corpora on cadence (code weekly, feeds daily) and rebuilds a pack only when upstream changed.
• Done when: two machines with the same lockfile give identical answers; the stale-pin warning shows.

Step 7. Background sync, status, privacy

• xerj hub sync on a user timer: refresh the catalogue, pull newer pins for cached packs, verify, prune to budget. xerj hub status lists cached packs, sizes, ages, last sync, hosted on or off.
• Hosted route default: on in public projects, off when the project remote is private or the user turns it off. The answer always says local or hosted.
• Done when: sync never blocks a query; status matches what is on disk.

Step 8. Docs, skill, MCP registration

• The hub section of llms.txt becomes one screen: the one-line command, the MCP call, when to use it, what success looks like. xerj init installs the hub skill. Copy prompts on hub pages become the one line.
• Done when: the shell-less persona of the first-time-agent harness reaches an answer in one call.

Step 9. Measure before publishing anything

• Arms: memory only; normal agent with git; normal agent with web only under realistic rate limits and bot blocks, recording bytes fetched and failed fetches; hosted regex code search; a web search API; XERJ today; XERJ one line, first use; XERJ one line, steady state.
• Task classes: exact lookups (the control, where a tie with git is expected); how-does-it-work questions across files; non-git multi-source data; sessions of ten questions on one corpus.
• Metrics: correct, cost, output tokens, turns, bytes downloaded, wall time on a quiet machine, failed fetches, routing accuracy. At least three repeats. Plan written down before the run.
• Done when: results are posted here with every arm, win or not. The site shows them only if XERJ beats the normal-agent arms.

Do not

• Do not make the client clone or index a hub corpus.
• Do not require a corpus name in the question. Do require it in the answer.
• Do not open a pack before its signature and checksums verify.
• Do not block an answer on a download.
• Do not pull above the size limit, or a restricted licence class, except as the policy above says.
• Do not send the question to the hosted node when hosted is off.
• Do not print passages when def mode answers the question.
• Do not publish benchmark numbers unless XERJ beats the normal-agent arms.

Where this saves tokens, from the measurement

Fifty questions, five corpora, same model, 2026-10-13:

• Agent from memory: 21/50 correct, $0.30, 5 turns.
• Normal agent, clone and grep: 49/50, $0.91, 22 turns, 9,361 output tokens.
• Agent via the hub as it is today: 50/50, $1.66, 52 turns, 18,342 output tokens.

Where the hub's cost went: each turn re-reads about 28,000 tokens of the agent's own context; the hub route took nine to twelve turns per session (read docs, install, start a node, fetch manifest, add, index, query, stop). The hub's answers themselves were about 600 tokens per session.

With the one-line design, as the agent: no docs to read (5,000 tokens gone), no install, node, add or index turns (seven turns gone), one call per question returning 50 to 300 tokens instead of raw grep output. Steady state is one turn per question. Against fetching web pages, where one page is thousands of tokens and may not load at all, the difference is larger. The one case it will not win is an exact symbol in a small public repo that a hosted regex search already answers in one call; the benchmark keeps that case as the honest control.

Two decisions

1. Hosted route in private projects: default off (proposed).
2. Automatic pull limit: 50 MB, approval above (proposed).

[xerj-org]

Implementation plan is up as a docs PR, written for whoever implements it: #1376 (file: docs/CORPUS_HUB_V2_IMPLEMENTATION.md).

It fixes what the discussion left open: a new xerj-hub crate with shared types in xerj-common::hub; packs built from engine snapshots (create/restore already exist), signed with the pack pipeline key, pulled by xerj hub pull; a definitions table with value extraction and a --def mode; a signed catalogue with term sketches; the router behind xerj "<question>" with exact scoring thresholds and the fetch, licence, disk and privacy policies; /_hub/v1 endpoints; MCP tools; a XerjHub gRPC service with fixed field numbers; lockfile, freshness, sync; the docs; and the pre-registered measurement. Every step has files, types, tests and a done condition, plus a do-not list.

Two defaults are encoded as proposed above (hosted off inside git repositories, automatic pull up to 50 MB); change them in the PR if you want different ones.