ARD v0.91 + OKF namespace, NLWeb protocol, and complete table descriptions

[rvguha]

Why

Discovery was picking the wrong table on ambiguous measures. On the bare query total revenue — which is what discovery actually searches, since the entity is expunged first — the IRS 990 leaf and the us-gaap concept scored within 0.001 of each other, so which one won was luck rather than ranking.

The cause wasn't description length as such: every source already declares an entityType, but only the classifier ever saw it. Leaves were embedded scope-blind, with descriptions truncated to 200 chars. The re-ranker was worse — its prompt tells the model to "match the table's SUBJECT and SCOPE," which the candidate cards never stated.

Results

	before	after
"total revenue" → SEC vs 990	990 by 0.0005 ❌	SEC ✅
Leaves with description < 200 chars	5,346 of 8,925	0
Routing accuracy (193-case corpus)	90.7% top-1	92.2% top-1 / 93.8% top-3
Discovery cost per question	~$0.0034	~$0.0007

tests/route_eval.py measures routing and discovery cost together, so both numbers are reproducible.

Descriptors

• tools/descriptions.py expands each leaf's one-line definition into a full description naming its subject and what it excludes. Batched, cached, concurrent.
• tools/check_descriptions.py verifies generated text against the definition it came from — a deterministic screen, then an LLM adjudication of only the suspicious ones. Ground truth is descriptions_input.json, recorded at generation time, because the generated leaves are gitignored and have no committed "before". Final rate: 38 unsupported of 6,171 (0.62%), each round fed back into the prompt.
• repr_queries.py dedupes, strips template placeholders and caps at the spec's 2–5 — on cached entries as well as fresh ones. This removed 19 queries that embedded literal {school} braces into the index.

ARD v0.91 (ards-project/ard-spec#70)

• Identifiers are domain-anchored URNs: urn:air:sec.gov:okf:sec-edgar.revenues. 8,925 URNs, no collisions.
• trustManifest from each source's existing trust block; url XOR data honoured.
• An OKF namespace, declared as the first key on every entry — not just full ones, and not only on the response envelope. An envelope-only context is valid while the entry sits in the document and breaks the moment a consumer lifts one entry out of entries[], which is exactly what a registry does. Verified with a real JSON-LD processor: every okf: term expands, nothing left unbound.
• GET /agents, POST /explore, GET /agents/entry, /.well-known/ard.json join POST /search. All shapes validate against the published ard-entry.schema.json.

NLWeb

/ask speaks NLWeb and nothing else — begin / intermediate_message / result / nlws / complete / end, SSE by default, blocking on streaming=false, plus /sites and /health. An OKF source is an NLWeb site; an OKF table is an Item whose schema_object is the descriptor's own frontmatter.

Instrumentation

Per-question LLM calls, tokens and cost, broken down by step (classify / resolve / check / synthesize) and by model. The Agent Finder is reported separately, since it's its own service. GET /costs gives the running total.

Grant graph

Moves to managed Postgres (tools/grants_to_postgres.py), with per-thread connection pooling and precomputed population rollups. The by-cause join went from ~280s live to ~6ms — which is what makes the smallest SKU viable rather than merely cheap. Row counts, totals and the case-insensitive match path are all verified against the source before anything is dropped.

Deployment

BIND_HOST / PORT, /healthz (no LLM call — a health probe that costs money per poll is a bill), a per-source daily cap on /ask, CORS, and DEPLOY.md covering what is not production-hardened.

---

Full audit before commit: 8,925 documents, zero issues (junk tags, placeholders, duplicate or out-of-range queries, thin descriptions, missing titles). All seven source types answer end to end.

🤖 Generated with Claude Code