Improvement plan till first official release

[xerj-org]

XERJ search-for-AI plan, next releases. Each item: acceptance criteria (AC), references (Ref), business value and metric (Val). Gates follow the project's own rule: measured on the release binary or it does not ship.

1. xerj_map MCP tool (rc.78)

• AC: returns per-index FieldSpec from autoindex-catalog (name, es_type, cardinality_est, coverage, up to 5 examples, date and numeric min/max); numeric min/max added to FieldSpec (FieldAcc already tracks int_min/int_max); default response ≤4 KB per index; schema pinned in landing/docs/agents/schemas/mcp-tools.json via tests/published_schema_drift.rs.
• Ref: xerj-autoindex/src/infer/mod.rs, catalog.rs, xerj-mcp/src/lib.rs tool_specs (10 tools, none returns the catalog).
• Val: agents stop guessing field names. Metric: unknown-field 400s from MCP-issued DSL down ≥80% on the reference-coding harness.

2. POST /_ask + xerj_plan MCP tool: prompt in, validated DSL out (rc.79)

• AC: input {index or pattern, prompt}, output {query DSL, plan[], confidence, indices[]}; every plan passes xerj_query::parse_request before return, zero invalid DSL; index routing over ax-* by catalog description, top-1 ≥0.9 on single-dataset prompts; field gating by one batched typed call; values from terms agg when cardinality_est ≤1,000, else BM25 over a distinct-values index; numeric and date thresholds offered as choices from FieldSpec min/max, never generated; unresolved phrases return 422 naming them, no silent guess; deterministic, 3 runs identical; p50 ≤300 ms CPU on a 20-field index.
• Gate: new benchmarks/ask-plan, ≥200 (prompt, gold result set) pairs over public tabular datasets, result-set F1 ≥0.9; agent harness (same method as the case study: 16 runs per arm, real claude -p counts) shows output tokens per solved structured-query task ≤50% of agent-written DSL at equal solve rate.
• Ref: case study 9,982 vs 26,477 output tokens; hybrid RRF orders tied fused scores by HashMap iteration: top-10 differs for 21 of 40 queries after a restart on unchanged data #940 stable tie order.
• Val: agents never write Elasticsearch DSL again. Metric: tokens per solved task, invalid-query rate.

3. Local zero-shot decision head, tier 2 of systemone (rc.79)

• AC: --decide-mode local loads a ModernBERT-class decision model through the existing candle path, feature-gated like neural; /v1/systemone and /_decide answer noul and choice with no history index; tier order: history vote when [decisions] index has support, else local model, else hosted key; model echoes its own id, never a Jev name; no new outbound client, xerj-rerank/tests/egress_inventory.rs unchanged; p50 ≤50 ms for 30 questions on 8 CPU cores; quantized weights, download ≤300 MB.
• Gate: zero-shot SMS accuracy ≥0.95 and ECE ≤0.05, AG News ≥0.85, Banking77 reported not gated; below the bar it ships behind a flag with the numbers published.
• Ref: systemone_api.rs (422 on no support), xerj-ai/embedder.rs, benchmarks/decisions-as-retrieval.
• Val: only self-hosted Jev-wire endpoint with zero-shot. Metric: share of decision traffic answered on the node, $0 vs Jev's measured $0.0145 per 1k on SMS.

4. Self-judged hits, zero-token rerank (rc.80)

• AC: opt-in judge: {local: true, min_p} on _search and MCP search; each hit carries _p_relevant; hits below min_p dropped and counted in the response; adds ≤40 ms p50 for top-30 on CPU.
• Gate: BEIR harness on the post-hybrid RRF orders tied fused scores by HashMap iteration: top-10 differs for 21 of 40 queries after a restart on unchanged data #940 deterministic path beats hybrid 0.699 SciFact / 0.345 NFCorpus by more than the 3-run spread, and FiQA ≥0.30 (hosted Jev rerank got 0.3638). Loses, does not ship, same rule as the cross-encoder.
• Ref: docs/ZERO_TOKEN_DIRECTION.md bar, benchmarks/neural-path-triage, benchmarks/beir-hybrid/results/2026-09-20-rerank-full.
• Val: fewer irrelevant passages enter the context window. Metric: output tokens per solved task in the case-study harness, target −20% vs 9,982.

5. Token-budgeted responses (rc.78)

• AC: max_tokens on all MCP search tools; count with one named tokenizer (state which); overlapping file:line passages deduped; response never exceeds budget, enforced by test; over-budget hits listed as locators only.
• Ref: _passage, full cap in xerj_code_search.
• Val: predictable context cost. Metric: tokens per solved task, zero budget overruns.

6. Decision cache flywheel (rc.80)

• AC: every tier-2 or tier-3 answer written to [decisions] index with text, label, p, source, ts; /_decide returns source per answer; human corrections stored with human: true and weighted ≥2x in the vote (configurable).
• Gate: Banking77 replay, after 2,000 cached hosted answers, ≥80% of later traffic answered by history at ≥0.8 confidence with accuracy ≥0.97.
• Ref: blog "A BM25 vote ties Jev on spam", 87% clearing 0.8 at 97.9%. Check TypeSafe terms on retaining outputs.
• Val: hosted spend falls every week. Metric: hosted share of decisions over time.

7. Detections and ingest-time labels: make _watcher real (rc.80)

• AC: PUT /_watcher/watch either evaluates on schedule or returns 501, no more accepted-and-ignored; evaluator reads .xerj_alert_rules, writes .xerj_alert_fires with calibrated p; pipeline percolate prefilter → decide → fire; xerj autoindex --label <question-set> writes label and label_p per document through /_decide; evaluator cost inside the idle-budget gate (<0.5% core, ≤0.2 MB RSS per idle index).
• Gate: on the classify-email boundary tier the wrong-and-confident rate at p ≥0.8 is ≤5% (today 0.625 accuracy at 0.902 mean confidence), abstain absorbs the rest.
• Ref: discussion Optional Jev classification in `xerj autoindex` (emails first): would you use it? #1012, benchmarks/systemone-classify-email, es_compat.rs put_watch.
• Val: Jev's headline use case at ingest, no egress. Metric: documents labelled per node-hour, false-alert rate.

8. Calibration layer (rc.80)

• AC: [decisions] calibration = isotonic|temperature fitted on a held-out 20% of history; /_decide returns p_raw and p_cal; GET /_decide/_calibration publishes the reliability curve; applies to local and hosted probabilities.
• Gate: Jev FiQA rerank ECE 0.3109 → ≤0.10 on held-out; no probability shipped without an ECE beside it.
• Val: min_score becomes a threshold people can trust. Metric: ECE per index.

9. Stemming by default for prose fields from autoindex (rc.78)

• AC: autoindex declares a stemming analyzer for fields the profiler marks semantic (high word_ratio); new generations only, documented; ES-YAML conformance stays 0 failed.
• Gate: NFCorpus zero-hit queries 25 → ≤10; BEIR BM25 ≥ +0.01 over 0.657 SciFact / 0.302 NFCorpus.
• Ref: A declared analyzer stops applying at _flush: analysis.analyzer.default is memtable-only, a per-field analyzer is accepted and ignored, an unknown analyzer name is accepted #937, fix(api): fields/docvalue_fields pierce an explicit _source includes filter (#932) #991, ZERO_TOKEN_DIRECTION "What the measuring found".
• Val: recall on plurals and inflections for free. Metric: zero-hit rate.

10. Open model release: xerj-decide (1.1)

• AC: training pairs generated from ≥50 real autoindex catalogs plus public decision sets; Apache-2.0 weights on Hugging Face with an eval card (Banking77, SMS, AG News, ask-plan); loads through item 3.
• Val: model roundups list models, not engines. Metric: HF downloads, inclusion in Jev-alternatives lists (DataCamp's seven omit XERJ today).

Packaging rule for all ten: a benchmarks/<name> directory with raw results and a blog post with the losses left in, same as the two Jev posts. Once 3 lands, Show HN on the one-line TYPESAFE_ENDPOINT switch.

[xerj-org]

The plan is now tracked for development — every item filed with its AC, measured gate, and verified code refs:

rc.79 (milestone 6): #1055 xerj_map · #1056 _ask/xerj_plan · #1057 local decision head · #1058 max_tokens · #1059 stemming by default
rc.80 (milestone 7): #1060 self-judged hits · #1061 decision cache flywheel · #1062 real _watcher · #1063 calibration layer
v1.1.0 (milestone 8): #1064 open xerj-decide model

Working index: the pinned plan-tracker issue. One correction applied at filing: items 1, 5 and 9 were targeted rc.78, which was published eight minutes before this discussion posted — they slide to rc.79.