Optional Jev classification in xerj autoindex (emails first): would you use it?

[Vinz2168]

XERJ can already talk to Jev for one thing: the optional rerank stage on _search (see docs/RERANK.md). Jev takes a question and a piece of text and returns a number from 0 to 1. I'm wondering whether xerj autoindex should be able to use it, optionally, to label documents while indexing them.

The idea, in plain words. You pass a small list of labels, for example invoice / newsletter / personal / needs-reply, and autoindex stores the best label and its score in a field on each record. Emails would be the first case, since .eml and mbox extraction is already on main. The same mechanism would work for any document kind.

What I'd want to keep true

• Off by default. It only runs when you pass a flag.
• The key stays on the node, never in the CLI or in a request, same as rerank.
• If the service fails, the record is still indexed, just without a label. A run never stops because of it.
• It says clearly, before it starts, that document text is sent to the provider.

What I don't know, and would like your view on

1. Would you use this for emails, or for something else (tickets, logs, code, notes)?
2. Would you accept sending text to a hosted service for it, or is "local only" a must? The local /v1/systemone can vote from examples you have already labelled, with nothing leaving the machine, but it needs those labelled examples.
3. Which matters more: one best label per document, or a score for each label?
4. Where should the label live: a plain field on the record, or a separate index?

Honest caveats. Jev's API is very new (our integration is from 20 Sep). We have measured it for ranking, not for classification, and its probabilities are not well calibrated (ECE 0.31 on FiQA), so I would treat the score as a ranking signal and not as a true percentage. Cost looks small (about $0.042 per million input tokens on their blog), but I haven't measured it for this. The current client only sends yes/no questions, so it would need extending.

Before writing any code I'd like to do a small measurement on a labelled dataset and share the numbers here. If nobody wants it, we skip it.

[Vinz2168]

The small measurement you mentioned before writing any code: PR up at #1026, benchmark at benchmarks/systemone-classify-email/.

It only measures the local path from Q2 — POST /_decide, which already ships (engine/crates/xerj-api/src/systemone_api.rs), needed no new code, and sends nothing off the node. No hosted-Jev number: that path needs jev-reranker extended to send classification questions instead of yes/no, which doesn't exist yet, so there's nothing honest to measure there.

Method: 260 originally-authored synthetic emails (no real mailbox, no licence/privacy question) across your four example labels — invoice/newsletter/personal/needs-reply — 200 train / 60 held-out test, k=10.

First run: 100% accuracy. Suspicious — the templated generators give each label near-exclusive vocabulary ("Invoice", "unsubscribe", "Hey <name>"), so that number says nothing. Added a second, hand-authored "hard" tier (8 emails) that deliberately crosses vocabulary — a personal note with a dollar figure in it, an invoice with the word "invoice" nowhere in it, a newsletter that ends by asking a direct question. That tier: 5/8 = 0.625 accuracy, mean confidence 0.902. The 3 misses were wrong and confident (0.836–1.000), not wrong-and-hedging — basically the same miscalibration shape you already flagged for the hosted path (ECE 0.31 on FiQA), just measured on the local one instead.

Answering your four questions from this, briefly (full version in the PR's README):

1. Not answered — this only shows the mechanism works across a label set, not which document kind benefits most.
2. Local needs zero egress and zero new client code, but it isn't better-calibrated than the hosted path's own known weakness, and it needs labelled examples before it can classify anything at all. Neither path arrives pre-calibrated; that's a real cost either way, not a reason to prefer one.
3. /_decide already returns both a winning label and a confidence (vote share), so no new work needed there — but per the hard tier, don't treat that confidence as a trustworthy abstain signal without tuning decisions.min_confidence against real-shaped held-out data first. n=8 is too small to say where that threshold should sit.
4. A plain field on the record (label/label_confidence) fits xerj-autoindex's existing convention (DOC_BODY_FIELDS = ["body","text"]) better than a separate index — a separate index is what /_decide reads from, not what this should write to.

Nothing wired into autoindex yet — the PR's README lists what that would still need (CLI flag off by default, an abstain/failure-tolerant pipeline step matching rerank's own failure policy, and a decision on whether the labelled-history index is operator-built or bootstrapped from a --label-examples file).

[xerj-org]

Verdict, and thank you — this is exactly the measurement the discussion itself said should come before writing any code.

What PR #1026 established (merged): local POST /_decide answers the four example labels with 1.000 accuracy on templated mail, but 0.625 accuracy at 0.902 mean confidence on the hand-authored boundary-crossing tier — wrong and confident, which is the same miscalibration shape you flagged for the hosted path, just measured on ours. And it got there with zero egress and no new engine code.

So, answering the title question honestly:

• As an interactive aid — a triage suggestion a human sees together with its confidence — yes. That works today against any node: label a history index, ask /_decide. I'd encourage trying it on real mail before any integration exists.
• As an unattended autoindex stage — not yet. A wrong-and-confident classifier silently foldering mail is worse than no classifier. The integration needs the three things bench(systemone): measure local /_decide for discussion #1012 email labelling #1026's README lists (CLI flag, default off; an abstain/failure-tolerant step with rerank's failure policy; a decision on how the labelled-history index is built) plus an abstain threshold — and the hard tier says that threshold must be conservative; n=8 is too small to tune it on.

The hosted Jev path remains unmeasured until jev-reranker can drive it, so no comparison claim is on the table from us. Keeping this discussion open for the wiring decision; the measurement and its caveat are the answer to the question as asked.

[xerj-org]

This thing is JEV don't really have many wins to just XERJ based on what we have measured https://xerj.org/blog/jev-vs-a-bm25-vote

And https://xerj.org/blog/does-xerj-beat-jev

We have be interested in making something better for XERJ like train decision making ranker for binary tree search for corpus. Plenty of reach but should give real impact.