Add Baton, a hosted end-of-turn model

[Ted-Johnson-JoinIn-AI]

What

Adds BatonAdapter, a streaming adapter for Baton — JoinIn AI's end-of-turn model, served as a hosted API — plus tests and README/.env.example entries.

Baton returns p_eot, a probability that the current turn has ended, on the harness's decision grid.

Submitted as a streaming adapter. It passes row["messages"] as prior conversational context and sends only the audio; it never reads row["words"], so the model is given nothing about the turn it is being asked to judge beyond the audio up to that moment — the same constraint it operates under in production. There is a unit test asserting exactly that.

Each turn is scored with a single stateless POST /v1/turn — the whole turn in, the full grid out. That is one request per turn rather than one per grid point, and because nothing is held open between calls your --concurrency maps straight onto horizontally scaled instances.

score_point = 0.2, matching the model adapters.

Results

English, livekit/eot-bench-data validation, standard harness settings — 1,105 scored spans (705 hold, 400 EOT):

	False cutoffs @ 300 ms	False cutoffs @ 600 ms	Latency @ 5% cutoff	Latency @ 10% cutoff
Baton	12.3%	4.8%	577 ms	350 ms

Measured through this adapter, against the hosted endpoint, with the key issued for your team — a full 400-turn run, n_missing_scores: 0, 45m33s, no --skip-errors. AUC 0.9577, AP 0.9118. Happy to send the run's summary.json and predictions.parquet if useful.

14-language macro mean: AUC 0.8762 · FC@300 25.1% · FC@600 8.9% — from our own pipeline, not measured through this adapter, unlike the English row above. We have spot-checked de/es/ja/hi through the adapter at 40 turns each (zero missing spans, in line with those figures) but not run them in full. English is materially stronger than the rest and we are not claiming multilingual parity.

A finding worth passing back

streaming_stt.pcm16le_bytes encodes as array * 32767.0, which shifts every sample by one LSB against the conventional round(x * 32768). Most consumers tolerate that, but we measured it as a real quality cost for anything sensitive to exact sample values — so it can quietly penalise some adapters here without that showing up anywhere in the output.

This adapter therefore does its own encoding, in _prepare_pcm16_audio, with a comment explaining why. Happy to send that as a separate PR against the shared helper if you would rather fix it centrally — we did not want to change a helper other people's published numbers depend on inside an unrelated submission.

Reproduce

English first — 400 turns, about 45 minutes. That is the board's headline table, so it is the run worth doing before anything else; multilingual can follow whenever suits you.

BATON_API_KEY=... uv run --with aiohttp eot-harness predict-streaming \
  --path livekit/eot-bench-data --name en --split validation \
  --adapter eot_harness.baton_adapter:BatonAdapter \
  --output-dir output --concurrency 12

then the usual compute-metrics / compare-models.

Three notes, each from a run we did ourselves:

• No --skip-errors. It appears in the examples for other adapters, and here it is the one flag that can quietly cost you turns: a dropped turn leaves the eval set smaller with nothing in the output saying so, and would read as our model failing rather than our server being busy. Without it a genuine failure aborts loudly, which is what we both want. We ran the full 400 without it: n_missing_scores: 0.
• --concurrency 12 is measured, not guessed. Our service takes 8 concurrent per instance and queues beyond that; 12 kept it busy with zero rejections across repeated bursts. Higher is untested.
• A 429 is capacity, not breakage. The adapter retries them with exponential backoff so a busy moment cannot cost a turn.

Baton is in beta and the endpoint is key-gated. A key for your team goes to you by email rather than in this thread — say the word if it has not arrived. We would rather you reproduce the table yourselves than take it on trust.

[CLAassistant]

All committers have signed the CLA.

[chenghao-mou]

Thanks for the PR! I've made a new PR #16 to include the artifacts and some minor code changes.