v1.1.0

FA-Bench v1.1.0 brings Track 2 to full scale, adds six Track 1 aligners, scores boundaries with their labels checked, and publishes the 2026-09 records.

Re-cut on 2026-09-26. This release was first tagged on 2026-09-25. The tag now points at the merge of #6, which adds the scoring fixes below and republishes records/202609/. Numbers taken from the first tag are superseded.

Re-cut changes that move published numbers

• An utterance a system returns nothing for is charged. Every boundary in it counts against F1 recall and every word and phone in it as a WER or PER deletion. MAE stays over what came back. Leaderboards gain n_absent and n_empty, and every records page says who returned nothing for how much.
• A one-step recognizer is scored on the words it wrote. The eight commercial APIs had their words rewritten onto the reference spelling when saved.
• Word MAE matches words without regard to case. It moved by at most 0.14 ms.
• TIMIT's degraded audio is cut to the clean length. Each file carried about 475 ms of noise after its last word. The TIMIT noisy cells were re-run, apart from ElevenLabs, which a spot check found unchanged, and NeuFA, whose output is cut at the clean length.
• The records' Noisy column averages unrounded values, so it matches the paper to the last digit.

The re-cut also records every system's version, checkpoint, training data and overlap, rewrites every README with stale facts corrected, sets fabench.__version__ to 1.1.0 and adds a CHANGELOG.

What's new

Track 2, timestamped ASR. Systems that decode their own words are scored on the same audio and conditions as the aligners, on a separate leaderboard.

• 13 one-step recognizers. Eight are commercial APIs: Amazon Transcribe, AssemblyAI Universal 3.5, Azure AI Speech, Deepgram Nova-3, ElevenLabs Scribe v2, Google Chirp 2, IBM Watson Large and Speechmatics Enhanced. The open ones are CrisperWhisper, Parakeet-TDT, TorchAudio (ASR), Whisper large-v3 and Whisper-timestamped.
• 19 two-step cascades, where a Track 1 aligner re-times a recognizer's words, such as Qwen3-ASR → MFA and Google Chirp 2 → Olign.
• A cloud adapter for the commercial APIs. cloud_check.py tests a credential on one utterance before a sweep, and cloud_cost.py estimates what the sweep will cost.

Six more Track 1 aligners, 17 in all: FALCON, MMS-FA, NeMo-FA (40 ms and 80 ms), NeuFA and UnitY2. NeuFA's outputs were produced by its first author with a checkpoint that is not released, and are scored here like every other system.

Label-checked boundary F1. A boundary counts only when the words or phones on both sides of it match the reference and its time is within the tolerance. The two utterance edges count, against silence. A word a recognizer drops or invents costs it the boundaries beside it.

Boundary classes. Every boundary is sorted by where it sits and how many of its neighbours match: interior with both matched, interior with one, utterance beginning, utterance end, and the rest. F1 and MAE are reported per class.

Analysis scripts behind the paper's numbers: measure_onset_bias.py (signed start and end bias), measure_grid.py (timestamp step size), measure_debiased_mae.py, measure_debiased_track2.py, boundary_class_tables.py, analyze_recognition_effects.py and report_labelled_f1.py.

Scoring changes

Numbers from v1.1.0 are not comparable with v1.0.0.

• MAE counts each boundary once. v1.0.0 charged a boundary shared by two adjacent units twice, which doubled the weight of the easy interior against the utterance edges. Word MAE rises by 3 to 5 ms.
• A boundary enters MAE only when every unit beside it matched, the same rule the F1 uses.
• The phone tier is classed by each phone's own label. Track 1 phones used to inherit their word's match, which turned the phone F1 into a time-only match.
• Aligner output is mapped back onto the words it was given before scoring (fabench/aligners/relabel.py), so an aligner that renormalizes its input is not charged for spelling.

Records

• New snapshot records/202609/, in a new layout records/<YYYYMM>/en/<transcript>/<tier>/<corpus>/, with gold/ for Track 1 and asr/ for Track 2. records/latest points to it.
• The comparison pages and the 10 to 100 ms sweep report the label-checked F1. Time-paired precision, recall and F1 stay on the Details pages as a diagnostic.
• The v1.0.0 snapshot is kept, at records/aligners/en/202608/ and in the new layout at records/202608/. Read 202608 against 202609 with the scoring changes above in mind.
• fabench report writes new B-F1all@t and W-F1all@t columns. Existing columns are unchanged.

Upgrading

Rescore existing outputs with ./evals/rescore_all.sh, then republish with evals/publish_records.py --release <tag>.

The license is unchanged, PolyForm Noncommercial 1.0.0.

Full changelog: v1.0.0...v1.1.0