v1.1.3

Records snapshot records/202610/, scored with this release. No change
moves a number published in records/202609/, which stands as scored for
1.1.0. The October snapshot adds PocketSphinx, Olign 1.0 and one cascade,
and its Olign rows are described below.

Added

• Olign 1.0 as its own row beside Olign 0.9, on both tracks. On Track 1 it
  is given the reference transcript. On Track 2 it recognizes the words
  itself and then aligns them, so the tables call it two-step, as they do
  Qwen3-ASR. It is reached through Olewave's job API at
  https://api.olewave.com/olign/v1, with the key in OLIGN_API_KEY. The
  call lives with the cloud providers, which cache every response, retry
  and run requests concurrently. Recipes evals/aligners/olign_v1/ and
  evals/timestamp_asrs/olign_v1_asr/.
• PocketSphinx as a Track 1 row, the HMM family's one member not built on
  Kaldi. It aligns the way pocketsphinx align does, with the bundled
  en-us model and CMUdict, and g2p_en for words outside the dictionary. It
  is built from a pinned main-branch commit rather than the PyPI wheel,
  because the alignment fixes that matter here landed after v5.1.1. An
  utterance that fails on the default beams is aligned again with the beams
  open. The utterances that still fail also fail upstream's own CLI, and
  they count against the row as every system's failures do. See
  fabench/aligners/pocketsphinx/README.md.
• Qwen3-ASR → PocketSphinx as a Track 2 cascade, generated the way the
  other cascades are.
• The records snapshot records/202610/, with 19 forced aligners on
  Track 1 and 34 systems on Track 2. records/latest and the README point
  at it. ElevenLabs Scribe v2 has 12 of its 20 cells, since the account ran
  out of credit before the dev split under the four degradations, and
  ElevenLabs forced alignment is still unpublished for the same reason.
• Each Olign 0.9 hyp record keeps the versions object of the response
  that made it, for responses fetched since 2026-10-11.
• A leaderboard badge at the top of the README, ahead of the paper badge,
  that opens the current snapshot's results page.

Changed

• The three cascades into Olign 0.9 on Track 2 were rerun against the
  Olign 0.9 service that produced its Track 1 row, so every Olign 0.9 row
  comes from the same service. Under noise their word MAE improves by 0.8
  to 1.6 ms and their F1 at 20 ms by 0.02 to 0.05. Clean cells move by
  0.3 ms or less.
• A cloud recipe's base_url is part of its response cache key, so a
  recipe aimed at one endpoint cannot replay the cached answers of another.
  No tracked recipe sets base_url, so no cached response is orphaned.
• The Olign live smoke test runs only when OLIGN_TEST_PORT is set. It
  used to default to one machine's port.

Fixed

• The cloud adapter drops a phone whose end is not after its start, as the
  Olign 0.9 adapter has since it was written. Olign 1.0 is the only cloud
  row with a phone tier and it had kept them, so the two versions of one
  system were scored under different rules. On Buckeye dev under additive
  noise the difference was close to 10 ms of phone MAE. No published number
  moves, since Olign 1.0 first appears in records/202610/.
• evals/gen_cascade_configs.py writes a base tool's relative paths
  relative to the cascade's directory, and decides what is a path by the
  key, as the composer does. A bare venv: venv was left pointing at
  nothing.
• evals/run_cpu_cells.sh printed an empty tool name in every status line.

v1.1.2

FA-Bench v1.1.2 is a maintenance release. No change moves a published number, so the records/202609/ snapshot stands as scored for v1.1.0.

Fixed

• Olign is named Olign 0.9, the version that produced its numbers. The records in both snapshots, the provenance block and the Olign adapter's docs had called it Olign 1.0. Only the name changes.
• docs/paper.pdf is the revised paper. Its title is now "FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech". Its word table names Olign 0.9, and the abstract and the introduction open with a new first sentence. The tables are unchanged.

Added

• ElevenLabs forced alignment as a Track 1 row, elevenlabs_fa. Each utterance goes to ElevenLabs' /v1/forced-alignment endpoint with its reference transcript, and the row is scored on word boundaries beside MFA and Olign. It ships with enabled: false and no published numbers, because every request bills the account.
• cloud_check.py takes --text, so it can check a forced aligner. cloud_cost.py prices the new row.
• A one-column, double-spaced copy of the paper, docs/paper_for_reviewer.pdf, which the README's paper links now open. Its tables are set larger and write every number with its leading zero, 0.40 where the paper has .40, and its Fig. 1 is larger. evals/gen_paper_tables.py writes its tables in the same run as the paper's, so the two cannot disagree. The two-column docs/paper.pdf stays, and a new section at the end of the README links both and the paper on arXiv, arXiv 2609.32396.

Changed

• The commercial cloud adapter can send the reference transcript with the audio, and the transcript then joins the response cache key. The Track 2 rows send none, so their keys and cached responses are unchanged.

The full changelog is at v1.1.1...v1.1.2

v1.1.1

FA-Bench v1.1.1 is a maintenance release. No change moves a published number, so the records/202609/ snapshot stands as scored for v1.1.0.

Fixed

• The Olign adapter reaches the hosted endpoint. Cloudflare in front of it rejects Python's default User-Agent, so the adapter names itself, and it sends the Cloudflare Access service token from .fabench.env when one is set. The default endpoint is https://api.olewave.com/olign/v0.9.
• Two workers kept an import they no longer use, which failed the lint job.
• evals/README.md again says the top-level run_all.sh uses the same option parser.

Added

• The paper as docs/paper.pdf, linked from the top of the README.
• tools/ci.sh runs the three CI jobs locally, exactly as GitHub and GitLab run them. Ruff is pinned to 0.16.9, so a new ruff release cannot fail an unchanged tree.

The full changelog is at v1.1.0...v1.1.1

v1.1.0

FA-Bench v1.1.0 brings Track 2 to full scale, adds six Track 1 aligners, scores boundaries with their labels checked, and publishes the 2026-09 records.

Re-cut on 2026-09-26. This release was first tagged on 2026-09-25. The tag now points at the merge of #6, which adds the scoring fixes below and republishes records/202609/. Numbers taken from the first tag are superseded.

Re-cut changes that move published numbers

• An utterance a system returns nothing for is charged. Every boundary in it counts against F1 recall and every word and phone in it as a WER or PER deletion. MAE stays over what came back. Leaderboards gain n_absent and n_empty, and every records page says who returned nothing for how much.
• A one-step recognizer is scored on the words it wrote. The eight commercial APIs had their words rewritten onto the reference spelling when saved.
• Word MAE matches words without regard to case. It moved by at most 0.14 ms.
• TIMIT's degraded audio is cut to the clean length. Each file carried about 475 ms of noise after its last word. The TIMIT noisy cells were re-run, apart from ElevenLabs, which a spot check found unchanged, and NeuFA, whose output is cut at the clean length.
• The records' Noisy column averages unrounded values, so it matches the paper to the last digit.

The re-cut also records every system's version, checkpoint, training data and overlap, rewrites every README with stale facts corrected, sets fabench.__version__ to 1.1.0 and adds a CHANGELOG.

What's new

Track 2, timestamped ASR. Systems that decode their own words are scored on the same audio and conditions as the aligners, on a separate leaderboard.

• 13 one-step recognizers. Eight are commercial APIs: Amazon Transcribe, AssemblyAI Universal 3.5, Azure AI Speech, Deepgram Nova-3, ElevenLabs Scribe v2, Google Chirp 2, IBM Watson Large and Speechmatics Enhanced. The open ones are CrisperWhisper, Parakeet-TDT, TorchAudio (ASR), Whisper large-v3 and Whisper-timestamped.
• 19 two-step cascades, where a Track 1 aligner re-times a recognizer's words, such as Qwen3-ASR → MFA and Google Chirp 2 → Olign.
• A cloud adapter for the commercial APIs. cloud_check.py tests a credential on one utterance before a sweep, and cloud_cost.py estimates what the sweep will cost.

Six more Track 1 aligners, 17 in all: FALCON, MMS-FA, NeMo-FA (40 ms and 80 ms), NeuFA and UnitY2. NeuFA's outputs were produced by its first author with a checkpoint that is not released, and are scored here like every other system.

Label-checked boundary F1. A boundary counts only when the words or phones on both sides of it match the reference and its time is within the tolerance. The two utterance edges count, against silence. A word a recognizer drops or invents costs it the boundaries beside it.

Boundary classes. Every boundary is sorted by where it sits and how many of its neighbours match: interior with both matched, interior with one, utterance beginning, utterance end, and the rest. F1 and MAE are reported per class.

Analysis scripts behind the paper's numbers: measure_onset_bias.py (signed start and end bias), measure_grid.py (timestamp step size), measure_debiased_mae.py, measure_debiased_track2.py, boundary_class_tables.py, analyze_recognition_effects.py and report_labelled_f1.py.

Scoring changes

Numbers from v1.1.0 are not comparable with v1.0.0.

• MAE counts each boundary once. v1.0.0 charged a boundary shared by two adjacent units twice, which doubled the weight of the easy interior against the utterance edges. Word MAE rises by 3 to 5 ms.
• A boundary enters MAE only when every unit beside it matched, the same rule the F1 uses.
• The phone tier is classed by each phone's own label. Track 1 phones used to inherit their word's match, which turned the phone F1 into a time-only match.
• Aligner output is mapped back onto the words it was given before scoring (fabench/aligners/relabel.py), so an aligner that renormalizes its input is not charged for spelling.

Records

• New snapshot records/202609/, in a new layout records/<YYYYMM>/en/<transcript>/<tier>/<corpus>/, with gold/ for Track 1 and asr/ for Track 2. records/latest points to it.
• The comparison pages and the 10 to 100 ms sweep report the label-checked F1. Time-paired precision, recall and F1 stay on the Details pages as a diagnostic.
• The v1.0.0 snapshot is kept, at records/aligners/en/202608/ and in the new layout at records/202608/. Read 202608 against 202609 with the scoring changes above in mind.
• fabench report writes new B-F1all@t and W-F1all@t columns. Existing columns are unchanged.

Upgrading

Rescore existing outputs with ./evals/rescore_all.sh, then republish with evals/publish_records.py --release <tag>.

The license is unchanged, PolyForm Noncommercial 1.0.0.

Full changelog: v1.0.0...v1.1.0

v1.0.0

v1.0.0