Feature request: decode-time vocabulary biasing via FluidAudio custom vocabulary (proper nouns + inflected languages)

[krzysztofgos-pixel]

First: thank you for shipping both #17 and #18 within a day. Presspeech is my daily driver for all dictation (Polish), and this request comes from heavy real-world use.

The problem

The Dictionary & Shortcuts feature works after transcription: the model still mishears, and a rule patches the string. For proper nouns this means one rule per surface form. I currently maintain 197 correction rules, and a large share are just inflection variants of the same names (Polish is heavily inflected — a surname like „Szypański" also appears as „Szypańskiego", „Szypańskiemu", „Szypańskim"…). This affects every inflected language (Polish, Czech, Finnish, German, …), not just mine.

The engine already solves this

Presspeech depends on FluidAudio (Package.swift pins FluidInference/FluidAudio), and FluidAudio ships a complete custom-vocabulary module that Presspeech currently does not call (grep for vocabular|boost|biasing in the Swift sources: no hits):

public struct CustomVocabularyTerm {
    public let text: String          // term to boost
    public let weight: Float?        // strength
    public let aliases: [String]?    // alternative spellings
    public let tokenIds: [Int]?      // decode-time biasing on TDT tokens
    public let ctcTokenIds: [Int]?   // auxiliary CTC keyword spotter
    public let minSimilarity: Float? // similarity threshold
}

FluidAudio's CLI already exposes --custom-vocab, --vocab-min-similarity, --vocab-cbw, --vocab-margin. Another app on the same engine (FluidVoice) exposes CTC vocabulary boosting to users via a plain dictionary list, so the integration path is proven in practice.

The proposal

An opt-in, fully on-device vocabulary list (plain text, one term per line, optional weight) that is passed to FluidAudio's CustomVocabularyContext at decode time. Because biasing works on tokens rather than exact strings, one entry covers all inflected forms — something no amount of post-hoc rules can do. A minimal first step could even reuse the existing Dictionary & Shortcuts entries as the bias source.

This stays squarely inside Presspeech's philosophy: local, deterministic, opt-in, no cloud, no AI rewriting.

What I can offer

Happy to test any build against real Polish dictation. I have a frozen 25-sentence benchmark harness (229 words, WER + critical-error scoring, fixed ground truth) that I used to compare Presspeech against FluidVoice, so I can give you before/after numbers, not just impressions.

[rcourtman]

Thanks—this is well motivated, and the frozen Polish benchmark is exactly the kind of evidence this needs.

I checked the FluidAudio v0.15.6 revision now pinned by Presspeech. The important integration detail is that Presspeech currently uses Parakeet v3 through AsrManager.transcribe, which has no vocabulary-configuration hook. Vocabulary boosting is exposed through SlidingWindowAsrManager and the newer Parakeet Unified offline UnifiedAsrManager; both use an auxiliary CTC model to rescore the transcript. This is therefore an engine and benchmark change, not simply passing a list into the existing decode call.

One correction to expectations: FluidAudio aliases are alternate matches, but an accepted candidate is replaced with the canonical term.text. A single nominative entry will not automatically generate or preserve Polish inflections; each desired output form may still need to be represented as a vocabulary term or handled carefully as a canonical form. It could still materially improve proper nouns and reduce brittle post-transcription rules.

I'm keeping this open as an enhancement. The right first step is an experiment in experiments/swift-bench/ comparing current Parakeet v3 with the Unified or sliding-window vocabulary path on the frozen set, measuring WER, critical errors, latency, peak memory, model download size, and false positives. If you're willing, please share a sanitized term list, ground truth, benchmark commands and results, and recordings only if they contain nothing private. We should prove the engine path before designing the user-facing controls.

[rcourtman]

Benchmark support is now on main in f7352a1.

The new lane compares three isolated processes on the same fixtures:

• production Parakeet v3 through AsrManager;
• sliding-window Parakeet v3 without vocabulary boosting;
• the same sliding-window path with FluidAudio's auxiliary CTC vocabulary rescorer.

It reports WER, weighted exact critical-term recall, inference latency, peak process memory, model-cache footprint, and preparation time. Reports redact transcripts, fixture paths, and vocabulary diagnostics by default. The exact surface-form scoring is deliberate: aliases do not generate Polish inflections, so expected inflected forms should be listed separately.

Once the sanitized fixture, vocabulary, and critical-term files are ready, the benchmark command is:

cd experiments/swift-bench
./run-vocabulary-bias-regression.sh \
  --input-dir polish-benchmark \
  --vocabulary vocabulary.txt \
  --critical-terms critical-terms.txt \
  --language pl \
  --trials 3

Setup and file formats are documented in the benchmark README. The full repository CI, including the new benchmark self-tests and privacy guard, is green for this commit.

[krzysztofgos-pixel]

This is more than I expected, thank you. Building the lane before we could produce any evidence removes the hardest part of the work from our side.

We are going to run it. A correction to expectations I set implicitly: our constraint was never the reads themselves, it was a protocol failure of our own making on an earlier benchmark, so that is not a blocker here.

Our plan:

• fixtures from our own existing Polish dictation recordings with hand-corrected references, kept local and ignored;
• vocabulary and critical-term lists sanitised of client and company names before anything is published;
• all three lanes, --trials 3, on a frozen set.

Since scoring is on exact surface forms, we will list the inflections we actually care about rather than assume aliases cover them. I will come back with the redacted report rather than with further requests.

[krzysztofgos-pixel]

Results are in.

Setup: 40 clips (10.3 min, 1295 reference words) drawn from our own existing Polish dictation recordings, selected by set cover so all 38 proper nouns in our correction dictionary are represented; 68 critical-term instances. References are the raw transcripts from the independent cloud ASR that originally produced those recordings, so the reference is not derived from Parakeet. Vocabulary: 123 canonical entries with aliases, derived from our 204 post-transcription correction rules. --trials 3, --language pl.

Variant	Avg WER %	Critical hits	Critical recall %	Avg p50 ms	Max peak MB	Cache MB
v3	12.05	36/68	52.9	105.2	79.8	460.9
sliding-v3	13.26	35/68	51.5	155.9	58.4	460.9
sliding-vocab	16.08	52/68	76.5	556.8	73.3	558.9

The three-process split earned its keep: sliding-v3 lands within noise of production on recall (51.5 vs 52.9), so the jump to 76.5 is attributable to the rescorer rather than to the engine-path change.

The per-clip breakdown is where I would hesitate to ship this as-is:

• clean win (more terms, WER same or better): 2 clips
• win with a cost (more terms, worse WER): 9
• pure loss (no terms gained, worse WER): 11
• unchanged: 18

WER regressed on 20 clips and improved on 2. Two sharpest cases: one clip went from 0.0% to 15.9% WER and lost a critical term it previously got right (2/2 to 1/2); another gained 2 terms but went 6.9% to 27.6%.

That pattern reads as canonical-form substitution firing on spans that contained no target term. Recall improves, the surrounding text pays for it. For our own use post-transcription correction still wins, because it cannot damage the rest of the sentence. If the rescorer could be gated on decoder confidence rather than applied across the transcript, the trade would look very different.

Caveats, plainly:

1. References are strong cloud-model transcripts, not hand-audited audio. Reference error hits all three lanes equally so the between-lane comparison holds; absolute WER is indicative only.
2. Clip selection is conditioned on the cloud model getting the term right, so this measures "does the lane recover a term that was definitely spoken", not the harder case where both models fail.
3. False insertions are not counted directly. WER regression with zero recall gain is strong circumstantial evidence, not a counter.
4. 68 critical-term instances is a small sample. The direction is clear, the point precision is not.

Audio and fixtures stayed local and outside any repository; these numbers come from the redacted report. Happy to re-run with a different vocabulary shape, a confidence gate, or a larger sample if that would be useful.

One incidental note: --self-test fails on a pl_PL.UTF-8 system before any of this runs. Filed separately as #26.

[rcourtman]

A fourth benchmark lane is now on main in 697919e: sliding-vocab-conservative. It keeps the same sliding-window + CTC path, but applies FluidAudio's documented opt-in over-fire controls: short-term CBW taper pivot 5 and spotter-rescue similarity floors 0.30/0.50.

The existing command is unchanged; run-vocabulary-bias-regression.sh now runs and reports all four isolated variants automatically. This is not yet a decoder-confidence gate or a product change—it is a targeted test of whether the upstream conservative policy recovers enough WER/false-insertion precision while retaining useful proper-noun recall on the frozen Polish set.

The benchmark compiled and passed its self-test on the native macOS worker, and CI is green for the exact commit. If you rerun the same redacted benchmark, the new fourth row and per-clip comparison should tell us whether this policy is worth pursuing further.

[rcourtman]

The vocabulary benchmark contract is tightened on main in b60af4d. It now rejects critical-term entries that are empty or duplicate after the exact Unicode/case/punctuation normalization used for scoring, preventing silent double-counting. It also reports critical-term precision alongside recall and unexpected-insertion counts, per clip and in the aggregate summary; variable trials use the same conservative envelope (lowest hits plus highest insertion count).

The benchmark command and four variants are unchanged. macOS CI compiled the benchmark package and passed its self-test for the exact commit; Windows CI is also green. This remains benchmark work, not a product integration—the original vocabulary policy WER regression still requires evidence from the conservative lane.

[rcourtman]

The benchmark evidence contract is now complete on main in 8be9c17, and both macOS and Windows CI are green for that exact commit.

Since the first report, the runner has gained four direct-production AsrManager vocabulary policies (including no-rescue and exact-similarity), same-language negative controls, per-clip regression gates, input/source fingerprints, and stricter scoring coverage. Thresholded runs now also require --references-hand-audited: every target and control sidecar must be listened to and corrected by a human. The earlier cloud transcripts remain useful exploratory evidence, but reference errors can change measured WER and critical-term outcomes, so they cannot clear a product gate without that audit.

If you are still willing to rerun, the decisive command is:

cd experiments/swift-bench
./run-vocabulary-bias-regression.sh \
  --input-dir polish-benchmark \
  --negative-control-dir polish-negative-controls \
  --vocabulary vocabulary.txt \
  --critical-terms critical-terms.txt \
  --language pl \
  --references-hand-audited \
  --trials 3

The existing 40 target clips already satisfy the target-size floor after audit. The same-language control corpus must contain no configured critical terms, at least 10 distinct clips, and at least 1,000 reference words. A redacted report is sufficient; audio and vocabulary remain local. No vocabulary policy has been integrated into the app pending this evidence.