Mark the words the model was unsure about, and learn Custom Words from corrections

[behdad-dot]

Problem

After a dictation is pasted, nothing tells me which words the model was unsure of. I have to reread everything to find the one name it mangled. Custom Words help, but only for words I remembered to add in advance; I rarely do, so the same name gets misheard every time.

Proposed solution

Two small pieces, both built on data the model already produces.

1. Mark the words the model was unsure about. Parakeet TDT (and other transducer/CTC models) give a probability per token. Taking each word's least likely token as its confidence separates the two cases cleanly in my tests: correctly heard words scored 0.94 to 1.0, while misheard names scored 0.19 to 0.56, so a 0.6 threshold flags almost only real mistakes. Those words can then be shown:

• on the inserted text itself, by asking the target app where those characters are on screen through the platform accessibility API (AT-SPI GetCharacterExtents on Linux, AXBoundsForRange on macOS, UI Automation TextRange.GetBoundingRectangles on Windows) and drawing a translucent box over each one for a few seconds;
• in the overlay or History as colored words, when the app exposes no text positions.

2. Learn Custom Words from corrections (opt-in). While the marks are up, watch that text field: when the user retypes a flagged word into something that is not a dictionary word (netbirt to NetBird), add it to Custom Words, with the misheard form as an alias. The list fills itself from the corrections the user makes anyway.

Alternatives considered

• Reviewing the whole transcript in the History panel: works, but means rereading everything.
• LLM post-processing: can fix some names from context, but also silently changes correct words, and does not tell you where to look.

Fit with Handy

Both pieces are optional, need no new model, and fail quietly (no accessibility access means no boxes). I know Handy is in a feature freeze; this is to see whether others want it.

I have a working prototype of both on GNOME Wayland (Python plus a small GNOME Shell extension, using Parakeet TDT 0.6B v3 int8 through onnx-asr). It works in GTK apps and in an Electron app once its accessibility tree is on; a VTE terminal exposes the same positions. Happy to share the code or measurements if useful.

[radhepa]

I built part 2 of this (learning Custom Words from corrections) on Windows, as an opt-in setting:

• After a paste, Handy reads the focused text field back through UI Automation until you stop editing for 10 seconds (60 s cap).
• When a pasted word is swapped for a similar one ("Ongoneyo" -> "Ongonyo", "Makari" -> "McCarrie" via a Soundex check), the new spelling is added to Custom Words, so the existing fuzzy correction fixes it next time.
• It skips rewording, dissimilar words, edits outside the transcript, short or common words, and password fields. Only the learned word is stored or logged, never the field text.
• Off by default: Settings > Advanced > Learn From Edits.

Branch: https://github.com/radhepa/Handy/tree/learn-custom-words

It doesn't do part 1 (confidence marks), so it learns from any correction, not only flagged words. Would a PR for this be welcome, Windows-only at first?