Decode Citrinet CTC output through the SentencePiece model

[5uck1ess]

Problem

Citrinet ASR transcription produces "subword salad" on every backend — fragments that look like valid subword tokens glued together in the wrong order (first reported downstream as 5uck1ess/audio.cpp#1, where it initially looked CUDA-specific; CPU reproduces byte-identically).

Root cause: the runtime resolves CTC token ids against the sidecar vocab.txt that model_manager.py copies verbatim from the NeMo archive. That file is NeMo's WordPiece-style compat vocab and omits <unk> (id 0), so every line sits one position off from the true SentencePiece ids — every decoded token maps to the wrong piece. The misalignment slipped past the vocab_size check because the line loader appended a phantom empty entry for the file's trailing newline (1023 lines counted as 1024).

Fix

Load the piece table from tokenizer.model — which the converter already exports and the config already names via tokenizer_model_file — using the framework's existing SentencePiece support, and decode CTC ids with decode_sentencepiece. This replaces the WordPiece ## join heuristics with the tokenizer's real spacing semantics and removes the vocab.txt dependency entirely. Existing converted model directories work unchanged (both files ship since the model's introduction); a piece-count mismatch now fails with an actionable message instead of silently decoding garbage.

Verification

On an RTX 3090 (linux-cuda-release build), NGC stt_en_citrinet_256 converted with this repo's own tooling, 16 kHz mono test clip with known content:

Before (CUDA and CPU identical):

tiadi had was more thatberendip day the newt fi whereeral soca an of tryt havep no heing very the close is behind to tea ofer theyated you for stuffy to even that thater

After (CUDA and CPU identical, model directory untouched):

the quick brown fox jumps over the lazy dog yesterday the weather in boston was cold and windy but day it is warm and sunny

A real human recording also transcribes accurately. New unit test (citrinet_tokenizer_decode_test) pins the id-alignment contract: <unk> at id 0, exact round trip through decode_sentencepiece, and shifted-by-one ids must not reproduce the input. sentencepiece_tokenizer1_test still passes.

[0xShug0]

@5uck1ess Thank you for the PR! Good catch. Citrinet was one of my earlier implementations and hasn’t received as much attention. audio.cpp is currently moving toward spec-based model loading to better decouple the model logic (check https://github.com/0xShug0/audio.cpp/blob/main/docs/gguf.md for status) . PR #46, which has already been merged, refactored Citrinet to use the new approach. Would you be willing to update your PR based on the latest main branch?

[5uck1ess]

@0xShug0 Rebased onto latest main (d88bab3) — the fix now goes through the #46 spec-based loading. Changes from the rebase:

• load_citrinet_weights resolves the tokenizer via resources.require_file("tokenizer"), which model_specs/citrinet_asr.json already declared for both sources, and validates the piece count against vocab_size.
• Dropped the vocab file entries from both the gguf and safetensors sources in the spec, since nothing reads vocab.txt anymore. Existing converted model directories keep working — the file is just ignored.

Verified on an RTX 3090 (safetensors source): citrinet_tokenizer_decode_test and sentencepiece_tokenizer1_test pass, and audiocpp_cli --task asr --family citrinet_asr produces a correct transcript on both --backend cuda and --backend cpu with an untouched converted model dir.

One gap to flag: I don't have a Citrinet GGUF bundle on hand, so the gguf source path of the spec is untested here. It should be unaffected — the tokenizer entry is identical in both sources and the loader change is source-agnostic — but worth a quick check if you have one handy.

[0xShug0]

@mirek190 Would you be willing to upload the Citrinet GGUF model to your HF repository? I’ll add the link to the README and also plan to include it in the WIP GGUF management tool. Please let me know whether you intend to maintain the repository long term.

(I also created a repo https://huggingface.co/audio-cpp/audio.cpp-gguf)

[5uck1ess]

Update: the GGUF gap I flagged above is now closed. I converted our local Citrinet-256 with audiocpp_gguf (f16, --root for the sidecars) and ran the CLI against the .gguf directly — correct transcript on both --backend cuda and --backend cpu, with vocab.txt absent from the directory. Both spec sources are now verified end-to-end. Also rebased once more onto 1627694 so the branch sits on current main.

[mirek190]

@mirek190 Would you be willing to upload the Citrinet GGUF model to your HF repository? I’ll add the link to the README and also plan to include it in the WIP GGUF management tool. Please let me know whether you intend to maintain the repository long term.

(I also created a repo https://huggingface.co/audio-cpp/audio.cpp-gguf)

Yes I am going to upload them and keep that repo
Make a copy its ok

[0xShug0]

@5uck1ess Just one minor issue: tests/unittests/test_asr_standalone_gguf.cpp still checks assets.require_file("vocab") for Citrinet, causing the test to fail with missing asset resource: vocab.

[5uck1ess]

@0xShug0 Good catch, thanks — fixed. test_asr_standalone_gguf.cpp no longer asserts a "vocab" resource for Citrinet; I kept the stray citrinet_256_vocab.txt fixture in the test root on purpose, so the case now also proves previously converted model directories (which still contain the file) load with it ignored. asr_standalone_gguf_test and citrinet_tokenizer_decode_test both pass locally.

[5uck1ess]

are our AIs just talking to each other? :D

[0xShug0]

are our AIs just talking to each other? :D

Nowadays we are just human middleboxes.

Merged! Thanks for your contributions!