Fix parakeet_runner blank priming by mattjcly · Pull Request #16509 · pytorch/executorch

mattjcly · 2026-01-08T15:30:07Z

Summary

parakeet_runner is currently missing transcription of the first part of audio. I believe this is due to a lack of "priming with blanks" that is done in https://github.com/NVIDIA-NeMo/NeMo.

Tested with this audio file:
audio.wav

Which I would manually transcribe as: "Can you please add to analyze.py the ability to pass in a dash dash dataset dash path or dash d that will take the place of dot input if present? Also dot input should be renamed to dot dataset"

ExecuTorch Before

Before Executorch misses the "Can you please add to analyze" and transcribes as "py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset."

-> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.000767 executorch:main.cpp:278] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte
I 00:00:00.001246 executorch:main.cpp:294] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.002790 executorch:wav_loader.h:98] WAV header detected, getting raw audio data.
I 00:00:00.002793 executorch:wav_loader.h:105] RIFF Header: RIFF
I 00:00:00.002794 executorch:wav_loader.h:106] Chunk Size: 853676
I 00:00:00.002795 executorch:wav_loader.h:113] WAVE Header: WAVE
I 00:00:00.002796 executorch:wav_loader.h:120] Format Header: fmt 
I 00:00:00.002797 executorch:wav_loader.h:121] Format Chunk Size: 16
I 00:00:00.002798 executorch:wav_loader.h:122] Audio Format: 3
I 00:00:00.002799 executorch:wav_loader.h:123] Number of Channels: 1
I 00:00:00.002800 executorch:wav_loader.h:124] Sample Rate: 16000
I 00:00:00.002801 executorch:wav_loader.h:125] Byte Rate: 64000
I 00:00:00.002801 executorch:wav_loader.h:126] Block Align: 4
I 00:00:00.002802 executorch:wav_loader.h:127] Bits per Sample: 32
I 00:00:00.002803 executorch:wav_loader.h:132] Subchunk2Size: 853604
I 00:00:00.002824 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.002826 executorch:main.cpp:297] Loaded 213401 audio samples
I 00:00:00.002838 executorch:main.cpp:308] Running preprocessor...
I 00:00:00.018590 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
I 00:00:00.018609 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.027469 executorch:main.cpp:333] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333
I 00:00:00.027480 executorch:main.cpp:336] Running encoder...
I 00:00:26.206136 executorch:main.cpp:353] Encoder output shape: [1, 1024, 167], len=167
I 00:00:26.206163 executorch:main.cpp:385] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000
I 00:00:26.206166 executorch:main.cpp:387] Running TDT greedy decode...
I 00:00:29.509575 executorch:main.cpp:397] Decoded 55 tokens
I 00:00:29.509588 executorch:main.cpp:400] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:59] invalid tiktoken line: 
I 00:00:29.514298 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
Transcription tokens: py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.
I 00:00:29.514337 executorch:main.cpp:415] Done!

NeMo

NeMo thru:

import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/parakeet-tdt-0.6b-v3")
output = asr_model.transcribe(['/Users/matt/Documents/parakeet_test_audio/audio.wav'])
print(output[0].text)

outputs "Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.":

-> % python examples/asr/asr_chunked_inference/rnnt/test.py
[NeMo W 2026-01-08 10:20:16 megatron_init:62] Megatron num_microbatches_calculator not found, using Apex version.
W0108 10:20:16.612000 27527 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.
[NeMo I 2026-01-08 10:20:22 mixins:184] Tokenizer SentencePieceTokenizer initialized with 8192 tokens
[NeMo W 2026-01-08 10:20:24 modelPT:188] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    use_lhotse: true
    skip_missing_manifest_entries: true
    input_cfg: null
    tarred_audio_filepaths: null
    manifest_filepath: null
    sample_rate: 16000
    shuffle: true
    num_workers: 2
    pin_memory: true
    max_duration: 10.0
    min_duration: 1.0
    text_field: answer
    batch_duration: null
    max_tps: null
    use_bucketing: true
    bucket_duration_bins: null
    bucket_batch_size: null
    num_buckets: 30
    bucket_buffer_size: 20000
    shuffle_buffer_size: 10000
    
[NeMo W 2026-01-08 10:20:24 modelPT:195] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    use_lhotse: true
    manifest_filepath: null
    sample_rate: 16000
    batch_size: 16
    shuffle: false
    max_duration: 40.0
    min_duration: 0.1
    num_workers: 2
    pin_memory: true
    text_field: answer
    
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
    Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
    Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda.
    Cuda graphs with while loops are disabled, decoding speed will be slower
    Reason: CUDA is not available
[NeMo I 2026-01-08 10:20:26 rnnt_models:226] Using RNNT Loss : tdt
    Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2026-01-08 10:20:26 label_looping_base:113] No conditional node support for Cuda.
    Cuda graphs with while loops are disabled, decoding speed will be slower
    Reason: CUDA is not available
[NeMo I 2026-01-08 10:20:29 save_restore_connector:284] Model EncDecRNNTBPEModel was successfully restored from /Users/matt/.cache/huggingface/hub/models--nvidia--parakeet-tdt-0.6b-v3/snapshots/6d590f77001d318fb17a0b5bf7ee329a91b52598/parakeet-tdt-0.6b-v3.nemo.
[NeMo W 2026-01-08 10:20:29 dataloader:760] The following configuration keys are ignored by Lhotse dataloader: use_start_end_token
[NeMo W 2026-01-08 10:20:29 dataloader:498] You are using a non-tarred dataset and requested tokenization during data sampling (pretokenize=True). This will cause the tokenization to happen in the main (GPU) process,possibly impacting the training speed if your tokenizer is very large.If the impact is noticable, set pretokenize=False in dataloader config.(note: that will disable token-per-second filtering and 2D bucketing features)
Transcribing: 1it [00:00,  1.11it/s]
Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.

ExecuTorch After

After ExecuTorch "Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset."

-> % ./cmake-out/examples/models/parakeet/parakeet_runner --model_path examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte --tokenizer_path /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model --audio_path /Users/matt/Documents/parakeet_test_audio/audio.wav
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.002165 executorch:main.cpp:298] Loading model from: examples/models/parakeet/parakeet_tdt_exports-portable/parakeet_tdt.pte
I 00:00:00.002909 executorch:main.cpp:314] Loading audio from: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.005307 executorch:wav_loader.h:98] WAV header detected, getting raw audio data.
I 00:00:00.005318 executorch:wav_loader.h:105] RIFF Header: RIFF
I 00:00:00.005320 executorch:wav_loader.h:106] Chunk Size: 853676
I 00:00:00.005322 executorch:wav_loader.h:113] WAVE Header: WAVE
I 00:00:00.005324 executorch:wav_loader.h:120] Format Header: fmt 
I 00:00:00.005325 executorch:wav_loader.h:121] Format Chunk Size: 16
I 00:00:00.005327 executorch:wav_loader.h:122] Audio Format: 3
I 00:00:00.005329 executorch:wav_loader.h:123] Number of Channels: 1
I 00:00:00.005330 executorch:wav_loader.h:124] Sample Rate: 16000
I 00:00:00.005332 executorch:wav_loader.h:125] Byte Rate: 64000
I 00:00:00.005333 executorch:wav_loader.h:126] Block Align: 4
I 00:00:00.005335 executorch:wav_loader.h:127] Bits per Sample: 32
I 00:00:00.005336 executorch:wav_loader.h:132] Subchunk2Size: 853604
I 00:00:00.005447 executorch:wav_loader.h:226] Loaded 213401 audio samples from WAV file: /Users/matt/Documents/parakeet_test_audio/audio.wav
I 00:00:00.005458 executorch:main.cpp:317] Loaded 213401 audio samples
I 00:00:00.005485 executorch:main.cpp:328] Running preprocessor...
I 00:00:00.025430 executorch:cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
I 00:00:00.025449 executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.035054 executorch:main.cpp:353] Mel spectrogram shape: [1, 128, 1334], mel_len: 1333
I 00:00:00.035067 executorch:main.cpp:356] Running encoder...
I 00:00:25.835932 executorch:main.cpp:373] Encoder output shape: [1, 1024, 167], len=167
I 00:00:25.835974 executorch:main.cpp:405] Model metadata: vocab_size=8192, blank_id=8192, num_rnn_layers=2, pred_hidden=640, sample_rate=16000
I 00:00:25.835977 executorch:main.cpp:407] Running TDT greedy decode...
I 00:00:29.269823 executorch:main.cpp:417] Decoded 65 tokens
I 00:00:29.269837 executorch:main.cpp:420] Loading tokenizer from: /Users/matt/Workspace/executorch/examples/models/parakeet/parakeet_tdt_exports-portable/tokenizer.model
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:59] invalid tiktoken line: 
I 00:00:29.274029 executorch:llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
Transcription tokens:  Can you please add to analyze.py the ability to pass in a dash-dataset-path or dash d that will take the place of dot input if present. Also dot input should be renamed dot dataset.
I 00:00:29.274057 executorch:main.cpp:435] Done!

Test plan

Tested manually, see above

pytorch-bot · 2026-01-08T15:30:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16509

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit ebc0f16 with merge base 8e8d97e ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / android / run-emulator (gh) (#16137)
Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-cla · 2026-01-08T15:30:13Z

Hi @mattjcly!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

github-actions · 2026-01-08T15:30:45Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR fixes a critical bug in the parakeet_runner where the first part of audio was not being transcribed. The issue was caused by missing "priming with blanks" initialization that is performed in the NVIDIA NeMo implementation. The fix initializes the prediction network state with an SOS (Start of Sequence) token, which is set to the blank_id, before beginning the greedy decoding process.

Key Changes:

Initialize the prediction network state by executing the decoder with SOS (blank_id) token before the main decoding loop
Update LSTM hidden and cell states (h_data, c_data) with the primed state from the SOS initialization
Replace the previous zero-initialized decoder state approach with proper NeMo-compatible priming

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

meta-cla · 2026-01-08T15:40:07Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

mattjcly added 2 commits January 8, 2026 10:21

Prime parakeet like NeMo does

cdf8183

Comment w/ refs

0fc387f

mergennachin requested a review from Copilot January 8, 2026 15:34

Copilot started reviewing on behalf of mergennachin January 8, 2026 15:34 View session

Copilot AI reviewed Jan 8, 2026

View reviewed changes

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 8, 2026

format fix

ebc0f16

mergennachin requested review from JacobSzwejbka and mergennachin January 8, 2026 15:54

JacobSzwejbka marked this pull request as ready for review January 8, 2026 17:01

JacobSzwejbka requested a review from lucylq as a code owner January 8, 2026 17:01

JacobSzwejbka approved these changes Jan 8, 2026

View reviewed changes

JacobSzwejbka merged commit 7fa1aaa into pytorch:main Jan 8, 2026
141 of 142 checks passed

Navigation Menu

Uh oh!

Fix parakeet_runner blank priming - #16509

Fix parakeet_runner blank priming#16509
JacobSzwejbka merged 3 commits into
pytorch:mainfrom
mattjcly:matt/parakeet-priming

mattjcly commented Jan 8, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Jan 8, 2026 •

edited

Loading

Uh oh!

meta-cla Bot commented Jan 8, 2026

Uh oh!

github-actions Bot commented Jan 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

meta-cla Bot commented Jan 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mattjcly commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

ExecuTorch Before

NeMo

ExecuTorch After

Test plan

Uh oh!

pytorch-bot Bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16509

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

meta-cla Bot commented Jan 8, 2026

Action Required

Process

Uh oh!

github-actions Bot commented Jan 8, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

meta-cla Bot commented Jan 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mattjcly commented Jan 8, 2026 •

edited

Loading

pytorch-bot Bot commented Jan 8, 2026 •

edited

Loading

This PR needs a `release notes:` label