Getting word level timestamps from the speechbrain ASR model

[vedvasu]

Hey folks,

I have been using the speech-brain transformer recipe for running predictions on Speech to Text model trained on the LibriSpeech dataset. Is there a way to get word-level timestamps/alignments along with the transcripts?

Ref: https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech/ASR/transformer

Currently, the transcripts are returned as a list of predicted words.

[Adel-Moumen]

Hi @vedvasu,

You might be interested in PyCTCDecode see: https://github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/ASR/CTC/train_with_wav2vec.py#L108 and https://github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/ASR/CTC/hparams/downsampled/train_hf_wavlm_signal_downsampling.yaml

It provides the frame of each tokens. Unfortunately, it does only works for CTC models.

[ilaouirine]

[Adel-Moumen]

Hello, please take a look at my last comment here : #1945 (comment)

[ilaouirine]

Sorry, I mistakenly deleted my comment. Thanks for your response!

[ilaouirine]

Hello, I'm using the provided code and PyCTCDecode, but I'm getting only words, how can I get the timestamps during the test?
Thank you!

[Adel-Moumen]

Yes, as you can see in PyCTCDecode https://github.com/kensho-technologies/pyctcdecode/blob/main/pyctcdecode/decoder.py#LL730C9-L730C21 you should use this function to get more details about your beams.

So instead of calling decode please call decode_beams here https://github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/ASR/CTC/train_with_wav2vec.py#L111 which will return this https://github.com/kensho-technologies/pyctcdecode/blob/main/pyctcdecode/decoder.py#L766

[ilaouirine]

Ok, I'll try this. Thank you very much!