Zipformer streaming model: repeated single-syllable words/characters stop abruptly

[rouseabout]

I have been experimenting with the pre-trained zipformer streaming models:

• https://huggingface.co/marcoyang/icefall-libri-giga-pruned-transducer-stateless7-streaming-2023-04-04
• https://huggingface.co/pkufool/icefall-asr-zipformer-streaming-wenetspeech-20230615

A problem I observe is the models do not recognise more than two single-syllable words/characters in a sequence. The recogniser stop emitting tokens after the first two words or characters.

For example:

• English: Speaking ONE ONE ONE is recognised only as ONE ONE. Speaking ONE over and over again produces no more tokens.
• 中文： Speaking 对对对对 is recognised only as 对对. Speaking more 对's produces no more tokens.

The problem goes away if single syllable words/characters are mixed with other single syllable words/characters. For example:

• ONE TWO ONE TWO ONE TWO ONE
• 对的对的对的

[rouseabout]

Problem also occurs with non-streaming models, e.g.:

• https://huggingface.co/yfyeung/icefall-asr-multidataset-pruned_transducer_stateless7-2023-05-04
• https://huggingface.co/pkufool/icefall-asr-zipformer-wenetspeech-20230615

Given this happens across different datasets and training runs, is there something about the zipformer model architecture that intentionally limits output of many identical single-syllable word/character tokens?

audio-samples-en-zh.zip

[joazoa]

I also experience this issue.

[danpovey]

Can it relate to the stateless transducer context length of 2? It doesn't know how many copies of the symbol it has already emitted and guesses it has already output them. At one point we experimented with augmenting the context with a repeat-count, but we didn't end up merging it.

…

On Wed, Jan 17, 2024, 4:53 AM joazoa ***@***.***> wrote: I also experience this issue. — Reply to this email directly, view it on GitHub <#1356 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO7XCV55ES7HDDPHJBDYO3SCZAVCNFSM6AAAAAA6XDOPLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJUGQ4TKMJXG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>