Vocab size to train LM and ASR

[ahmedalbahnasawy]

what is the best vocab size to train huge english corpus+librispeech and which unit is better (unigram or bpe) ?

[sw005320]

what is the best vocab size to train huge english corpus+librispeech

We did not observe large improvement when we use large BPE size. So, you can just use our default librispeech BPE size (5,000)

and which unit is better (unigram or bpe) ?

We did not observe a big difference. I think either is fine.

[ahmedalbahnasawy]

@sw005320 I tried 10000 vocab size for english dataset mixed with Librispeech and 5000 also (unigram unit)
Both giving the same result.
I think vocab size should be 30K if we are using BPE unit

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue is closed. Please re-open if needed.

[rajeevbaalwan]

Have you observed the effect when we reduce nbpe or moving towards character based ASR ?

[ahmedalbahnasawy]

@rajeevbaalwan not observed any improvement. Character based model suffers from contextual informations.