Multi-dataset recipes

[asumagic]

🚀 The feature

It could be interesting to have a recipe and tools that help combine multiple training datasets. Some other toolkits do it, picking one example: k2/icefall.

To my knowledge, we do not have such a recipe in SpeechBrain nor do we provide documentation on how to do it.

This issue is for the sake of tracking progress and centralize discussion. If one wants to implement it, it should probably get more discussion beforehand.

Solution outline

Things to investigate:

• Whoever desires to implement this would need to examine what mechanisms are already there to help merge datasets, and decide on the best approach if any APIs should be added.
• We could make a tutorial that shows how to do it and explaining concerns around dataset merging.
• ASR is possibly one of the most viable targets for this: Could try LibriSpeech+CommonVoice for example (or also for example GigaSpeech c.f. add CTC recipe GigaSpeech #1210). There are probably normalization and BPE LM pre-training concerns here.
• Should train a model and compare performance with the baseline, e.g. do we save some WER% on LibriSpeech if we train on LS+CV?

Additional context

No response

[Adel-Moumen]

Hey @pplantinga and @Gastron, what do you think about this? Thanks! :)

[Gastron]

In SpeechBrain we currently don't have a specific format for datasets, which makes dataset operations tricky. I think it's simply a matter of us not forcing any specific filenames or sample keys.

Cases where some datasets do use the same conventions can probably already be handled with something like a ConcatDataset.

On the other hand, SpeechBrain is compatible with any normal torch dataset implementations. Maybe for these kinds of cases users could look at Lhotse or other dataset providers that do standardize the datasets.

[Adel-Moumen]

In SpeechBrain we currently don't have a specific format for datasets, which makes dataset operations tricky. I think it's simply a matter of us not forcing any specific filenames or sample keys.

Cases where some datasets do use the same conventions can probably already be handled with something like a ConcatDataset.

On the other hand, SpeechBrain is compatible with any normal torch dataset implementations. Maybe for these kinds of cases users could look at Lhotse or other dataset providers that do standardize the datasets.

Hey @Gastron, thanks for your comment. We are planning to train some large ASR models on multiple datasets with SpeechBrain. Do you think you could contribute into this project by adding this new feature of multi-datasets recipes?

Thanks.
Adel