AVSR recipe for Easycom Dataset

[ms-dot-k]

What?

New recipe for training audio-visual speech recognition model on Easycom dataset.
The recipe is based on LRS3 avsr recipe which utilizes pre-trained AV-HuBERT model. (Dumped features)

I added data augmentation techniques to the espnet2/asr/encoder/avhubert_encoder.py

1. acoustic noise perturbation: a babble noise is corrupted with random noisy strengths at the feature level.
2. modality dropout: audio and video streams are randomly dropped out, so that we can still perform audio-visual or audio-only, visual-only prediction after the model is trained.

See also

Easycom dataset is too small to achieve proper performances by using the dataset only. The recipe utilizes both Easycom and LRS3 datasets to train the model.

[sw005320]

I added several comments.

[sw005320]

The audio-only data significantly degrades the performance.
Can you provide some reasons?
Probably due to the AV HuBERT architecture?

[ms-dot-k]

The dataset is very challenging due to noise and long-distance voice.
Previous ASR model (wav2vec2.0) trained on 60k hours of data achieves 87.5% WER (https://arxiv.org/pdf/2212.11377.pdf). Therefore, by employing the visual information, we can improve the performance greatly by complementing the insufficient audio information (due to noise, overlapped speech, and long-distance voice) during speech recognition.

The trained model using the recipe was trained on 1,759 hours of data for pre-training (AV-HuBERT) and 438 hours of data for finetuning. Considering the data amount, the current performance seems reasonable.

One possible direction to improve the performance is using more audio-visual data including LRS2, VoxCeleb, and AVSpeech.

[ftshijt]

Very cool extension! Many thanks for the effort.

Could you please also add en entry in egs2/README.md for the dataset?

Also, two minor comments as follows:

[codecov]

Codecov Report

Attention: 26 lines in your changes are missing coverage. Please review.

Comparison is base (27f292d) 76.11% compared to head (0582547) 76.13%.

Files	Patch %	Lines
espnet2/asr/encoder/avhubert_encoder.py	33.33%	26 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5630      +/-   ##
==========================================
+ Coverage   76.11%   76.13%   +0.01%     
==========================================
  Files         743      743              
  Lines       69117    69151      +34     
==========================================
+ Hits        52608    52647      +39     
+ Misses      16509    16504       -5     

Flag	Coverage Δ
test_configuration_espnet2	∅ <ø> (∅)
test_integration_espnet1	62.92% <ø> (+0.14%)	⬆️
test_integration_espnet2	48.48% <2.56%> (-0.04%)	⬇️
test_python_espnet1	18.39% <0.00%> (-0.01%)	⬇️
test_python_espnet2	52.66% <33.33%> (+0.03%)	⬆️
test_utils	22.15% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sw005320]

Thanks a lot!