Support LoRA based large model finetuning.

[pengchengguo]

What?

Support LoRA-based large model finetuning and also provide results of LoRA-based Whisper finetuning on Aishell corpus.

By finetuning the Whisper large model, we are able to achieve superior results of 2.5/2.7 CERs on dev/test sets, respectively.

See also

1. LoRA: Low-Rank Adaptation of Large Language Models (https://arxiv.org/pdf/2106.09685.pdf)
2. https://github.com/microsoft/LoRA.git
3. https://github.com/huggingface/peft

[codecov]

Codecov Report

Merging #5400 (97f855c) into master (71dc9a3) will decrease coverage by 6.82%.
Report is 187 commits behind head on master.
The diff coverage is 86.61%.

@@            Coverage Diff             @@
##           master    #5400      +/-   ##
==========================================
- Coverage   77.14%   70.33%   -6.82%     
==========================================
  Files         684      707      +23     
  Lines       62713    64932    +2219     
==========================================
- Hits        48383    45673    -2710     
- Misses      14330    19259    +4929     

Flag	Coverage Δ
test_configuration_espnet2	∅ <ø> (∅)
test_integration_espnet1	?
test_integration_espnet2	48.76% <53.27%> (-0.31%)	⬇️
test_python_espnet1	19.19% <1.57%> (-0.76%)	⬇️
test_python_espnet2	51.30% <78.74%> (-1.01%)	⬇️
test_utils	23.10% <66.66%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
espnet2/bin/whisper_export_vocabulary.py	93.22% <100.00%> (+0.36%)	⬆️
espnet2/text/build_tokenizer.py	78.37% <ø> (ø)
espnet2/text/whisper_token_id_converter.py	89.47% <100.00%> (+3.36%)	⬆️
espnet2/text/whisper_tokenizer.py	89.18% <100.00%> (+0.95%)	⬆️
espnet2/train/preprocessor.py	77.31% <100.00%> (-0.12%)	⬇️
espnet/nets/e2e_mt_common.py	63.93% <66.66%> (-0.48%)	⬇️
espnet2/bin/asr_inference.py	87.50% <83.33%> (+0.51%)	⬆️
espnet2/tasks/st.py	86.66% <88.88%> (-1.66%)	⬇️
espnet2/layers/create_lora_adapter.py	95.74% <95.74%> (ø)
espnet2/tasks/abs_task.py	76.84% <83.33%> (+0.04%)	⬆️
... and 3 more

... and 102 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sw005320]

Thanks, @pengchengguo!
The results are very impressive.
Can you also add a test so that this implementation becomes more solid?

[pengchengguo]

Sure, the HF model link will also be updated later.

[sw005320]

@pyf98, can you review this PR?

@pengchengguo, can you add a test?

[pyf98]

Thanks! It looks good to me!

[pyf98]

Just one question:
Does it support LoRA for all ESPnet models? Or just Whisper? I guess it should support all.

[pengchengguo]

Yes, it supports all.

[pengchengguo]

I am changing this PR to "work in progress" because I also want to add Whisper fine-tuning to ST. The ST part is being refactored now and will be finished soon.

[mergify]

This pull request is now in conflict :(

[sw005320]

Can you remove “WIP” from the PR title?

[pengchengguo]

Sure, now I am trying to increase the codecov, it seems the CI test file misses some functions.

[sw005320]

We seem to have an issue for CI. Can you check it?

[pengchengguo]

I fixed the CI issues and ST errors as I mentioned before. Now, Whisper fine-tuned ST shows better results compared with E-Branchformer (60.9 vs 53 BLEU scores on Fishercallhome dev sets).
I need time to decode all test sets and upload the pre-trained model, so I want to do it in another PR.
For this PR, I think it's okay to be merged.

[sw005320]

Thanks, @pengchengguo!