Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script and readme

[mpierrau]

What does this PR do?

This PR was requested by @patrickvonplaten following my question and following discussion in the Discord ask-for-help channel under the title Wav2vec2 - why is mask_time_prob=0.05?

This PR adds the arguments mask_time_prob and mask_time_length to the examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py script and the corresponding example use in the README.md.

mask_time_prob is a variable describing two things, depending on context:

1. the percentage of the encoded feature vector to be masked during the contrastive learning task in pre-training
2. to imitate SpecAugment during fine-tuning

In this script, we are considering it in the context of 1).

mask_time_length is a variable describing the length (in # of frames frames) of each applied mask. It is added for completion.

Background

In the original wav2vec 2.0 article, the variable mask_time_prob is set to 0.65, which (due to overlap) results in an effective masking of approximately 49% of the feature vectors during pretraining. mask_time_length corresponds to the M variable in the article and is set to 10 there.

However, when considering the config file of wav2vec2-base, one finds that mask_time_prob=0.05. This is because this model is usually used for finetuning, and not for (continued) pretraining, and for finetuning 0.05 is a better hyperparameter value (see Appendix B of wav2vec 2.0 article). This is a bit confusing.

By considering the config file of the wav2vec2-base-v2 model, which was used during Patricks experimentation (see the speech-pretraining readme) one finds that indeed mask_time_prob=0.65 was used for pretraining.

The values 0.65 and 10 are set as default values for the DataCollatorForWav2Vec2Pretraining class defined in the script (as this class may be extracted from the script by users), but no defaults are given in the argparser, as the argument values are also specified in the wav2vec2-base and wav2vec2-base-v2 model configs, and if setting defaults in the argparser, the model config values would never be applied, which may be desired. Hence, the parser argument will only be relevant if it is explicitly specified as an argument when executing the script.

I believe this PR may also lift a bigger question, which is if mask_time_prob should be split into two different variables to avoid confusion in the future.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Participants in discussion on Discord: @osanseviero @patrickvonplaten

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[sgugger]

cc @sanchit-gandhi

[patrickvonplaten]

This looks good to me, but would be good to get a review from @sanchit-gandhi as well here

[sanchit-gandhi]

Hey @mpierrau! Sorry for the long delay in getting you a final review. Thanks for digging into this and documenting the motivations so well - the change is most welcome!

Regarding your question:

if mask_time_prob should be split into two different variables to avoid confusion in the future

I would say 'no' to this. At heart, mask_time_prob controls the same behaviour for pre-training and fine-tuning. It's simply the interpretation of this behaviour that is different:

• For pre-training, we're masking latents for our contrastive loss
• For fine-tuning, we're applying SpecAug in the latent space

-> this is the same behaviour for both (masking in the latent space), just different interpretations of what they mean for training

What we could do is clarify the meaning of this variable in the docs (with something similar to the above) - this would help clear the confusion without introducing extra variables and the potential for breaking changes!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[mpierrau]

Hey! I'm not sure what the next step is here, like I said, this is my first PR :) Some checks failed, not sure why. All tests passed before I commited @sanchit-gandhi's suggestions, but it seems they are mainly failing due to timeouts?

[sgugger]

You will need to rebase your PR on main for the tests to pass, as your branch does not have the fixes for the last release of TensorFlow.

[sanchit-gandhi]

Hey @mpierrau! Exciting to see that you've picked-up this PR again! Let me know if you need any final help - we're close to merging now!

As Sylvain has mentioned, you'll need to rebase onto main to fix the failing tests (see 5. in this guide: https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request, just remember to force push as detailed 😉)

[mpierrau]

Hey, sorry, I somehow managed to miss the force push flag anyways... I hope it still works?

[sanchit-gandhi]

Hey @mpierrau! Unfortunately the commit history gets messed up after a rebase + non-force push. Not to worry though! Let's open a new PR with your changes in favour of this one. You can create a new branch and copy over the relevant file (run_pretraining_...):

git checkout -b new-branch-mask-time-prob
git restore --source adding-mask_time_prob-args-to-wav2vec2-pretraining-script -- /path/to/relevant/file

You can then commit, rebase, and force push to origin to open a new PR with just the required changes.

[sanchit-gandhi]

Closing in favour of #20985.