Add TF<>PT and Flax<>PT everywhere

[patrickvonplaten]

What does this PR do?

This PR adds the PT<>TF and PT<>Flax equivalence tests to the common PyTorch tests so that the equivalence tests are also fetched when just the PyTorch files are changed. The PR also uncovered a couple of smaller bugs in TFHuBERT and FlaxAlbert.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[patrickvonplaten]

There were some hidden bugs in the Flax implementation of Albert. The architecture didn't allow config.inner_group_num > 1. Issue is solved at full backwards compatibility.

[sgugger]

Thanks for adding those. The only downside is that they will be run twice when we do the full suite, but I don't think it's a big issue.

[patrickvonplaten]

Thanks for adding those. The only downside is that they will be run twice when we do the full suite, but I don't think it's a big issue.

Yes, but I think it's much more important to be sure that if one changes PyTorch's CLIP that both PT<>Flax and PT<>TF is run with the test fetcher.

Also, the tests are actually slighly different in a sense that the ones that I added:

• Use the dummy model config of the PyTorch test suite
• Use the model inputs created in the PyTorch tests

where as the previous tests used Flax's or TF's config and inputs. So I think having those tests also forces us to be consistent with the testing configs and inputs across frameworks