[TF Longformer] Add Multiple Choice, Seq Classification Model

[patrickvonplaten]

🚀 Feature request

modeling_longformer.py has the classes LongformerForSequenceClassification, LongformerForMultipleChoice and LongformerForTokenClassification which are not present in modeling_tf_longformer.py at the moment.
Those classes should be equally added to modeling_tf_longformer.py.

Motivation

The pretrained weights for TFLongformer are available so that these classes could be used for finetuning.

Your contribution

This issue is a good first issue because it is not too complicated to add these models. One should take a look at modeling_tf_roberta.py to see how these models are implemented for TFRoberta and implement them analogous for TFLongformer. Please make sure that the docstring is correct and that test are added for each class (again Roberta can serve as an example here, check out test_modeling_tf_roberta.py).

I am happy to guide interested community contributors through the PR and help them get it merged.

[Groskilled]

Hi !
I'd like to help and work on this if that's ok.

[patrickvonplaten]

Awesome, feel free to open an issue :-)

[Groskilled]

Hello !

I'm a bit lost here. I've looked at modeling_tf_roberta.py and modeling_longformer.py to create the class TFLongformerForSequenceClassification. I'm not sure if I am going in the right direction here and same goes for the tests.
I used python -m pytest -n auto --dist=loadfile -s -v ./tests/test_modeling_tf_roberta.py to get an idea on what should I do for testing but it seems the test for TFRobertaForSequenceClassification is skipped and my test on the class I created (which is basically just a copy/paste of the roberta's test) is skipped too.

Here is a link to what I've done so far: Groskilled@461ee62

Am I on the right track ? And what am I missing on the tests ?

Sorry to ask such simple questions, it's my first time participating in an open source project.

[patrickvonplaten]

No worries ;-). This looks alright! Could you open a PR so that we can see your changes directly on the PR? You can checkout this doc to understand how to do PRs: https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md. Would be great if you can ping me on the PR and then we look together!

[Zigur]

HI @Groskilled and @patrickvonplaten, I have been playing a bit around this issue, as I have some familiarity with Keras/TF2 but no previous experience with transformers, and I was figuring out a way to start familiarising with them. As I am interested in classifying long documents Longformer is of interest to me.
I have a draft of my current changes here. The test suite seems to pass (using Python 3.7.5, they did not on Python 3.8.2 on my Mac machine), but I would need extensive feedback as I have mostly lifted code from test_modeling_tf_roberta.py and the testing counterpart.
If it is of interest, I can open a pull request with all the details, or @Groskilled you can feel free to cherry-pick part of it if it's useful for your own pull request (as you were working on this earlier on, apologies for the intromission)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[patrickvonplaten]

Issue is still open! If stuck, feel free to take a look at the unfinished PR.