Feed forward chunking for all pretrained models

[Pradhy729]

Based on this card: Feed forward chunking

@patrickvonplaten

I'd like to help contribute and implement this for all the other models if this is still pending?

[Pradhy729]

Any opinions here? I will create a PR if there is interest and would like to get your ideas and suggestions. @patrickvonplaten @sshleifer

[sshleifer]

@patrickvonplaten would be the point person and he is on Vacation until August 3.
In the interim, if you want to start working on this go right ahead. Make sure it's actually faster/needed before you start though. I don't really know.

[patrickvonplaten]

Hey @Pradhy729,

Yes it would be great to start a PR to add feed forward chunking to other models. Maybe you can start with BERT in your PR and ping us to get Feedback :-)

A couple of things to consider:

1. You should probably move the config param config.chunk_size_feed_forward to the general configuration_utils.py file.

2. As @sshleifer said it would be good to benchmark the gains in a very similar way to this Notebook:
   https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_2_4.ipynb

3. as said earlier we should start with BERT and config.chunk_size_feed_forward.

[Pradhy729]

Awesome! I will start with BERT and share with you for feedback.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.