[WIP] Add BART for summarization training with CNN/DM using pytorch-lightning

[andr-ec]

This pull request adds to the example for BART for summarization. I used the example for NER using pytorch-lightning as guidance. This example will train on CNN/DM and evaluate, and get decent results, though I haven't trained it on the full dataset just yet. I'm sure there are better defaults for the hyperparams but these seem to work.
I based this PR on the code I wrote in this colab.

This would hopefully close #3004

TODO

• Be able to train the model on a GPU.
• remove unused args
• add test step and save results.

Happy to hear any feedback!

[codecov-io]

Codecov Report

Merging #3236 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3236   +/-   ##
=======================================
  Coverage   77.56%   77.56%           
=======================================
  Files         100      100           
  Lines       16970    16970           
=======================================
  Hits        13162    13162           
  Misses       3808     3808           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d4a019...9d4a019. Read the comment docs.

[sshleifer]

Nice! @yjernite might be interested!

[sshleifer]

Thanks for starting this! I left a couple nitpicks, but looks reasonable to me. Were you planning on running finetuning for longer and posting results?

[andr-ec]

I made those requested changes. And yes I'm planning to run finetuning this weekend and share results. I only have access to a k80 so it'll take a while 🤷🏽‍♂️

[srush]

This looks awesome. Let's coordinate with #3290 as well to share whatever code is possible.

[srush]

@nateraw can you do a review of this PR as well?

[nateraw]

Once #3290 gets merged, you'll have to update a few things, so I marked some here so you can get ahead of the curve on that. Once it's merged I'll be able to give a little more specific advice. Great work 👍

[srush]

@acarrera94 I will try to get this working this week. If you are in the pytorch-lightning open slack we can also chat a bit more about the design.

LGTM pending other reviewers!

[andr-ec]

@nateraw I've made all of those changes and it looks like #3290 has been merged, anything else that needs to change? Thanks!

[srush]

It's blocked on me, I should be able to get to it tonight.

[srush]

Thanks! This code is nicely done. If you integrate with @nateraw 's work, I think it will eliminate about half the code. Also I would recommend just moving it to one file (utils will become very small.)

[srush]

New code looks great. Excited to try it out!

[srush]

Thanks for sticking with it @ACarrera I'm really impressed how concise this became. Next we can get some numbers.

[sshleifer]

@acarrera94 run_train.sh is using 19GB on my system.
Does your system use less?
I am also seeing no memory savings from adding --fp16.
Thanks!

[andr-ec]

@sshleifer I usually ran it using --max_seq_lengt=756. And that used less than 16gb of memory with a batch size of 4, so we might want to change that default. And I haven’t tried it using --fp16. That comes from BaseTransformer right?