Complete pytorch transformers interface, deprecate old GPT implement

[HaokunLiu]

• Replacing old GPT implement with the one from huggingface pytorch transformers
• Add GPT2, Transformer-XL, XLM to pytorch transformer interface
• Refactor pytorch transformer interface a little bit to reduce duplicated / outdated code & comments

[HaokunLiu]

Generally, this PR looks good. Thanks for taking the time to do this Haokun! My only concern is about TaskModulator. I do see what you mean in that some of preprocessing belongs to the model, but I think we need to be careful about introducing any more complexity/abstraction to jiant (and making functions that are already long even longer and harder to parse). I'd have to think a bit harder about how to make this abstraction the most clear as it can be, but a start would be renaming TaskModulator to something like ModelPreprocessingInterface.

I was thinking about including tokenizer and indexer inside model_preprocessing_interface as well, and change the main process from create_tasks -> preprocess -> create model to create_tasks -> create model -> preprocess, so that members of model_preprocessing_interface can be passed from model, instead of first creating from args in preprocess, and again creating from args in model. But this is unnecessarily radical for the sake of #881 .

Since you are one of the major developers of jiant, maybe you can consider this idea, and when the time comes, figure out a well-rounded overall architecture for jiant.

[HaokunLiu]

This looks good to me. Do experiments with GPT1 match the old implementation?

	openai-gpt(old)	openai-gpt
cola	cola_mcc: 0.57882, cola_accuracy: 0.82838	cola_mcc: 0.554, cola_accuracy: 0.818
sts-b	sts-b_corr: 0.748, sts-b_pearsonr: 0.738, sts-b_spearmanr: 0.758	sts-b_corr: 0.857, sts-b_pearsonr: 0.859, sts-b_spearmanr: 0.856
mnli	mnli_accuracy: 0.71000	mnli_accuracy: 0.779
mrpc	mrpc_acc_f1: 0.76916, mrpc_accuracy: 0.71569, mrpc_f1: 0.82263, mrpc_precision: 0.71733, mrpc_recall: 0.96416	mrpc_acc_f1: 0.825, mrpc_accuracy: 0.794, mrpc_f1: 0.855, mrpc_precision: 0.824, mrpc_recall: 0.889
qnli	qnli_accuracy: 0.750	qnli_accuracy: 0.846
qqp	qqp_acc_f1: 0.75931, qqp_accuracy: 0.78620, qqp_f1: 0.73242, qqp_precision: 0.66895, qqp_recall: 0.80918	qqp_acc_f1: 0.824, qqp_accuracy: 0.847, qqp_f1: 0.801, qqp_precision: 0.771, qqp_recall: 0.833
rte	rte_accuracy: 0.560	rte_accuracy: 0.603
sst	sst_accuracy: 0.928	sst_accuracy: 0.939
wnli	wnli_accuracy: 0.437	wnli_accuracy: 0.493

Some results are different, but I think, considering how old gpt and new gpt differs, it meets the expectation.
We didn't include concatenating two sentences in a pair for old gpt, when it process pairwise tasks, it make embedding for two sentences separately, then combine the two embedding to make prediction.
As we can see, on single sentence tasks, i.e. cola and sst, old gpt and new gpt have similar results. While on pair tasks, new gpt is better.

[sleepinyourhat]

Thanks! That's not that informative of a comparison, but since you're getting numbers in the same ballpark as what OpenAI published, I think that's enough. I agree that it's okay to leave in some awkward abstractions for now—better to get this out there and refactor later than to put too much burden on you for doing it.

There are some new merge conflicts (we moved the config dir), BTW.

[pruksmhc]

LGTM!

[sleepinyourhat]

Ready to merge?

[HaokunLiu]

Yes, it’s ready.

[sleepinyourhat]

Great—I'll make a proper release tm unless someone beats me to it.