Load BERT-esque checkpoints in pytorch formats

[jonathanbratt]

The original BERT checkpoints released by Google are in a TensorFlow format.
It seems that most of the related work done by other teams is in the PyTorch implementation.
In particular, pre-trained models such as RoBERTa and DistilBERT have been released for PyTorch.

Many of these models are compatible with the BERT architecture, though possibly with different parameters or vocabularies. It would be great to be able to easily load these into RBERT.

[jonathanbratt]

One possible approach is to write some code to convert PyTorch models into TensorFlow checkpoints, at which point it should be possible to use existing code to load/use. I don't know how to do this, though. Anybody with more PyTorch experience want to give this a shot?

[jonathanbratt]

Some possibly helpful links:

https://forums.fast.ai/t/converting-a-pytorch-model-to-tensorflow-or-keras-for-production/14016/30

https://github.com/onnx/tutorials

https://github.com/onnx/tutorials/blob/master/tutorials/PytorchTensorflowMnist.ipynb

[jonathanbratt]

Related: SciBERT checkpoints have been released in tensorflow format, so those are already available in RBERT.
https://github.com/allenai/scibert