Adding to an existing language model

[mbachtell]

If I wanted to add to an existing model how would I do that? I have topic specific language from scientific domains that I would like to add.

I didn't see anything in the open or closed tickets.

Thanks!

[mayeulk]

I have the same question. I do not know whether this is currently achievable or not, but the question seems like a duplicate of #12 (which says it is not straightforward). From a mathematical point of view, this is doable as it is a transformer model. From a software point of view, it is already done in some fields, see for instance:

• https://gmihaila.github.io/tutorial_notebooks/finetune_transformers_pytorch/
• https://medium.com/@lokaregns/fine-tuning-transformers-with-custom-dataset-classification-task-f261579ae068

I believe there is at least one strategy:

1. taking the initial aligned bilingual corpus of the existing model
2. adding your aligned documents to that corpus
3. retraining the model on the new corpus.

This would require to add local data, see #24

[mayeulk]

Duplicate of #12