GH-1041: add support for DistilBERT

[stefan-it]

Hi,

this PR adds support for DistilBERT, a distilled version of BERT.

The following example shows how to use it in Flair:

from flair.data import Sentence
from flair.embeddings import BertEmbeddings

embeddings = BertEmbeddings("distilbert-base-uncased")

s = Sentence("Berlin and Munich are nice cities .")
embeddings.embed(s)

for token in s.tokens:
  print(token.embedding)
  print(token.embedding.shape)

Please also refer to this great blog post about distillation 🤗 🤗

I made some experiments on the full CoNLL-2003 corpus. DistilBERT (with this PR-ready version) is only -0.35% behind the BERT (base, uncased) model.

Notice: to try DistilBERT please make sure that you've installed the latest master version of pytorch-transformers (currently, no released version with DistilBERT exists).

[alanakbik]

Perhaps move these declarations to the if bert_model_or_path.startswith("distilbert"): part of the BertEmbeddings?

This way, it won't trigger an error in Travis when initializing the embeddings classes.

[stefan-it]

Thanks :) I changed the import logic. Now an error message is thrown whenever DistilBert* is not found!

[alanakbik]

Cool, thanks for adding this!

[alanakbik]

👍

[yosipk]

👍

[DecentMakeover]

@stefan-it Hi ,i wanted to try this out, can i just do a git pull ? , i am currently at

 * (HEAD detached at origin/GH-873-pytorch-transformers)
 master
 origin/master

[stefan-it]

Yes, checkout to master and then a git pull should work :)

[DecentMakeover]

When trying to run this i get

I have already installed pytorch-transformers and have pulled the latest master code.
Any ideas?

2019-08-30 18:32:04,354 ATTENTION! To use DistilBert, please first install a recent version of pytorch-transformers!
2019-08-30 18:32:04,354 ----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
File "distill_bert.py", line 53, in
BertEmbeddings("distilbert-base-uncased")
File "/media/bubbles/fecf5b15-5a64-477b-8192-f8508a986ffe/ai/flair/flair/embeddings.py", line 1990, in init
self.tokenizer = DistilBertTokenizer.from_pretrained(bert_model_or_path)
UnboundLocalError: local variable 'DistilBertTokenizer' referenced before assignment

[stefan-it]

Try:

$ git clone https://github.com/huggingface/pytorch-transformers.git
$ cd pytorch-transformers
$ pip install -e .

I used that during development :)

[DecentMakeover]

Okay let me check On 30-Aug-2019, at 6:59 PM, Stefan Schweter <notifications@github.com<mailto:notifications@github.com>> wrote: Try: $ git clone https://github.com/huggingface/pytorch-transformers.git $ cd pytorch-transformers $ pip install -e . I used that during development :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1044>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGD5QFYLHLODM73AVQFF3Q3QHEODBANCNFSM4ISBWN7Q>.

[DecentMakeover]

works like a charm , Thanks.