BERT / ELMO embeddings for NER

[RyanDsilva]

When are the pretrained embeddings for BERT and ELMO with regards to NER planned for ?
Could you help me with the development process, I could try to contribute these features.

[amaiya]

This sounds like a good first issue if you want to take a crack at it. I'm happy to answer any questions regarding development process.

One way to implement this is to maybe leverage TensorFlow Hub to generate the embeddings dynamically as sentences are transformed for input into the NN.

[RyanDsilva]

Great! I'd definately want to try this! Taking this up! Thanks @amaiya

[code4kunal]

Hey @amaiya, Thanks a lot for this great work till date. This package has been so intuitive and helpful for most of the tasks. I have tested BERT with text classification, worked like a charm.

Wanted to know when we can have NER tasks supported with bert, distilled-bert embeddings?

Best Regards
Kunal

[amaiya]

@code4kunal Thanks for your comments.

The user above volunteered to look into this a couple of months ago, but I don't know where it stands., The original idea was to maybe use TensorFlow Hub for this. However, now that the Hugging Face transormers library supports TensorFlow 2, it would probably make more sense to generate the embeddings using the transformers library in ktrain. This is still on the TODO list, which is why this issue is open, but I don't have an exact timeframe for this, unfortunately. Thanks again.

[mdavis95]

This might help someone:
huggingface/transformers#1950 (comment)

[amaiya]

@mdavis95 Thanks - it shouldn't be too difficult to incorporate this for sequence-labeling.

[amaiya]

As v0.12.x of ktrain, BERT and Elmo embeddings for downstream tasks like NER are now supported:

# English NER with BERT embeddings
import ktrain
from ktrain import text
(trn, val, preproc) = text.entities_from_conll2003(train.txt, val_filepath=valid.txt)
model = text.sequence_tagger('bilstm-bert', preproc, bert_model='bert-base-cased')
learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=128)
learner.fit(0.01, 2, cycle_len=5)