keras.io Tutorial : Pre-training ELECTRA Model

[kanpuriyanawab]

Related issues : #107 #189

ELECTRA's excellent efficiency means it works well even at small scale . We could show the GAN-like setup used by ELECTRA with our low-level preprocessing and transformer modeling KerasNLP layers.

I'll be working on it.

[mattdangerw]

Thank you!

[tc-wolf]

Glad you're taking this on!

Would love if masking could be done on-the-fly for this:
#107 (comment)

There is an example ELECTRA implementation in the Tensorflow Model Garden:
https://github.com/tensorflow/models/blob/master/official/nlp/tasks/electra_task.py

And an example of using TF Text ops for dynamic (AFAICT, unless cached this would be dynamic) MLM data generation, although they use TF Model garden-specific 'Dataloader' code:
https://github.com/tensorflow/models/blob/master/official/nlp/data/pretrain_text_dataloader.py

[kanpuriyanawab]

Hi @mattdangerw & @jbischof ,
Actually working on it now.
First of all thank you for amazing Pretraining Transformer from scratch guide !

I did a lot of googling but could not found something, which is intutive and I could refer to, so getting my hands dirty myself. I realized that we can actually use this guide.

My approach :

1. Creating a small generator, using the guide
2. Creating a discriminator using the transformer encoder layers.
3. Getting the predictions from the MaskedLMHead, shape = (pretraining_batch_size, masked_length, embedding_dimension), converting it to shape=(pretraining_batch_size, seq_length) with use of a linear layer
4. transformer encoder discriminator model would transforms (pretraining_batch_size, seq_length) to (pretraining_batch_size, seq_length, model_dim)
5. attach a linear layer to get outputs into shape of input
6. getting predictions and comparing with input to output
7. Training this workflow.

Need your views if it's correct thing to do.

PS: Code : Notebook

PPS: I've been using KerasNLP for a while now on kaggle, the hack by @abheesht17 for python 3.7 seems to work totally fine, !

[kanpuriyanawab]

Hey @mattdangerw, I finally was able to put everything together on ELECTRA's end. Sorry for the delay !

Please checkout this colab, and let me know your thoughts about this, I'm excited about this one tbh !

I would like to give an input from UX perspective, I wish we had a ClassificationHead layer just like MaskedLMHead. We do have individual classifiers, but I believe that such kind of layers that can be stacked on top of a KerasNLP backbone will make library more flexible to use. (Yes, I'm interested in contributing this layer, if you think it is worth a place in KerasNLP) Thanks !

[mattdangerw]

Nice! For now I would just create a classification head if you need it. For BERT the head is really just a single dense layer, so super simple!

I would keep pushing this example along to the point where you are actually running a full example, on say wiki103, and getting some reasonable results. I would also somewhat suspect we need a custom train step to accommodate the full loss setup, but I'm not totally sure.