retrained bert_tiny_uncased_en_sst2_training.ipynb

[susnato]

What does this PR do?

As discussed in this issue, this PR updates the previous training script for finetuning BERT on SST2.

The colab link is this : https://colab.research.google.com/drive/1afTO0ahF3vZrJtkVSGXwBLV2OGeDfomL?usp=sharing

Valdiation Scores achieved :

Epoch 1/2
4210/4210 [==============================] - 828s 191ms/step - loss: 0.3782 - sparse_categorical_accuracy: 0.8299 - val_loss: 0.4344 - val_sparse_categorical_accuracy: 0.8165
Epoch 2/2
4210/4210 [==============================] - 783s 186ms/step - loss: 0.2409 - sparse_categorical_accuracy: 0.9039 - val_loss: 0.4626 - val_sparse_categorical_accuracy: 0.8222

@chenmoneygithub

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[susnato]

I signed the form so cla/google is green now.

[susnato]

Hi @chenmoneygithub I finetuned BertClassifier again on SST2 as you said. Please check it.

[chenmoneygithub]

@susnato Thanks a lot! Looks beautiful overall!

Another thing you may want to do is to use a decayed learning rate:

lr = tf.keras.optimizers.schedules.PolynomialDecay(
    5e-5,
    decay_steps={total_training_steps},
    end_learning_rate=0.0,
)

Please let me know how it works, thx again!

[mattdangerw]

Just a drive by comment, but we should replace, not copy, the old colab with this PR. We don't want to keep spawning versions of the colab each time we update something.

[susnato]

Hi @chenmoneygithub thanks for the reply! I ran as you instructed, it gave -

Epoch 1/2
4210/4210 [==============================] - 871s 201ms/step - loss: 0.4560 - sparse_categorical_accuracy: 0.7970 - val_loss: 0.5304 - val_sparse_categorical_accuracy: 0.7534
Epoch 2/2
4210/4210 [==============================] - 818s 194ms/step - loss: 0.3556 - sparse_categorical_accuracy: 0.8541 - val_loss: 0.5900 - val_sparse_categorical_accuracy: 0.7385

Since it's working relatively worse should I stick with constant lr?

[chenmoneygithub]

@susnato Hi! sorry for the late reply, I was on vacation the last week. This is a bit odd to me, in my experiments the decayed lr generally works better (but I tried it with BERT base model). If this is stable on your side, it sounds good to stay with the constant lr.

[chenmoneygithub]

nit: newline at the end

[chenmoneygithub]

nit: delete the empty cell

[susnato]

Hi @chenmoneygithub thanks a lot for your comments! I made the changes you requested, please check them.

[chenmoneygithub]

@susnato Oh one more thing - could you run ./shell/format.sh in your branch? the style check is broken.

[susnato]

@chenmoneygithub Done!

[susnato]

Hi @chenmoneygithub the Check the code format is passed! but there seems to be an error with keras-nlp-accelerator-testing is this something related to my code?

[chenmoneygithub]

@susnato That one is okay, don't worry.