Use KerasTuner to hyper-parameter search for the BERT finetuning script

[mattdangerw]

From the BERT paper...

For fine-tuning, most model hyperparameters are
the same as in pre-training, with the exception of
the batch size, learning rate, and number of training
epochs. The dropout probability was always
kept at 0.1. The optimal hyperparameter values
are task-specific, but we found the following range
of possible values to work well across all tasks:

• Batch size: 16, 32
• Learning rate (Adam): 5e-5, 3e-5, 2e-5
• Number of epochs: 2, 3, 4

We should allow our BERT finetuning script to do this search automatically. KerasTuner is a good fit for this.

Steps:

• Add an setup.py examples dependency on keras-tuner.
• Remove epochs, batch size and learning rage arguments from run_glue_finetuning.py.
• Use keras tuner to hyperparemeter search on the above value ranges with the validation set.

[Stealth-py]

Hey @mattdangerw! I'd like to take this up.

[mattdangerw]

@Stealth-py perfect! Was about to ask based on conversation on #107

[Stealth-py]

Hey @mattdangerw, so my current progress is:

• For step 1: I've added it to the examples requirement list under the extras_require requirements.
• For step 3: The main problem right now is that, I can't test the actual model in run_glue_finetuning.py.
  And, I've been poking around the keras-tuner example and tried to implement the model used in the example, the way our BERT model has been implemented, or should be implemented. The google colab link for the same is here. Been playing around the number of max_trials and executions_per_trial, I went with 3 each but I guess we can change it. It does take a while to train but it seems to be giving interesting results. Although, by the 3rd trial it seems to get less accurate, it probably depends on model to model/dataset to dataset.

Would love to know your opinion on this!

[mattdangerw]

Looks like the BERT example broke on a recent commit. Just pushed a fix. You should be able to test a whole flow with block of code:
https://github.com/keras-team/keras-nlp/tree/master/examples/bert#quickly-test-out-the-code

That should be enough to see that things are working. You should definitely see the loss going down on fine tuning runs.

[Stealth-py]

Oh..
I'll test it soon, thank you!

[Stealth-py]

One more thing, I'm confused while finetuning the batch_size, because when I try to do it, train_ds, test_ds, and validation_ds all three of them uses the batch_size before preprocessing, hence when I try to reference the same thing inside build_model, I get the UnboundLocalError exception so I don't really know how to proceed here. We could either make them global variables or I was thinking if maybe we can do the same thing for all batch_sizes but manually, rather than using keras-tuner, but that defeats the purpose of this issue and using global variables is generally a bad practice.

[Stealth-py]

@mattdangerw, update: So, after fine-tuning with the parameters given in the paper I've managed to get an accuracy of 0.6829 after evaluation on the test set. I couldn't figure out a way to fine-tune the batch_size, because of the above mentioned issue, so I went ahead with taking batch_size as 32.
The best hyperparameters as per the trials come out to be learning_rate = 5e-05 and epochs = 3.

[Stealth-py]

Would it be beneficial to use this(from the BERT paper):

We use a batch size of 32 and fine-tune for 3
epochs over the data for all GLUE tasks. For each
task, we selected the best fine-tuning learning rate
(among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set.

I've been trying a lot of things, and this seems to work quite well.
Would love to know what you think about this!

[Stealth-py]

Hey @mattdangerw, would love to get your review on this :D

[mattdangerw]

Hey! Sorry about the delay. I would not worry about any accuracy in particular, once we can see the loss is going down, and keras tuner is effectively choosing between alternatives, that's enough to open a PR for further review.

Re that line in the paper, makes sense. Let's leave the defaults for batch size and num_epochs, and just use keras_tuner to choose the best learning rate.

Re batch size question above, I think you would need to follow this section and definte a keras_tuner.HyperModel to effectively tune batch size. But given that snippet you just linked from the paper, let's just skip tuning batch size entirely.

Thanks! (also did I miss anything?)

[Stealth-py]

Thanks for the review! No, I think that's everything yeah.
Alright, I'll open a PR for this in a bit!