How to set hyperparameters when fine-tune?

[WeiyueSu]

I fine tune bert in some classification task and get better result, but I have some confuse.

1. I found the fine tuning is unstable. The f1 has 2% diff when fine-tuning with the same hyperparameters. (I tried to tf.set_random_seed(1024) but the result is still different.)
2. A classification task with about 80million train data. In the first epoch, the train loss always keep descending, the valid loss descend at first and increase after about 1 million train data is feed. It means model overfit in the first epoch?
3. Any guide about how to set a better warmup step? I found the fine-tuning performance is highly depend on the epoch num and warmup.

[ymcui]

For question 1, this is an issue of TensorFlow framework. It is known that whether you use fixed random seed or not, the final result may differ when running on GPU and TPU. My own experience is to run it for multiple times and check its avg/std.