Reproduce the results on CoLA

[cooelf]

I try to reproduce the CoLA results reported in the BERT paper but the numbers are far from the reported one. My best mcc (BERT large) for dev is 64.79% and the test result is 56.9% while the reported test result is 60.5%. The learning rate is 2e-5 and the total number of epochs is 5. For BERT base，the result is also lower by 3-5%.

As the paper said,
for BERTLARGE we found that fine-tuning was sometimes unstable on small data sets (i.e., some runs would produce degenerate results), so we ran several random restarts and selected the model that performed best on the Dev set.

I also tried several restarts with different learning rates and random seeds but it seems no improvement. I'm quite confused for the reproduction. Any suggestions would be greatly appreciated.

[Iwontbecreative]

Cola is probably one of the most unstable tasks for BERT. For us it mostly boiled down to running many seeds. If all you care about is a good pre-trained model checkpoint, we have a 65 / 61 run at https://github.com/zphang/bert_on_stilts

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[boyang-nlp]

@cooelf hi, can you reproduce cola score of BERT now? I work on this recently but still can't reach the reported score on the test set, even if restart with different seed multiple times, and I further noticed that improvement on dev set is inconsistent with that on the test set at all.

[cooelf]

Thanks for all the feedbacks!
@scissorsy The results of dev and test are indeed incosistent. I have changed the seeds, lr, warmup rates and max_epochs, and finally the test result reached about 60% for some runs. Using multi-tasking or transfer from bigger dataset such as MNLI seems to be more stable.

[boyang-nlp]

Thanks for all the feedbacks!
@scissorsy The results of dev and test are indeed incosistent. I have changed the seeds, lr, warmup rates and max_epochs, and finally the test result reached about 60% for some runs. Using multi-tasking or transfer from bigger dataset such as MNLI seems to be more stable.

Thanks!

[bgg11117]

Hi @cooelf, what parameter number did you change in order to fit a better result, thanks!