Gradual unfreezing and discriminative fine-tuning for BERT

[dchang56]

Three of the tips for fine-tuning proposed in ULMFIT are slanted triangular learning rates, gradual unfreezing, and discriminative fine-tuning.

I understand that BERT's default learning rate scheduler does something similar to STLR, but I was wondering if gradual unfreezing and discriminative fine-tuning are considered in BERT's fine-tuning implementations. Has anyone had experience implementing these two features in BERT fine-tuning? I'd like to hear your thoughts on it. Thanks!

[thomwolf]

I've tried a bit to play with these training schemes on a deep transformer for our tutorial on Transfer Learning in Natural Language Processing held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).

You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.

You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).

It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[forjiuzhou]

I've tried a bit to play with these training schemes on a deep transformer for our tutorial on Transfer Learning in Natural Language Processing held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).

You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.

You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).

It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.

I find the standard fine-tuning procedure having a unstable issue, that different shuffle order affects a lot. I wander if unfreezing help the bert finetune get a relative stable result?
This is also mentioned in bert paper, they just use different random seeds to get a best result.