Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementing data augmentation from TinyBERT #1874

Closed
Tracked by #1551
MichelBartels opened this issue Dec 13, 2021 · 0 comments · Fixed by #1923
Closed
Tracked by #1551

Implementing data augmentation from TinyBERT #1874

MichelBartels opened this issue Dec 13, 2021 · 0 comments · Fixed by #1923

Comments

@MichelBartels
Copy link
Contributor

MichelBartels commented Dec 13, 2021

Is your feature request related to a problem? Please describe.
A basic version of model distillation was implemented with #1758. However, there is still room for improvement. The TinyBERT paper (https://arxiv.org/pdf/1909.10351.pdf) details an approach for finetuning an already pretrained small language model.

Describe the solution you'd like
Adding the functionality to generate more data samples by using approach outlined in TinyBERT paper. This could be implemented as an additional DataSilo.

Describe alternatives you've considered
https://arxiv.org/pdf/1910.08381.pdf: Seems to depend too heavily on expensive retraining and seems to be too task specific.
https://arxiv.org/pdf/2002.10957.pdf, https://arxiv.org/pdf/1910.01108.pdf: Seem only to focus on pretraining

Additional context
This is the second of two issues for implementing finetuning as described in the TinyBERT paper. This issue focusses on data augmentation. The first issue focussed on the loss functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants