Skip to content

Commit

Permalink
BERT fine-tuning with huggingface transformers
Browse files Browse the repository at this point in the history
Add a notebook that shows what steps need to be taken to fine-tune a
BERT model in skorch with the help of the huggingface transformers
library.

Huggingface transformers is probably the most popular library that
implements BERT and other similar models. Ideally, skorch can work with
transformers with as little friction as possible.

There are actually surprisingly few steps involved, but some of them may
not be obvious to most users, which is why the notebook could be
helpful.

As a cherry on top, the notebook also shows mixed precision training
with huggingface accelerate, which indeed results in a very nice boost
of training and prediction speed.

Other than that, the notebook is actually quite sparse in explanations,
instead referring to the fine-tuning docs of hugginface itself.
  • Loading branch information
BenjaminBossan committed Mar 4, 2022
1 parent c0df4d1 commit cda140e
Show file tree
Hide file tree
Showing 3 changed files with 941 additions and 1 deletion.
4 changes: 3 additions & 1 deletion docs/user/tutorials.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,6 @@ The following are examples and notebooks on how to use skorch.

* `Advanced Usage <https://nbviewer.jupyter.org/github/skorch-dev/skorch/blob/master/notebooks/Advanced_Usage.ipynb>`_ - Dives deep into the inner works of skorch. `Run in Google Colab 💻 <https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/Advanced_Usage.ipynb>`_

* `Gaussian Processes <https://nbviewer.jupyter.org/github/skorch-dev/skorch/blob/master/notebooks/Gaussian_Processes.ipynb>`_ - Train Gaussian Processes with the help of GPyTorch `Run in Google Colab 💻 <https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/Gaussian_Processes.ipynb>`_
* `Gaussian Processes <https://nbviewer.jupyter.org/github/skorch-dev/skorch/blob/master/notebooks/Gaussian_Processes.ipynb>`_ - Train Gaussian Processes with the help of GPyTorch. `Run in Google Colab 💻 <https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/Gaussian_Processes.ipynb>`_

* `Huggingface Finetunging <https://nbviewer.jupyter.org/github/skorch-dev/skorch/blob/master/notebooks/Huggingface_Finetuning.ipynb>`_ - Fine-tune a BERT model for text classification with the huggingface transformers library and skorch. `Run in Google Colab 💻 <https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/Huggingface_Finetuning.ipynb>`_

0 comments on commit cda140e

Please sign in to comment.