Skip to content

Commit

Permalink
[DOC] Add to the gallery an example on a small language model
Browse files Browse the repository at this point in the history
This example demonstrates how to train a small-scale transformer-based language model (inspired by NanoGPT) on the Tiny Shakespeare dataset.  The core idea is to train a model that can predict the next character in a sequence of text based on the characters that came before it.

PiperOrigin-RevId: 611085293
  • Loading branch information
fabianp authored and OptaxDev committed Mar 25, 2024
1 parent b555b4f commit 36f6a87
Show file tree
Hide file tree
Showing 4 changed files with 752 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ def new_process_docstring(app, what, name, obj, options, lines):
nb_execution_allow_errors = False
nb_execution_excludepatterns = [
# slow examples
'nanolm.ipynb',
'cifar10_resnet.ipynb',
'adversarial_training.ipynb',
'reduce_on_plateau.ipynb',
Expand Down Expand Up @@ -296,6 +297,7 @@ def linkcode_resolve(domain, info):

intersphinx_mapping = {
'jax': ('https://jax.readthedocs.io/en/latest/', None),
'flax': ('https://flax.readthedocs.io/en/latest/', None),
}

source_suffix = ['.rst', '.md', '.ipynb']
17 changes: 17 additions & 0 deletions docs/gallery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,23 @@
<div class="sphx-glr-thumbnail-title">Adversarial training of CNN on MNIST.</div>
</div>


.. raw:: html

<div class="sphx-glr-thumbcontainer" tooltip="Character-level Transformer on Tiny Shakespeare.">

.. only:: html

.. image:: /images/examples/tiny_shakespeare.png
:alt: Character-level Transformer on Tiny Shakespeare

:doc:`_collections/examples/nanolm`

.. raw:: html

<div class="sphx-glr-thumbnail-title">Character-level Transformer on Tiny Shakespeare.</div>
</div>

.. raw:: html

</div>
Expand Down
Binary file added docs/images/examples/tiny_shakespeare.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 36f6a87

Please sign in to comment.