[DOC] Add to the gallery an example on a small language model

This example demonstrates how to train a small-scale transformer-based language model (inspired by NanoGPT) on the Tiny Shakespeare dataset. The core idea is to train a model that can predict the next character in a sequence of text based on the characters that came before it. PiperOrigin-RevId: 611085293
google-deepmind · Mar 14, 2024 · e8194c5 · e8194c5
1 parent f45b2eb
commit e8194c5
Show file tree

Hide file tree

Showing 4 changed files with 768 additions and 0 deletions.
diff --git a/docs/conf.py b/docs/conf.py
@@ -227,6 +227,7 @@ def new_process_docstring(app, what, name, obj, options, lines):
 nb_execution_allow_errors = False
 nb_execution_excludepatterns = [
     # slow examples
+    'nanogpt.ipynb',
     'cifar10_resnet.ipynb',
     'adversarial_training.ipynb',
     'reduce_on_plateau.ipynb',

diff --git a/docs/gallery.rst b/docs/gallery.rst
@@ -150,6 +150,23 @@
       <div class="sphx-glr-thumbnail-title">Adversarial training of CNN on MNIST.</div>
     </div>
 
+
+.. raw:: html
+
+    <div class="sphx-glr-thumbcontainer" tooltip="Small Transformer Language Model on Tiny Shakespeare.">
+
+.. only:: html
+
+  .. image:: /images/examples/tiny_shakespeare.png
+     :alt: Small Transformer Language Model on Tiny Shakespeare
+
+  :doc:`_collections/examples/nanogpt`
+
+.. raw:: html
+
+      <div class="sphx-glr-thumbnail-title">Small Transformer Language Model on Tiny Shakespeare.</div>
+    </div>
+
 .. raw:: html
 
     </div>

diff --git a/docs/images/examples/tiny_shakespeare.png b/docs/images/examples/tiny_shakespeare.png