Text Generation with Transformer Decoder example #914
Conversation
| @@ -0,0 +1,229 @@ | |||
| """ | |||
| Title: Text Generation with Keras NLP TransformerDecoder | |||
There was a problem hiding this comment.
Let's add gpt in the title somewhere, it will be popular :)
There was a problem hiding this comment.
added gpt to title
| start_tokens.append(sample_token) | ||
| num_tokens_generated += 1 | ||
| txt = self.tokenizer.detokenize(start_tokens) | ||
| print(f"generated text: \n{txt}\n") |
There was a problem hiding this comment.
maybe let's show greedy, top-k and random all together each epoch?
There was a problem hiding this comment.
Added the utility functions in the inference section. However, changed it from printing every epoch because it would get pretty messy with too many prints and also a long callback class. Instead, I gave a short callback wrapper example at the end with the top k utility.
| Description: Implementation of a small GPT-like model using the TransformerDecoder class. | ||
| """ | ||
| """ | ||
| # Download Library |
There was a problem hiding this comment.
Note that section titles should use ##
| Date created: 2022/06/13 | ||
| Last modified: 2022/06/13 | ||
| Description: Implementation of a small GPT-like model using the TransformerDecoder class. | ||
| """ |
There was a problem hiding this comment.
Add an Introduction section explaining what the example is about, what dataset you will use, etc.
There was a problem hiding this comment.
added introduction section, with a high level description of the components in the notebook
| import tensorflow as tf | ||
| from tensorflow import keras | ||
| import numpy as np | ||
| from keras_nlp.layers.transformer_decoder import TransformerDecoder |
| model = keras.Model(inputs=inputs, outputs=outputs) | ||
| loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) | ||
| model.compile( | ||
| "adam", loss=loss_fn, |
There was a problem hiding this comment.
Add metrics and use a keyword argument for the optimizer
| start_tokens.append(sample_token) | ||
| num_tokens_generated += 1 | ||
| txt = self.tokenizer.detokenize(start_tokens) | ||
| print(f"generated text: \n{txt}\n") |
|
|
||
| model = create_model() | ||
|
|
||
| model.fit( |
There was a problem hiding this comment.
Add a section for evaluation / inference and a conclusion section summarizing what was learned from the example.
There was a problem hiding this comment.
added inference section and concluding paragraph
examples/nlp/text_generation_gpt.py
Outdated
| """ | ||
|
|
||
| # Download vocabulary data. | ||
| vocab_file = keras.utils.get_file( |
There was a problem hiding this comment.
should we use the word piece vocab learner utility here? that could also simplify the code explanation above.
examples/nlp/text_generation_gpt.py
Outdated
|
|
||
|
|
||
| def create_model(): | ||
| inputs = keras.layers.Input(shape=(SEQ_LEN,), dtype=tf.int32) |
There was a problem hiding this comment.
given that this is only called once, why not just move this out of the function?
examples/nlp/text_generation_gpt.py
Outdated
| x = embedding_layer(inputs) | ||
| # Transformer decoders. | ||
| for _ in range(NUM_LAYERS): | ||
| transformer_block = keras_nlp.layers.TransformerDecoder( |
There was a problem hiding this comment.
decoder_layer maybe to agree with embedding_layer naming
examples/nlp/text_generation_gpt.py
Outdated
| model = create_model() | ||
|
|
||
| """ | ||
| Let's take a look at our model summary here! We can see that a large majority of the |
There was a problem hiding this comment.
remove the exclamation point here, so the one in the next sentence hits better.
examples/nlp/text_generation_gpt.py
Outdated
| ## Training | ||
|
|
||
| Now that we have our model, let's train it. We use a subset of the training data to save | ||
| on training time. It would also be beneficial to use a GPU to speed up the training |
There was a problem hiding this comment.
I'm not sure the GPU part bears mentioning here, maybe just at the top of the colab, say if you are running in a colab make sure to enable the gpu runtime for faster training performance.
examples/nlp/text_generation_gpt.py
Outdated
| # Training | ||
| LEARNING_RATE = 5e-4 | ||
| EPOCHS = 12 | ||
| NUM_TRAINING_BATCHES = 1000 |
There was a problem hiding this comment.
why do we need this hyperparameter, can we just train over the full dataset?
There was a problem hiding this comment.
removed hyperparameter
examples/nlp/text_generation_gpt.py
Outdated
|
|
||
| With our trained model, we can test it out to gauge it's performance. Since | ||
| this is a dataset of mostly fictional books, there is bound to be a hero, so let's use | ||
| "The hero" as our starting string! We run it through the tokenizer to get the input for |
examples/nlp/text_generation_gpt.py
Outdated
|
|
||
| def preprocess(inputs): | ||
| outputs = tokenizer(inputs) | ||
| features = outputs[:, :-1] |
There was a problem hiding this comment.
should we add bos tokens here? then you could sample without seed text, and I think we would be a little closer to actual gpt
examples/nlp/text_generation_gpt.py
Outdated
| MAX_PREDICT_LEN = 80 | ||
| start_prompt = "The hero" | ||
| # Unpadded token sequence. | ||
| start_tokens = [tokenizer.token_to_id(_) for _ in start_prompt.lower().split()] |
There was a problem hiding this comment.
this feels awkward, we should actually use a tokenizer to tokenize the text. maybe let's just instantiate the tokenizer without a sequence length, and either use ragged.to_dense() or a packer layer to densify (especially if we decide to add start tokens)
There was a problem hiding this comment.
edited to use packer layer for start tokens
| model.fit(ds.take(1), verbose=2, epochs=2, callbacks=[text_generation_callback]) | ||
|
|
||
| """ | ||
| ## Conclusion |
There was a problem hiding this comment.
i'm not sure this conclusion section adds too much. we should either replace it with a further reading/where to next section, or remove it entirely
examples/nlp/text_generation_gpt.py
Outdated
| model with few parameters. | ||
|
|
||
| This example combines concepts from [Text generation with a miniature GPT](https://keras.io/examples/generative/text_generation_with_miniature_gpt/) | ||
| with KerasNLP abstractions. We will demonstrate how KerasNLP tokenization, model, metrics, and |
There was a problem hiding this comment.
Maybe just:
We will demonstrate how KerasNLP tokenization, layers and metrics simplify the training
process, and then show how to generate output text using sampling utilities.
And then remove the whole next paragraph. Readers can read on to see the exact layers used.
examples/nlp/text_generation_gpt.py
Outdated
| @@ -0,0 +1,401 @@ | |||
| """ | |||
| Title: Simple GPT Text Generation with KerasNLP transformers | |||
There was a problem hiding this comment.
Just "with KerasNLP" (no transformers)
There was a problem hiding this comment.
Also don't capitalize all words unless they're proper nouns
examples/nlp/text_generation_gpt.py
Outdated
| Author: [Jesse Chan](https://github.com/jessechancy) | ||
| Date created: 2022/07/25 | ||
| Last modified: 2022/07/25 | ||
| Description: Using KerasNLP transformers to train a mini-GPT model for text generation. |
examples/nlp/text_generation_gpt.py
Outdated
| ## Introduction | ||
|
|
||
| In this example, we will use KerasNLP layers to build a scaled down Generative | ||
| Pre-trained (GPT) model. GPT is a transformer based model that allows you to generate |
examples/nlp/text_generation_gpt.py
Outdated
|
|
||
| In this example, we will use KerasNLP layers to build a scaled down Generative | ||
| Pre-trained (GPT) model. GPT is a transformer based model that allows you to generate | ||
| sophisticated text from a small input. |
examples/nlp/text_generation_gpt.py
Outdated
| metrics simplify the training | ||
| process, and then show how to generate output text using sampling utilities. | ||
|
|
||
| Note: If you are running this on a colab make sure to enable GPU runtime for faster |
examples/nlp/text_generation_gpt.py
Outdated
| prompt_tokens = tf.convert_to_tensor([tokenizer.token_to_id("[BOS]")]) | ||
|
|
||
| """ | ||
| We will use the `keras_nlp.utils` library for inference. Every text generation |
There was a problem hiding this comment.
Say "module" rather than library
examples/nlp/text_generation_gpt.py
Outdated
|
|
||
| """ | ||
| We will use the `keras_nlp.utils` library for inference. Every text generation | ||
| utility would require a `token_logits_fn()` wrapper around the model. This wrapper takes |
examples/nlp/text_generation_gpt.py
Outdated
| ) | ||
|
|
||
| """ | ||
| ## Train Tokenizer |
There was a problem hiding this comment.
In every section title, only capitalize the first word, not all words
examples/nlp/text_generation_gpt.py
Outdated
| """ | ||
| ## Conclusion | ||
|
|
||
| Congrats, you made it through the example! To recap, in this example, we use KerasNLP |
There was a problem hiding this comment.
Drop "Congrats, you made it through the example!"
| model, and perform inference with the text generation library. | ||
|
|
||
| If you would like to understand how transformers work, or learn more about training the | ||
| full GPT model, here are some further readings: |
jessechancy
force-pushed
the
jesse-transformerdecoder-tutorial
branch
from
2332f50
to
c79ee11
Compare
The colab I've written before to test out transformer decoder. I've formatted it to be one of the example on the keras.io website.