This is a fork of aitextgen notebooks by Max Woolf.
This repository makes a number of demo notebooks available for use in Paperspace Gradient:
| Notebook | Run on Gradient Link |
|---|---|
| Train a GPT-2 model + tokenizer from scratch (GPU) | |
| Finetune OpenAI's 124M GPT-2 model (or GPT Neo) on your own dataset (GPU) | |
| aitextgen Generation Hello World | |
| aitextgen Training Hello World | |
| Hacker News aitextgen | |
| Reddit aitextgen | |
aitextgen is a robust Python tool for text-based AI training and generation using OpenAI's GPT-2 and EleutherAI's GPT Neo/GPT-3 architecture.
aitextgen is a Python package that leverages PyTorch, Hugging Face Transformers and pytorch-lightning with specific optimizations for text generation using GPT-2, plus many added features. It is the successor to textgenrnn and gpt-2-simple, taking the best of both packages:
- Finetunes on a pretrained 124M/355M/774M GPT-2 model from OpenAI or a 125M/350M GPT Neo model from EleutherAI...or create your own GPT-2/GPT Neo model + tokenizer and train from scratch!
- Generates text faster than gpt-2-simple and with better memory efficiency!
- With Transformers, aitextgen preserves compatibility with the base package, allowing you to use the model for other NLP tasks, download custom GPT-2 models from the HuggingFace model repository, and upload your own models! Also, it uses the included
generate()function to allow a massive amount of control over the generated text. - With pytorch-lightning, aitextgen trains models not just on CPUs and GPUs, but also multiple GPUs and (eventually) TPUs! It also includes a pretty training progress bar, with the ability to add optional loggers.
- The input dataset is its own object, allowing you to not only easily encode megabytes of data in seconds, cache, and compress it on a local computer before transporting to a remote server, but you are able to merge datasets without biasing the resulting dataset, or cross-train on multiple datasets to create blended output.
NLP, GPT-2, Educational
By clicking the Run on Gradient button above, you will be launching the contents of this repository into a Jupyter notebook on Paperspace Gradient.
Docs are available at docs.paperspace.com.
Be sure to read about how to create a notebook or watch the video instead!