This Repo is for the Training of Gavin. A Transformer Based Chat Bot. At current time he uses the Reddit Comment Dataset.
Started back in October 2019 (under a different repo), Gavin became my (ScotSurvivor) project outside of studying. At the time, being only 16 years old, Gavin ended up becoming a lot more than just a project, Gavin became the backbone to my applications towards university, A-Levels (UK examinations) & even taking an extra a-level to complete in just one year. Despite my age at the time, I still knew just how much work would be required to achieve the level of coherency & contextual awareness I was aiming for (in fact I still am working towards this!).
At present day Gavin has come a long way, integrating other papers & modules, even having some C++ modules (written by ShmarvDogg & myself) to speed up certain parts. Such as, GavinBackendDatasetUtils as well as GavinTokenizers, which was adapted from the SubwordTextEncoder system that TensorflowDataset uses. Furthermore, Gavin now has ties with an incredible discord bot known as Gerald Written by a close friend of mine Seb Gerald reaches over 60,000 discord members (as of 26/10/2021), which also means, Gavin is being spoken to by this many members.
Gavin's main goal is to be able to speak like your average redditor, while remaining at least some-what humble & polite. This goal is constantly growing in complexity as I aim for better & better coherency. This is being achieved in several ways, check out the whole organisation & the repos within for the individual goals.
Just to be able to train Gavin, this repo also includes some Dataset tools, which are primarily written for my machine (this is due to change, working on an ob#ject-oriented approach now.) For specifics on the training script look at main.py.
- Charles the AI.
- Sentdex Deep Learning Tutorial.
- Sentdex The Man, the myth and the legend.
- Ghosty! Well respected Member of sentdex discord server, he helps test and develop Gavin in many ways. As well as keeping me motivated to work on Gavin, he also helps Give ideas for Gavin's Discord frontend..
- ShmarvDogg. Without him this project wouldn't be possible, special thanks to him for lending me his hardware to train and continue to develop Gavin.
- WhoIsAbishag. For his work with Irene the ChatBot (AI based on OpenAI). He incites, coherent and contextual responses from Chat Bots.
- Transformer Model. Some designs are re-made or reforged, this was a key inspiration.
- Performer Paper. Vital to speed improvements during in train.
(TODO)
(TODO)
(TODO)
(TODO)
- "REFORMER: THE EFFICIENT TRANSFORMER": https://arxiv.org/pdf/2001.04451.pdf
- "Practical and Optimal LSH for Angular Distance": https://arxiv.org/pdf/1509.02897.pdf
- "When and Why is Document-level Context Useful in Neural Machine Translation?": https://arxiv.org/pdf/1910.00294.pdf
- "Attention Is All You Need": https://arxiv.org/pdf/1706.03762.pdf
- "Improved Transformer Architecture for Sequence to Sequence Translation": https://www.cs.princeton.edu/sites/default/files/austin_wang_spring_2019.pdf- "Combining Local and Document-Level Context: The LMU Munich Neural Machine Translation System at WMT19": https://www.aclweb.org/anthology/W19-5345.pdf
- "Improving the Transformer Translation Model with Document-Level Context": https://arxiv.org/pdf/1810.03581.pdf
- "Illustrating The Reformer": https://towardsdatascience.com/illustrating-the-reformer-393575ac6ba0
- "Reformer: The Efficient Transformer: https://openreview.net/pdf?id=rkgNKkHtvB"
- "Rethinking attention with performers":
GNU GPLv3. Should have a copy with this software.