v.2.0.0

Tinyapper 2.0.0

More Tinyapper, the LLM trained from scratch! This time the model is trained to be conversational, along with a saving KV cache.

Tech Changes

• New conversational demo at tinyapper.craisin.tech
• Updated attention conditional for multiple infills for chat
• Updated KV cache so chats can be cached
• Reiterated how datasets are made
• Retrained model for yapperpedia so the model is slightly more coherent

Future Development

• Standardizing datasets to use memmap for larger datasets
• Retrain conversational model
• Fine tune the model

v1.1.0

• Implemented variable learning rate
• Added a validation split and test
• Shrunk container size
• Retrained model for 5 epochs

v1.0.0

Tinyapper 1.0.0

The first release for Tinyapper, the tiny LLM built from scratch! It so far still only exists as a pretrained transformer, I hope to finetune v2 using a dataset for alpaca. A lot of time was spent cleaning up yapperpedia, the demo I created to generate incoherent Wikipedia articles. However, a lot of time was also spent making the code accessible and modular, as I want to further build on this project and encourage others to understand machine learning fundamentals as well. If you are interested, check out local.ipynb!

Tech

• 50M parameter pretrained transformer model trained on wikitext recreating transformer architecture similar to GPT-2
• Flask backend with plain HTML, CSS, and JS website showing the abilities of Tinyapper

Struggles

• KV caching and embedding: I tried for too long to create a looping cache, but that was both slow and impractical to do at the current state of this project
• Streaming: Getting the stream sent over without bugs and glitches was a giant headache!

Future Development

• Tuning the language to be conversational
• Creating a context independent KV cache
• Increasing model size

v1.0-beta.1

KV cache is implemented! This runs somewhat ok on CPU, but is still too slow for prod.

v1.0-beta.0

Tinyapper is now trained! Provided GPU access, local.ipynb works for both testing and inference, and Yapperpedia can be deployed. However, I would like to add a KV cache and model quantization before publishing the demo site in order to enable CPU inference.

Full Changelog: https://github.com/craisined/tinyapper/commits/v1.0-beta.0