Skip to content

v1.0.0

Choose a tag to compare

View all tags
@craisined craisined released this 04 Sep 02:17
· 46 commits to master since this release
v1.0.0
771375d

Tinyapper 1.0.0

The first release for Tinyapper, the tiny LLM built from scratch! It so far still only exists as a pretrained transformer, I hope to finetune v2 using a dataset for alpaca. A lot of time was spent cleaning up yapperpedia, the demo I created to generate incoherent Wikipedia articles. However, a lot of time was also spent making the code accessible and modular, as I want to further build on this project and encourage others to understand machine learning fundamentals as well. If you are interested, check out local.ipynb!

Tech

  • 50M parameter pretrained transformer model trained on wikitext recreating transformer architecture similar to GPT-2
  • Flask backend with plain HTML, CSS, and JS website showing the abilities of Tinyapper

Struggles

  • KV caching and embedding: I tried for too long to create a looping cache, but that was both slow and impractical to do at the current state of this project
  • Streaming: Getting the stream sent over without bugs and glitches was a giant headache!

Future Development

  • Tuning the language to be conversational
  • Creating a context independent KV cache
  • Increasing model size
Assets 2
Loading