LLMs can learn their own compression

TL;DR: I co-trained a summarizer and a generator to learn a compression scheme for text in the same token space as the base model, so it can continue with almost the same quality using an order of magnitude fewer context tokens. Along the way the model discovers its own compression tricks: aggressive pruning, dense punctuation (lots of semicolons), and even occasionally switching into Mandarin to pack more information per token.

You can read the full blog post here.

Run the Code

It's super simple.

1. Set your TINKER_API_KEY and WANDB_API_KEY in .env file.
2. Run uv sync
3. Run uv run run_train.py