TinyLLM Lab

TinyLLM Lab is a pure-programming AI systems project for learning and demonstrating LLM internals without a dashboard or web UI.

It implements the core building blocks behind modern AI systems:

• BPE-style tokenizer training, encoding, decoding, save/load
• Scaled dot-product attention and a tiny transformer block
• Hashing-vector retrieval with cosine search
• Greedy, top-k, top-p, and temperature sampling
• Evaluation metrics for generation and retrieval
• CLI commands for reproducible experiments
• Unit tests for the important behavior

This is built as a code-first portfolio project: small enough to understand from scratch, but structured like a real Python package.

Quick Start

python -m tinyllm_lab tokenize --train-file examples/corpus.txt --text "attention learns context"
python -m tinyllm_lab attention-demo --tokens "llms retrieve context"
python -m tinyllm_lab retrieve --corpus examples/corpus.txt --query "retrieval augmented generation"
python -m tinyllm_lab sample --corpus examples/corpus.txt --prompt "language models" --steps 12 --strategy top-k
python -m tinyllm_lab evaluate --corpus examples/corpus.txt --questions examples/questions.jsonl

Run tests:

python -m unittest discover -s tests

Project Structure

tinyllm_lab/
  tokenizer.py    BPE-style tokenizer from scratch
  attention.py    Pure Python attention functions and multi-head self-attention
  model.py        Tiny transformer block using attention, layer norm, and MLP
  retrieval.py    Hashing embeddings and vector index
  sampling.py     Decoding algorithms and tiny bigram generation
  evaluation.py   Generation and retrieval metrics
  cli.py          Command line interface
tests/            Unit tests
examples/         Small corpus and retrieval questions

Why This Project Is Strong For A Profile

Many AI projects only call APIs. This project shows the internals:

• how text becomes tokens
• how attention mixes context
• how retrieval finds relevant documents
• how decoding controls model output
• how evaluation checks whether a system works

That makes it useful for deep learning, LLM, RAG, and AI systems interviews or research conversations.

No Dashboard

This project intentionally has no frontend and no dashboard. Everything is code, tests, and CLI experiments.