Transformers

A pair of from-scratch, dependency-light Python language models that share a tokenizer format and a CLI shape, so you can compare a "real" neural approach against a tiny statistical baseline on the same data, on hardware as small as a phone.

• transformers.py — a small decoder-only Transformer implemented directly on top of NumPy (or CuPy if available), with multi-head attention, positional encoding, layer norm, feedforward blocks, dropout, train/val split and early stopping.
• tinytot.py — a trie-backed n-gram model with the same Tokenizer / predict / saveModel / loadModel surface, used as a baseline.
• benchmark_quick.py — a hardened side-by-side benchmark that runs each model in its own subprocess with a hard timeout, so a hang or OOM in one side can never lock up the comparison.

Both models train and run on plain CPU NumPy. CuPy and rich are picked up automatically if installed, but neither is required.

Features

• From-scratch Transformer: Multi-head self-attention, sinusoidal positional encoding, residual + LayerNorm blocks, dropout, softmax/CE loss, manual backprop, validation-loss early stopping.
• Trie n-gram baseline (TinyTot): Builds a prefix trie up to MAX_NGRAM_SIZE tokens deep, samples the next token from observed continuations with temperature.
• Shared tokenizer format: Word-level Tokenizer saved as .tkn, model weights saved as .npz. Tokenizer files are interchangeable in shape between the two models.
• Optional GPU: transformers.py and tinytot.py both import cupy as cp if available, otherwise fall back to NumPy. No code changes needed.
• Optional progress bars: rich.progress is used if installed; otherwise training is silent but functional.
• Hardened benchmark: benchmark_quick.py pins BLAS to one thread, runs each side in a subprocess with a wall-clock timeout, parses results from a single JSON line, and always cleans up its scratch files — even on KeyboardInterrupt or timeout.

Installation

The only hard requirement is NumPy.

pip install numpy
# Optional, for nice progress bars:
pip install rich
# Optional, for GPU (only if your machine has CUDA + a matching CuPy wheel):
pip install cupy-cuda12x   # pick the variant that matches your CUDA version

Then clone and run directly — there is no build step.

git clone https://github.com/guilt/Transformers.git
cd Transformers

Usage

Train / generate with the Transformer

transformers.py looks for transformers.npz + transformers.tkn in the current directory. If they exist, it loads them. Otherwise it reads input.txt (or falls back to a built-in sample text) and trains.

python transformers.py
# Enter a prompt (or press Enter for 'hello'): the quick brown
# Generated text: the quick brown fox jumps over the lazy dog ...

To train on your own corpus, drop a plain-text file at input.txt and delete any stale transformers.npz / transformers.tkn so it retrains.

Train / generate with TinyTot

python tinytot.py
# Enter a prompt (or press Enter for 'hello'): the quick brown
# Generated text: the quick brown fox jumps over the lazy dog ...

Same convention: tinytot.npz + tinytot.tkn are the persisted artefacts.

Run the side-by-side benchmark

python benchmark_quick.py

This trains both models on the same ~92-word corpus with the same prompt ("the quick brown", maxLength=12, temperature=0.5), then prints a table.

Benchmark Results

Below is a representative run from benchmark_quick.py on a Termux/Android device with BLAS pinned to a single thread. Numbers will of course differ on your hardware — the shape of the comparison is what matters.

================================================================
Metric                     Transformer         TinyTot     Ratio
----------------------------------------------------------------
Train (1 iter)                 17.59 s         1.99 ms  8830.92x
Inference                       1.10 s        29.27 ms    37.64x
Model file                    27.27 MB         4.29 KB  6513.18x
Tokenizer file                869.00 B        758.00 B     1.15x
Total on disk                 27.27 MB         5.03 KB  5554.34x
Peak RSS                     127.42 MB        31.58 MB     4.03x
Wall time                      19.09 s       388.38 ms    49.15x
================================================================

Sample output:
  Transformer: 'the quick brown is and is and fox coding python in coding'
  TinyTot    : 'the quick brown fox jumps over the lazy dog machine learning is'

Reading the table

The Ratio column is Transformer / TinyTot for every row, so a ratio of Nx means "the Transformer cost N times more on this metric." Higher = the Transformer paid more.

• Train (1 iter) — 17.59 s vs 1.99 ms (~8,800x). TinyTot's "training" is just walking the corpus once and incrementing counters in a trie, so it finishes in milliseconds. The Transformer runs a full forward pass, manual backprop, and weight update over the dataset for one epoch. Even one epoch dominates by ~4 orders of magnitude.

• Inference — 1.10 s vs 29.27 ms (~38x). The gap shrinks dramatically at generation time. TinyTot just looks up a prefix in the trie and samples from observed continuations — O(context length). The Transformer has to run the full attention stack for every generated token. Inference is where the Transformer is "only" ~38x slower, not thousands.

• Model file — 27.27 MB vs 4.29 KB (~6,500x). The Transformer stores embeddings, attention weights (Q/K/V/O for every head and layer), feedforward matrices, and layer-norm parameters as .npz. TinyTot stores a pickled trie of token-id transitions; on a small corpus that is just a few kilobytes.

• Tokenizer file — 869 B vs 758 B (~1.15x). Both models use the same word-level Tokenizer, so this row is essentially a sanity check: tokenizer size is driven by vocabulary, not by model choice, and the ~1.15x difference here is just the handful of extra tokens the Transformer's training routine happened to surface.

• Total on disk — 27.27 MB vs 5.03 KB (~5,500x). Dominated by the model file. If you ship the model with your app, this is the bytes-on-the-wire cost.

• Peak RSS — 127.42 MB vs 31.58 MB (~4x). Resident memory while training + generating. The Transformer pulls in NumPy, allocates activation buffers for batched forward/backward passes, and keeps gradient tensors live. TinyTot just holds the trie and the tokenizer.

• Wall time — 19.09 s vs 388.38 ms (~49x). End-to-end "import → tokenize → train → save → generate → exit" measured from the parent process. This is the number you actually feel when you run the script.

What the sample output tells you

Transformer: 'the quick brown is and is and fox coding python in coding'
TinyTot    : 'the quick brown fox jumps over the lazy dog machine learning is'

On a 92-word corpus and a single training iteration, the Transformer hasn't seen the data nearly enough times to learn meaningful next-token distributions, so its sampling collapses to high-frequency function words (is, and, in) with only the occasional content word leaking through. TinyTot, by contrast, has memorised the exact bigrams and trigrams it saw — the quick brown → fox → jumps → over is literally a path in its trie — so on in-distribution prompts it looks better than the Transformer.

This is the headline takeaway of the benchmark: on tiny data with tiny compute, a memorising n-gram trie beats a from-scratch Transformer on every axis that matters — speed, size, memory, and even apparent fluency. The Transformer's advantages (generalisation, longer-range structure, compositionality) only start to pay back at scales well beyond what benchmark_quick.py exercises. The benchmark is intentionally small so it finishes in seconds on a phone; if you want to see the Transformer pull ahead on quality, increase numEpochs, the corpus size, and MAX_LEN, and re-run.

Reproducing

python benchmark_quick.py

Tunables live at the top of benchmark_quick.py:

• TINY_TEXT — the corpus (currently ~92 words).
• PROMPT, MAX_LEN, TEMPERATURE — generation knobs.
• TRANSFORMER_TIMEOUT_S, TINYTOT_TIMEOUT_S — per-side wall-clock kill switch (180 s and 60 s by default).
• SINGLE_THREAD_ENV — pins OpenMP/OpenBLAS/MKL/NumExpr/VecLib to one thread so a small CPU isn't oversubscribed.

The benchmark exits non-zero if either side fails or times out, so you can drop it into CI.

Files

File	What it is
transformers.py	NumPy/CuPy decoder-only Transformer + training loop.
tinytot.py	Trie n-gram baseline with the same model interface.
benchmark_quick.py	Subprocess-isolated benchmark with hard timeouts.
benchmark_quick_results.txt	Last captured benchmark output, for reference.
transformers.npz / .tkn	Pre-trained Transformer weights and tokenizer.
tinytot.npz / .tkn	Pre-trained TinyTot trie and tokenizer.

Development

To extend or modify:

• Change Transformer architecture: edit the constants block at the top of transformers.py (EMBED_SCALE, BASE_NUM_HEADS, HEADS_PER_ENTROPY, BASE_NUM_LAYERS, LAYERS_PER_ENTROPY, FF_MULTIPLIER). Layers and heads scale with the entropy of the training corpus.
• Change training: TrainingConfig (numEpochs, batchSize, learningRate, validationSplit, patience) is the single struct consumed by trainModel(...).
• Change TinyTot context size: MAX_NGRAM_SIZE in tinytot.py, or pass maxNgramSize= to TrieModel(...).
• Run on GPU: just pip install a CuPy wheel that matches your CUDA version; both scripts will pick it up automatically via the try: import cupy as cp shim at the top.

Dependencies

• numpy — required, the actual math.
• cupy — optional, drop-in GPU acceleration.
• rich — optional, training progress bars.

License

MIT License. See LICENSE.md for details.

Feedback

Built by Vibe coding. Pull requests, issues, and feature requests are welcome! Vibe code it, Vibe debug it, Vibe test it, Vibe PR it, Vibe everything it.

Authors: xAI Grok, Anthropic Claude and Debugger: Karthik Kumar Viswanathan Web : http://karthikkumar.org Email : me@karthikkumar.org

Happy transforming!