A production-grade, decoder-only Autoregressive Transformer built from scratch in PyTorch, compiled to WebAssembly via ONNX, and deployed client-side on GitHub Pages.
TwixourGPT is a lightweight, fully modular Causal Language Model designed to showcase modern Large Language Model (LLM) engineering.
It covers the full machine learning lifecycle:
- Model Engineering: Multi-Head Attention, GELU activations, Causal Self-Attention masking, and Layer Normalization built ground-up in PyTorch.
- Instruction Fine-Tuning: Trained on formatted
<|user|>and<|assistant|>dialogue sequences for targeted QA tasks. - WebAssembly Deployment: Exported to ONNX format for 100% serverless, zero-backend, real-time client-side generation inside web browsers via
onnxruntime-web.
Experience real-time token streaming right in your browser: 👉 TwixourGPT Studio (Live WebAssembly App)
TwixourGPT uses a GPT-style decoder-only Transformer setup:
- Embedding Layer: Combines token embeddings and learnable absolute positional embeddings: $$\mathbf{X} = \mathbf{E}{\text{tok}}(\text{idx}) + \mathbf{E}{\text{pos}}(\text{pos})$$
-
Scaled Dot-Product Causal Self-Attention: Enforces autoregressive masking so tokens only attend to past positions:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$$ where$M_{i,j} = -\infty$ for$j > i$ (lower-triangular mask). -
Feed-Forward Network (FFN): Expands embedding dimension by
$4\times$ with GELU non-linearity:$$\text{FFN}(x) = \text{GELU}(xW_1 + b_1)W_2 + b_2$$ - LayerNorm & Residual Connections: Pre-LayerNorm architecture for stable gradient flow.
TwixourGPT/
├── src/
│ ├── __init__.py # Package initialization
│ ├── model.py # PyTorch Transformer architecture (Head, MultiHeadAttention, FFN, Block, TwixourGPT)
│ ├── tokenizer.py # CharacterTokenizer class with JSON export/import
│ └── dataset.py # PyTorch Dataset & DataLoader utilities
├── tests/
│ └── test_model.py # PyTorch shape & forward pass unit tests
├── train.py # CLI training script with dataset loading & weight saving
├── generate.py # CLI interactive terminal generation interface
├── requirements.txt # Python dependencies
├── TwixourGPT.ipynb # Prototyping, training & ONNX export notebook
├── .gitignore
├── LICENSE
├── README.md # Project documentation
└── docs/ # WebAssembly Frontend (GitHub Pages)
├── index.html # iOS Glassmorphic UI layout
├── styles.css # Smooth glassmorphism styling
├── logic.js # ONNX Runtime Web token generation loop
├── model.onnx # Exported ONNX model graph
└── vocab_config.json # Vocabulary character mapping