Home

Transformer Encoder From Scratch — Project Wiki

Project scope: A PyTorch Transformer encoder trained from scratch for binary sentiment classification. This is an encoder-only, next-token-generating LLM.A new small GPT-style decoder-only model will be built from scratch in the next iteration . Real-data experiments reuse MiniLM tokenization and a frozen token-embedding matrix; the encoder and classifier are custom modules.

Start here

• Getting Started: install, run the offline smoke test, and start Yelp training.
• System Architecture (HLD): system context, data flow, and component-level model pipeline.
• Low-Level Design (LLD): class relationships, methods, key tensors, and module responsibilities.
• Attention Internals: manual multi-head attention and SDPA comparison.
• Training Workflow: optimizer, validation, checkpoint, and test sequence.
• Data and Embeddings: Yelp splits, tokenization, static embeddings.
• Execution Modes: --smoke-test, --train, and --full.
• CLI Reference: configurable model/training arguments.
• Checkpoint Design: what is saved and what must be reconstructed.
• Validation: smoke-test evidence and validation limits.
• Design Decisions and Roadmap.
• Troubleshooting for Windows, CUDA, package imports and downloads.

Architecture at a glance

flowchart LR
    User["Developer / Researcher"] --> CLI["Python CLI: transformer_from_scratch.py"]
    CLI --> Mode{"Execution mode"}
    Mode -->|"--smoke-test"| Synthetic["Synthetic reviews + SmokeTokenizer"]
    Mode -->|"--train / --full"| Yelp["Hugging Face Yelp Polarity dataset"]
    Mode -->|"--train / --full"| MiniLM["MiniLM tokenizer + frozen embedding weights"]
    Synthetic --> Collator["Tokenizer + frozen embedding lookup on CPU"]
    Yelp --> Collator
    MiniLM --> Collator
    Collator --> Loaders["PyTorch DataLoader: train / validation / test"]
    Loaders --> Trainer["Training / evaluation loop"]
    Trainer <--> Model["Custom Transformer Encoder + Classification Head"]
    Trainer --> Checkpoint["Best validation checkpoint: .pt"]
    Checkpoint --> Test["Reload checkpoint + held-out test evaluation"]
    CLI -.->|"--visualize optional"| Graphviz["torchview + Graphviz PNG"]

Loading

Source code

The main entry point is transformer_from_scratch.py in the code repository. The Wiki is a documentation companion; place executable source files in the main code repository, not only in the Wiki.

Repository contents

transformer_from_scratch.py (original implementation), README.md (original project notes), transformer_from_scratch_requirements.txt (original dependency list), requirements.txt (install-friendly copy), .gitignore, docs/mermaid/ (editable diagrams), wiki/ (Wiki source pages), and scripts/ (publication helpers).