Getting Started

Getting Started

Requirements

• Python 3.12+ (as described by the supplied README).
• PyTorch >=2.6, tqdm>=4.66 for the offline smoke test.
• Transformers >=5,<6 and Datasets >=4,<6 for Yelp-based experiments.
• A compatible CUDA-enabled PyTorch build if you intend to use an NVIDIA GPU.
• Internet connectivity on first real-data run for Hugging Face artifacts unless already cached.

The original dependency list is preserved as transformer_from_scratch_requirements.txt. The repository package additionally supplies requirements.txt with identical dependency declarations because the project's README expects that filename.

Windows PowerShell

cd E:\YourPath\LLM_from_scratch
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip check

If activation fails, run .\.venv\Scripts\python.exe directly.

Offline smoke test

python transformer_from_scratch.py --smoke-test

Expected: 64 training, 8 validation, 8 test synthetic reviews; 2 epochs; CPU; a checkpoint at transformer_checkpoints/best_transformer_static_emb.pt; final metrics. These numbers do not constitute a Yelp benchmark.

Small Yelp experiment

python transformer_from_scratch.py --train --sdpa

Custom experiment

python transformer_from_scratch.py --train --sdpa --train-size 2000 --val-size 300 --test-size 300 --batch-size 4 --max-len 128 --epochs 3

Larger Yelp preset

python transformer_from_scratch.py --full --sdpa

--full forces its own sizes, batch size, sequence length, and epoch count (see Execution Modes).

Optional architecture PNG

python -m pip install torchview graphviz
python transformer_from_scratch.py --smoke-test --visualize

The Graphviz system executable is also needed to render PNG successfully. The Mermaid charts in this Wiki require no optional Python packages to view on GitHub.