Design Decisions

Design Decisions and Tradeoffs

Decision	Rationale	Tradeoff
Implement encoder blocks manually	Good for learning attention, LayerNorm, residuals and masking	Less optimized than importing a full pretrained encoder
Reuse a frozen pretrained input embedding matrix for Yelp	Provides a meaningful token vocabulary/representation without training embeddings	This is not an end-to-end from-scratch language model
Provide offline smoke mode	Checks core end-to-end training without network/models	Synthetic scores cannot be interpreted as real quality
Separate Q/K/V projections	Makes attention internals explicit	Manual attention materializes an S × S matrix
Offer SDPA	Can use efficient backends when supported	Backend and speed are environment dependent
Pre-LN with GELU FFN	Stable, clear teaching architecture	Does not include newer variations such as RMSNorm / SwiGLU
Fixed sinusoidal encoding	Simple deterministic position signal	No learned positions / RoPE
Mask-aware mean pooling	Aggregates token states for classification	Loses token-level output structure
Save best accuracy checkpoint	Avoids testing the last epoch by default	No optimizer state for resuming training
Run as a CLI	Easy reproducible experiments	Not a serving API or deployment system

Terminology

The repository folder may be called “LLM_from_scratch”, but its code implements a Transformer sentiment classifier. Do not describe this as decoder-only autoregressive text generation, foundation-model pretraining or LLM serving: those components do not exist in the uploaded source.

License and third-party material

The uploaded project contains no user-selected source-code license. This package deliberately does not add one. Choose a license you are authorized to apply separately, and respect the external dataset and pretrained model's respective license terms.