LLM-Tutor

A GPT-style language model implementation built from scratch using PyTorch. This project demonstrates the core concepts of transformer-based language models, including multi-head attention, positional encoding, and character-level text generation.

Features

• Transformer Architecture: Full implementation of GPT-style transformer blocks
• Multi-Head Self-Attention: Parallel attention mechanisms for capturing different aspects of text
• Character-Level Tokenization: Simple and effective character-based encoding/decoding
• Memory-Mapped File Reading: Efficient handling of large text datasets
• CUDA Support: GPU acceleration for faster training
• Customizable Hyperparameters: Easily adjustable model configuration

Model Architecture

The model consists of:

• Token and position embeddings
• 4 transformer blocks with:
  • Multi-head self-attention (4 heads)
  • Feed-forward neural networks
  • Layer normalization
  • Residual connections
• Dropout regularization (20%)
• Final linear projection to vocabulary

Hyperparameters

batch_size = 128       # Number of sequences per batch
block_size = 64        # Maximum context length
max_iters = 3000       # Training iterations
learning_rate = 3e-3   # AdamW optimizer learning rate
n_embd = 384          # Embedding dimension
n_head = 4            # Number of attention heads
n_layer = 4           # Number of transformer blocks
dropout = 0.2         # Dropout probability

Requirements

• Python 3.7+
• PyTorch
• CUDA-capable GPU (recommended)

Install dependencies:

pip install torch

Project Structure

LLM-tutor/
├── gpt_v1.ipynb          # Main notebook with model implementation
├── wizard_of_oz.txt      # Training text dataset
├── vocab.txt             # Vocabulary file (required)
├── README.md             # Project documentation
└── LICENSE               # License file

Usage

Training the Model

1. Prepare your dataset:

   • Place your training text in vocab.txt for vocabulary generation
   • For larger datasets, use the memory-mapped file approach with train/val splits

2. Open and run the notebook:

   jupyter notebook gpt_v1.ipynb

3. The model will:

   • Initialize with random weights
   • Train for 3000 iterations
   • Evaluate on train/val splits every 100 iterations
   • Save the trained model to model-01.pkl

Generating Text

After training, generate text using:

prompt = 'Hello! Can you see me?'
context = torch.tensor(encode(prompt), dtype=torch.long, device=device)
generated_chars = decode(m.generate(context.unsqueeze(0), max_new_tokens=100)[0].tolist())
print(generated_chars)

Model Components

Head (Self-Attention)

Single attention head implementing scaled dot-product attention with causal masking.

MultiHeadAttention

Combines multiple attention heads in parallel to capture diverse patterns.

FeedForward

Two-layer MLP with ReLU activation and dropout for non-linear transformations.

Block

Complete transformer block combining self-attention and feed-forward layers with residual connections.

GPTLanguageModel

Full model orchestrating embeddings, transformer blocks, and output projection.

Training Tips

1. CUDA Errors: If you encounter CUDA errors, ensure:

   • Vocabulary size matches your vocab.txt
   • Input indices are within valid range
   • Sufficient GPU memory is available

2. Memory Management: For large datasets, use the memory-mapped file reading approach

3. Hyperparameter Tuning: Adjust learning rate, model size, and training iterations based on your dataset

Future Improvements

• Add validation loss monitoring
• Implement learning rate scheduling
• Support for byte-pair encoding (BPE)
• Model checkpointing during training
• Beam search for better text generation
• Support for custom datasets
• Training visualization with tensorboard

License

Copyright (c) 2025 EgoHackZero. All rights reserved.

See LICENSE file for details.

Author

EgoHackZero

Acknowledgments

This implementation is inspired by the GPT architecture and modern transformer-based language models. The code demonstrates fundamental concepts in deep learning and natural language processing.

References

• "Attention Is All You Need" - Vaswani et al., 2017
• "Language Models are Unsupervised Multitask Learners" - Radford et al., 2019
• PyTorch Documentation

---

Built with ❤️ for educational purposes and deep learning exploration.