This repository contains the code and results of a research assignment focused on building a task-specific small language model (SLM) from scratch, tailored for legal document generation using the EUR-Lex dataset from the LexGLUE benchmark.
This project explores the training of a small transformer-based language model specialized in legal text processing. We designed and evaluated four different training configurations, analyzing the impact of training iterations, architecture scaling, and hyperparameters on model performance and text generation quality.
- Source: EUR-Lex from LexGLUE
- Domain: European Union legal documents
- Language: Primarily English
- Size:
- Training: ~81 million tokens
- Validation: ~8.7 million tokens
- Preprocessing:
- HTML tag removal
- Unicode normalization
- Byte Pair Encoding (BPE) with 32k vocabulary
- Filtering short/noisy documents
- Batch Size: 16
- Block Size: 64
- Learning Rate: 1e-4
- Results:
- Final Train Loss: 4.44
- Final Val Loss: 4.66
- Partial success in legal tone but low coherence
- Same architecture as Case 1
- Learning Rate: Cyclic schedule (0.0001 to 0.0005)
- Results:
- Final Val Loss: 4.63
- Slight improvement, but diminishing returns after 10k iterations
- Batch Size: 32
- Block Size: 128
- Embedding Dim: 512
- Layers: 8
- Heads: 8
- Results:
- Final Val Loss: 4.43
- Better fluency and structure in legal generation
- Same model as Case 3, extended training
- Embedding Dim: 256
- Layers: 6
- Results:
- Final Train Loss: 3.57
- Final Val Loss: 3.78
- Best performing configuration, significantly more coherent output
- Larger models with longer training significantly improve legal text generation quality.
- Cyclic learning rate schedules proved stable for extended runs.
- Even small models can capture legal structure and terminology with correct configuration.
- Limitations observed in coherence and grammar can be improved with deeper architectures and more training data.
- Complex formatting and structure in legal texts required careful preprocessing.
- Difficulty in balancing model size with memory constraints.
- Managing learning rate schedules to avoid instability in early training.
- Scale up model depth and width
- Use gradient checkpointing for longer contexts
- Incorporate structured awareness of legal document format
- Explore multi-GPU training and legal-specific data augmentation
You can read the full research report here for an in-depth discussion on methodology, experiments, and results.
- Mansi Borle – 2023301002 – mansi.borle23@spit.ac.in
- Manjiri Chavande – 2023301003 – manjiri.chavande23@spit.ac.in