Sequential Deep Learning Models for Text Generation

This project implements and compares three different sequential deep learning models for text generation: RNN, LSTM, and Transformer. The models are trained on a small text dataset from Project Gutenberg and evaluated using perplexity and BLEU scores.

Project Structure

• data/: Contains raw and processed data
  • raw/: Raw text files from Project Gutenberg
  • processed/: Processed data files (train.jsonl, test.jsonl)
  • models/: Saved tokenizer models
• src/: Source code
  • data_processing/: Data loading and tokenization
  • models/: Model implementations
  • training/: Training functionality
  • evaluation/: Evaluation metrics
  • visualization/: Plotting and visualization
• outputs/: Generated outputs
  • models/: Saved model checkpoints
  • figures/: Generated plots
  • results/: Evaluation results

Setup

1. Clone the repository
2. Install the required packages: pip install -r requirements.txt

Usage

Place your raw text files in the data/raw/ directory
Ensure train.jsonl and test.jsonl files are in the data/processed/ directory

Training the Tokenizer

python main.py --train_tokenizer

Training Models

To train a specific model:

# Train the RNN model
python main.py --train_rnn

# Train the LSTM model
python main.py --train_lstm

# Train the Transformer model
python main.py --train_transformer

Training All Models

To train all three models sequentially:

python main.py --train_rnn --train_lstm --train_transformer

Evaluating Models

Evaluating Individual Models

To evaluate a specific model:

# Evaluate the RNN model
python main.py --eval_rnn

# Evaluate the LSTM model
python main.py --eval_lstm

# Evaluate the Transformer model
python main.py --eval_transformer

Evaluating All Models

To evaluate all models and compare their performance:

python main.py --eval_rnn --eval_lstm --eval_transformer --evaluate --custom_prompt "Your custom prompt here"

You can customize the prompt by changing the text inside the quotes.

Full Pipeline

python main.py

Models

1. RNN: A vanilla RNN-based language model
2. LSTM: An LSTM-based language model
3. Transformer: A Transformer-based language model

Evaluation Metrics

• Perplexity (PPL): Measures how well the model predicts the next word
• BLEU Score: Compares generated text against ground truth sentences

Results

Plots and evaluation results will be saved in the outputs/ directory.