GenAI-Assignment2: Deep Learning for Vision and NLP

A comprehensive deep learning assignment implementing three state-of-the-art generative AI tasks:

1. CycleGAN for Face-to-Sketch Translation
2. Transformer for English-to-Urdu Machine Translation
3. Diffusion Transformers (SiT) for Image Generation

Author: Muhammad Ibraheem (i212508)
Course: Generative AI
Institution: FAST-NUCES

---

📋 Table of Contents

• Overview
• Project Structure
• Task 1: CycleGAN Face-Sketch Translation
• Task 2: English-Urdu Machine Translation
• Task 3: Diffusion Transformers
• Installation
• Usage
• Results
• References

---

🎯 Overview

This repository contains implementations of three cutting-edge generative AI models:

Task	Model	Description
Task 1	CycleGAN	Unpaired image-to-image translation for photo↔sketch conversion
Task 2	Transformer	Sequence-to-sequence translation for English→Urdu
Task 3	SiT (Scalable Interpolant Transformer)	Diffusion-based image generation on CIFAR-10

---

📁 Project Structure

GenAI-Assignment2/
├── Task1/                          # CycleGAN Face-Sketch Translation
│   ├── model.py                    # Generator & Discriminator architectures
│   ├── train.py                    # Training loop with adversarial loss
│   ├── test.py                     # Inference and evaluation
│   ├── data_loader.py              # Custom dataset loader
│   ├── classifier.py               # Auxiliary classifier
│   ├── gui.py                      # PyQt5 GUI for demo
│   ├── requirements.txt
│   ├── checkpoints/                # Saved models (.pth, .safetensors)
│   └── Data/                       # Train/Val/Test splits
│       ├── train/
│       ├── val/
│       └── test/
│
├── Task2/                          # English-Urdu Translation
│   ├── model.py                    # Transformer architecture
│   ├── train.py                    # Training with teacher forcing
│   ├── evaluate.py                 # BLEU score evaluation
│   ├── evaluate_mbart.py           # mBART fine-tuning evaluation
│   ├── finetune_mbart.py           # Fine-tune mBART for translation
│   ├── dataset.py                  # Parallel corpus loader
│   ├── preprocess.py               # Data preprocessing
│   ├── train_tokenizers.py         # BPE tokenizer training
│   ├── demo.py                     # Interactive translation demo
│   ├── gui.py                      # PyQt5 GUI
│   ├── en_tokenizer.json           # English BPE tokenizer
│   ├── ur_tokenizer.json           # Urdu BPE tokenizer
│   ├── requirements.txt
│   ├── checkpoints/                # Model weights
│   └── Data/                       # Parallel corpus
│       ├── train.en / train.ur
│       ├── val.en / val.ur
│       ├── test.en / test.ur
│       └── umc005-corpus/          # Additional corpora
│
├── Task3/                          # Diffusion Transformers
│   ├── Task3_Diffusion_Transformers.ipynb  # Complete notebook
│   ├── requirements.txt
│   ├── checkpoints/                # SiT model weights
│   ├── data/                       # CIFAR-10 dataset
│   ├── runs/                       # TensorBoard logs
│   └── samples/                    # Generated images
│
├── runs/                           # TensorBoard logs (CycleGAN)
├── samples/                        # Sample outputs
└── README.md

---

🎨 Task 1: CycleGAN Face-Sketch Translation

Architecture

Implements the CycleGAN architecture from "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks" (Zhu et al., 2017).

Generator (ResNet-based):

• Initial convolution: c7s1-64
• Downsampling: d128, d256
• 9 Residual blocks: R256×9
• Upsampling: u128, u64
• Output: c7s1-3 with Tanh activation

Discriminator (PatchGAN):

• 70×70 PatchGAN discriminator
• Instance normalization

Features

• ✅ Cycle consistency loss
• ✅ Adversarial loss (LSGAN)
• ✅ Identity mapping loss
• ✅ Learning rate scheduling
• ✅ Interactive GUI demo

Quick Start

cd Task1
pip install -r requirements.txt

# Train
python train.py

# Test
python test.py

# Launch GUI
python gui.py

---

🌐 Task 2: English-Urdu Machine Translation

Architecture

Implements the vanilla Transformer from "Attention is All You Need" (Vaswani et al., 2017).

Model Configuration:

• Encoder: 6 layers, 8 attention heads
• Decoder: 6 layers, 8 attention heads
• d_model: 512
• d_ff: 2048 (feedforward dimension)
• Dropout: 0.1

Tokenization:

• BPE (Byte Pair Encoding) tokenizers for both languages
• Custom-trained on English-Urdu parallel corpus

Features

• ✅ Sinusoidal positional encoding
• ✅ Multi-head self-attention
• ✅ Teacher forcing during training
• ✅ BLEU score evaluation
• ✅ mBART fine-tuning option
• ✅ Interactive GUI demo

Quick Start

cd Task2
pip install -r requirements.txt

# Train tokenizers
python train_tokenizers.py

# Train model
python train.py

# Evaluate (BLEU score)
python evaluate.py

# Launch GUI
python gui.py

---

🖼️ Task 3: Diffusion Transformers (SiT)

Architecture

Implements SiT (Scalable Interpolant Transformer) with REG (Representation Entanglement for Generation) based on recent diffusion transformer research.

Key Components:

• Patch embedding for image tokenization
• Transformer blocks with self-attention
• Adaptive layer normalization (adaLN)
• Continuous-time diffusion with DDPM/DDIM sampling
• DINOv2 integration for representation entanglement

Features

• ✅ CIFAR-10 subset training (cats & dogs)
• ✅ REG loss for improved generation
• ✅ Classifier-free guidance support
• ✅ DDPM & DDIM sampling
• ✅ SafeTensors model saving
• ✅ TensorBoard visualization

Quick Start

cd Task3
pip install -r requirements.txt

# Open and run the notebook
jupyter notebook Task3_Diffusion_Transformers.ipynb

---

🛠️ Installation

Prerequisites

• Python 3.8+
• CUDA-capable GPU (recommended)
• PyTorch 2.5+ (for CVE-2025-32434 mitigation)

Setup

# Clone the repository
git clone https://github.com/thewitcher41/GenAI-Assignment2.git
cd GenAI-Assignment2

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate     # Windows

# Install dependencies for each task
pip install -r Task1/requirements.txt
pip install -r Task2/requirements.txt
pip install -r Task3/requirements.txt

---

🚀 Usage

Training Models

# Task 1: CycleGAN
cd Task1 && python train.py

# Task 2: Transformer Translation
cd Task2 && python train.py

# Task 3: Diffusion Transformer
# Open and run Task3_Diffusion_Transformers.ipynb

GUI Applications

Both Task 1 and Task 2 include interactive PyQt5 GUIs:

# Face-Sketch GUI
python Task1/gui.py

# Translation GUI
python Task2/gui.py

TensorBoard

Monitor training progress:

tensorboard --logdir=runs

---

📊 Results

Task 1: Face-Sketch Translation

Metric	Value
Cycle Consistency Loss	Low
Visual Quality	High fidelity sketch generation

Task 2: English-Urdu Translation

Metric	Value
BLEU Score	See predictions.txt
Tokenizer	BPE with 32K vocab

Task 3: Diffusion Transformer

Metric	Value
Dataset	CIFAR-10 (Cats & Dogs)
Sampling	DDPM & DDIM

---

📚 References

1. CycleGAN: Zhu, J.Y., et al. "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks." ICCV 2017.

2. Transformer: Vaswani, A., et al. "Attention is All You Need." NeurIPS 2017.

3. Diffusion Models: Ho, J., et al. "Denoising Diffusion Probabilistic Models." NeurIPS 2020.

4. DiT: Peebles, W., & Xie, S. "Scalable Diffusion Models with Transformers." ICCV 2023.

5. REG: "Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think."

---

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• FAST-NUCES for the course structure
• PyTorch team for the deep learning framework
• Hugging Face for Transformers library
• Original paper authors for their groundbreaking research

---

Made with ❤️ for Generative AI Course