Diffusion Models: From MNIST to Pretrained Models

A comprehensive exploration of diffusion models, from training custom models on MNIST to leveraging pretrained Stable Diffusion for advanced image generation tasks.

📁 Project Structure

mnist_diffusion.ipynb

Training and evaluation of custom diffusion models on MNIST digit dataset.

Contents:

• Task 1: Unconditional DDPM

  • Implementation of Denoising Diffusion Probabilistic Models (DDPM)
  • Forward diffusion process with noise scheduling (linear and cosine)
  • Reverse diffusion sampling for image generation
  • Analysis of key insights: Markov property, Gaussian distributions, beta schedules, timesteps

• Task 2: Class-Conditional Diffusion

  • Conditional U-Net architecture with class embeddings
  • Time and class embedding integration
  • Training loop with class labels
  • Digit-specific generation (0-9) with controlled sampling

• Task 3: Text-Guided MNIST Generation

  • Integration of CLIP model for text embeddings
  • CLIP-guided loss implementation
  • Text-guided diffusion sampling with gradient-based guidance
  • Generation of digit images from text prompts ("two", "five", "nine")

Key Concepts:

• Diffusion schedule impact on training quality
• Classifier-Free Guidance principles
• Multi-modal learning with CLIP and diffusion
• Progressive denoising in reverse process

---

stable_diffusion_guide.ipynb

Practical applications using pretrained Stable Diffusion models for guided generation.

Contents:

• Task 1: Classifier-Free Guidance (CFG)

  • Pretrained Stable Diffusion v1.5 loading and setup
  • CFG sampling with configurable guidance scales
  • Effect analysis of guidance strength on generation quality
  • Testing across multiple prompts and scales [1.0, 5.0, 7.5, 12.0]

• Task 2: Image Inpainting

  • Stable Diffusion Inpainting Pipeline setup
  • Image and mask preparation (512x512 resolution)
  • Text-guided region filling
  • Multiple prompt testing for diverse inpainting results

• Task 3: Style Transfer with ControlNet + IP-Adapter

  • ControlNet edge detection for structural guidance
  • IP-Adapter for style influence
  • Multi-image style transfer experiments
  • Edge map generation and style blending
  • Content, structure, and style control through prompts

Key Techniques:

• Classifier-Free Guidance for prompt adherence control
• ControlNet for structural preservation
• IP-Adapter for visual style transfer
• Prompt engineering for content and style direction

---

🚀 Getting Started

Requirements

Install required dependencies:

pip install torch torchvision
pip install diffusers transformers accelerate
pip install pillow matplotlib numpy tqdm
pip install clip controlnet-aux safetensors xformers

Running the Notebooks

1. MNIST Diffusion Models (mnist_diffusion.ipynb)

   • Self-contained training and sampling
   • GPU recommended for training
   • Dataset automatically downloaded from torchvision

2. Stable Diffusion Guide (stable_diffusion_guide.ipynb)

   • Uses pretrained models from HuggingFace
   • Requires GPU for inference
   • Models automatically downloaded on first use

---

📊 Key Findings

From MNIST Diffusion Models:

• Guidance Scale Impact: Higher guidance scales improve prompt adherence but reduce diversity
• Prompt Sensitivity: Model responds better to detailed, specific prompts
• Training Efficiency: Cosine noise schedules provide better learning curves than linear schedules
• Class Conditioning: Successfully enables controlled digit generation across all 10 classes

From Stable Diffusion Applications:

• CFG Effectiveness: Guidance scale of 7.5-10 provides optimal balance between quality and prompt fidelity
• Inpainting Quality: Highly dependent on mask clarity and prompt specificity
• Style Transfer: ControlNet + IP-Adapter combination effectively preserves content structure while applying visual style
• Prompt Engineering: Rich, descriptive prompts yield significantly better results than simple single-word prompts

---

🛠️ Architecture Details

Custom DDPM (MNIST)

• Model: U-Net with time embeddings
• Training: 30 epochs with learning rate warmup
• Loss: Mean Squared Error on predicted noise
• Inference: 300-step reverse process with adaptive noise scheduling

Conditional Diffusion (MNIST)

• Class Embedding: 10-dimensional embeddings for digit classes
• Architecture: TimeEmbedCond with combined time and class vectors
• Training: 20 epochs with class labels
• Generation: Class-specific sampling without classifier

Stable Diffusion Pipeline

• Model: runwayml/stable-diffusion-v1-5
• Precision: float16 for memory efficiency
• Features: Attention slicing, model CPU offloading
• Variants: Inpainting (stabilityai/stable-diffusion-2-inpainting), ControlNet (lllyasviel/sd-controlnet-canny)

---

📈 Performance Metrics

MNIST DDPM:

• Training Loss convergence: Epoch 1 → ~150, Epoch 30 → ~10
• Generated images: Recognizable digits with clear class distinction
• Inference time: ~60 seconds for 300-step generation

Stable Diffusion:

• Generation time: ~30-50 seconds per image (batch_size=1)
• Memory usage: ~8GB VRAM (float16)
• Quality improves with guidance scale and prompt specificity

---

🎨 Sample Outputs

MNIST Diffusion

• Unconditional: Random digit generation with realistic styling
• Class-conditional: Precise digit generation (0-9) matching target class
• Text-guided: Digit images generated from text prompts with CLIP alignment

Stable Diffusion

• High-quality photorealistic and artistic image generation
• Controlled composition and content through guidance scales
• Seamless inpainting with context-aware content
• Style transfer preserving both structure and artistic direction

---

📚 References

• Diffusion Models: Ho et al., "Denoising Diffusion Probabilistic Models" (2020)
• Classifier-Free Guidance: Ho & Salimans, "Classifier-Free Diffusion Guidance" (2021)
• ControlNet: Zhang et al., "Adding Conditional Control to Text-to-Image Diffusion Models" (2023)
• CLIP: Radford et al., "Learning Transferable Models for Unsupervised Learning" (2021)
• Stable Diffusion: Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models" (2022)

---

🔗 Model Sources

• Stable Diffusion v1.5
• Stable Diffusion Inpainting
• ControlNet Canny
• IP-Adapter
• OpenAI CLIP

---

💡 Future Directions

• Fine-tune models on custom datasets
• Implement negative prompt guidance
• Explore other ControlNet variants (depth, pose)
• Multi-model ensemble for improved quality
• Optimization techniques for faster inference

---

Created for: Exploring Diffusion Models in Deep Learning Project Focus: Understanding and applying diffusion models from first principles to practical applications