This repository does not treat VAE, DDPM, DiT, and Flow Matching as isolated one-off implementations. The main line puts them under one small Stable Diffusion-style framework:
image -> VAE latent -> latent generator -> VAE decoder -> image
In other words, the main generative methods in this repo operate in the latent space produced by a shared VAE. The main difference is which method is used for the middle latent generator.
Chinese documentation is kept in README_CN.md.
VAEcompresses128x128images into[4, 16, 16]latents.DDPM,DiT,Flow Matching, and related methods can all consume the same VAE latent format.Stable Diffusionin this repo means an engineering pattern:VAE latent + generative model + decoder.- The latent generator can have different implementations: latent DDPM, latent DiT, latent Flow Matching, latent Schrödinger Bridge, and so on.
- When a mature library exists, use it. Prioritize
diffusers,accelerate, and native PyTorch components over hand-written core models.
flowchart TD
A[Raw anime portrait image<br/>3 x 128 x 128] --> B[VAE Encoder<br/>Diffusers AutoencoderKL]
B --> C[Shared latent space<br/>4 x 16 x 16]
C --> D1[latent DDPM<br/>U-Net + scheduler]
C --> D2[latent DiT<br/>Transformer diffusion]
C --> D3[latent Flow Matching<br/>ODE velocity field]
C --> D4[latent Schrödinger Bridge<br/>stochastic bridge]
C --> D5[latent interpolation<br/>VAE morph GIF]
D1 --> E[Generated latent<br/>4 x 16 x 16]
D2 --> E
D3 --> E
D4 --> E
D5 --> E
E --> F[VAE Decoder<br/>Diffusers AutoencoderKL]
F --> G[Generated image<br/>3 x 128 x 128]
For a fuller discussion of method relationships, equations, and comparisons, see:
THEORY.md
dataset/
Dataset notes and local data conventions. The current default data directory is:
dataset/raw/fullMin256
vae/
The shared latent interface. The current implementation uses Diffusers AutoencoderKL, with default pretrained weights:
stabilityai/sd-vae-ft-mse
Core shape:
[B, 3, 128, 128] -> [B, 4, 16, 16]
ddpm/
The latent DDPM route. It does not diffuse directly in pixel space; it trains a denoising model in VAE latent space.
dit/
The latent DiT route. Compared with DDPM, the main difference is swapping the U-Net backbone for a Transformer backbone.
flow_matching/
The latent Flow Matching route. It learns a continuous-time ODE velocity field in the same shared VAE latent space.
vis-flow-matching/
A standalone MNIST Flow Matching visualization project. It does not depend on the VAE latent pipeline. Instead, it demonstrates:
Gaussian noise -> ODE flow -> 0..9 handwritten digit GIF
This is useful for understanding Flow Matching formulas, velocity-field training, and sampling trajectories directly. Run instructions are in:
vis-flow-matching/README.md
vis-2d-flowmatching/
A standalone 2D Flow Matching visualization project. It does not use VAE latents. Instead, it shows how points move along a learned velocity field toward a target distribution:
Gaussian noise -> ODE flow -> ring / moons
The project generates trajectory GIFs and vector-field plots. Run instructions are in:
vis-2d-flowmatching/README.md
Visualization preview:
MNIST conditional Flow Matching sampling animation: each column corresponds to digit 0..9, each row is a different random sample, and the animation shows Gaussian noise gradually flowing into handwritten digits.
MNIST training curve: the x-axis is epoch and the y-axis is loss, used to inspect whether training and validation errors decrease steadily.
Ring sampling animation: 2D points start from Gaussian noise and move along the learned velocity field toward a ring distribution.
Ring vector field: shows the direction of the 2D velocity field at t=0.0, t=0.5, and t=1.0.
Ring training curve: used to judge whether the 2D ring experiment is converging.
Moons sampling animation: 2D points gradually flow from Gaussian noise to the two-moons target distribution.
Moons vector field: shows the velocity-field structure of the two-moons experiment at different time slices.
Moons training curve: used to inspect the loss trend and convergence stability of the two-moons experiment.
gan/
Baseline generative-model directory. GAN can serve as a comparison baseline and can also support later latent-GAN experiments, but it is not the current main-line priority.
All main-line methods share one VAE latent interface:
latent shape = [batch, 4, 16, 16]
Why this is useful:
- each method avoids inventing a different input / output format, which keeps comparison cleaner
- DDPM, DiT, Flow Matching, and related methods can be compared on generation mechanics rather than representation mismatch
- the same decoder can be reused so all sampled results map back into the same image space
- this is much closer to an actual Stable Diffusion-style engineering layout
There are two VAE routes. Pick one.
Route A: download a pretrained VAE and fine-tune it on DAF.
python vae/download_pretrained.py \
--model-id stabilityai/sd-vae-ft-mse \
--output-dir vae/pretrained/sd-vae-ft-msepython vae/train_vae.py \
--data-dir dataset/raw/fullMin256 \
--pretrained-vae vae/pretrained/sd-vae-ft-mse \
--output-dir vae/runs/vae_daf_finetune \
--image-size 128 \
--batch-size 64 \
--epochs 50 \
--patience 8 \
--min-delta 1e-4 \
--checkpoint-every-epochs 5 \
--max-epoch-checkpoints 10If full-data fine-tuning is too slow, first validate the training workflow on a fixed-random 10% subset of valid images:
python vae/train_vae.py \
--data-dir dataset/raw/fullMin256 \
--pretrained-vae vae/pretrained/sd-vae-ft-mse \
--output-dir vae/runs/vae_daf_finetune_10pct \
--image-size 128 \
--batch-size 64 \
--epochs 50 \
--dataset-fraction 0.1 \
--seed 42 \
--patience 3 \
--min-delta 1e-4 \
--checkpoint-every-epochs 5 \
--max-epoch-checkpoints 10Route B: start from a random-initialized Diffusers AutoencoderKL and train your own VAE.
python vae/train_vae.py \
--data-dir dataset/raw/fullMin256 \
--init-from-scratch \
--output-dir vae/runs/vae_daf_from_scratch \
--image-size 128 \
--batch-size 64 \
--epochs 50 \
--patience 8 \
--min-delta 1e-4 \
--checkpoint-every-epochs 5 \
--max-epoch-checkpoints 10Route B can also be validated first on a fixed-random 10% subset of valid images:
python vae/train_vae.py \
--data-dir dataset/raw/fullMin256 \
--init-from-scratch \
--output-dir vae/runs/vae_daf_from_scratch_10pct \
--image-size 128 \
--batch-size 64 \
--epochs 50 \
--dataset-fraction 0.1 \
--seed 42 \
--patience 3 \
--min-delta 1e-4 \
--checkpoint-every-epochs 5 \
--max-epoch-checkpoints 10The two routes use different output directories so they do not overwrite one another:
Route A: vae/runs/vae_daf_finetune/best_diffusers
Route B: vae/runs/vae_daf_from_scratch/best_diffusers
Both routes use the same data-processing and validation logic:
- scan
dataset/raw/fullMin256first, filter bad images, and get the valid-image list - if
--dataset-fraction 0.1 --seed 42is passed, sample a fixed random10%subset from valid images - split into train / validation according to
--val-ratio trainupdates VAE weights andvalselectsbest_diffusers/- the only difference between Route A and Route B is initialization: Route A starts from a pretrained VAE and Route B starts from random weights
- full-data commands explicitly use early stopping
--patience 8 --min-delta 1e-4, while the 10% quick-validation commands explicitly use--patience 3 --min-delta 1e-4
VAE training and latent caching both scan images at startup:
- default
--scan-workers 8with atqdmprogress bar - scanning generates
dataset/valid_images.txtanddataset/bad_images.txt - each run rescans so dataset changes or interrupted runs do not leave stale lists
- if the scan result is unchanged, the list files are not rewritten, reducing unnecessary SSD writes
--dataset-fraction 0.1 --seed 42samples a fixed random10%subset after bad-image scanning completes; the default1.0means full-data training
By default the process only guarantees saving best_diffusers/, rather than keeping unlimited checkpoints. More details are in:
vae/dataset_treatment.md
Once the best VAE is chosen, encode the full image set into a single HDF5 file.
Route A:
python vae/cache_latents.py \
--data-dir dataset/raw/fullMin256 \
--vae-dir vae/runs/vae_daf_finetune/best_diffusers \
--output dataset/latents/vae_daf_128_finetune.h5Route A 10% quick VAE:
python vae/cache_latents.py \
--data-dir dataset/raw/fullMin256 \
--vae-dir vae/runs/vae_daf_finetune_10pct/best_diffusers \
--output dataset/latents/vae_daf_128_finetune_10pct.h5Route B:
python vae/cache_latents.py \
--data-dir dataset/raw/fullMin256 \
--vae-dir vae/runs/vae_daf_from_scratch/best_diffusers \
--output dataset/latents/vae_daf_128_from_scratch.h5Route B 10% quick VAE:
python vae/cache_latents.py \
--data-dir dataset/raw/fullMin256 \
--vae-dir vae/runs/vae_daf_from_scratch_10pct/best_diffusers \
--output dataset/latents/vae_daf_128_from_scratch_10pct.h5The outputs are:
Route A: dataset/latents/vae_daf_128_finetune.h5
Route A 10% quick VAE: dataset/latents/vae_daf_128_finetune_10pct.h5
Route B: dataset/latents/vae_daf_128_from_scratch.h5
Route B 10% quick VAE: dataset/latents/vae_daf_128_from_scratch_10pct.h5
The HDF5 stores float16 latents that have already been multiplied by vae.config.scaling_factor, with default gzip compression level 1. Here 10% means the VAE itself was trained on 10% of the valid images; cache_latents.py still encodes all currently valid images from the data directory into HDF5 by default.
Pick one or more methods to train. The concrete commands are listed in the next section, Training And Sampling By Method.
Different VAE routes read different latent caches:
Route A: dataset/latents/vae_daf_128_finetune.h5
Route A 10% quick VAE: dataset/latents/vae_daf_128_finetune_10pct.h5
Route B: dataset/latents/vae_daf_128_from_scratch.h5
Route B 10% quick VAE: dataset/latents/vae_daf_128_from_scratch_10pct.h5
After training finishes, run the matching sample command to generate images. The exact commands are also in the next section.
Notes:
- later-stage methods do not reread the raw images or rerun the VAE encoder by default
- later-stage methods all read from HDF5 latent caches by default
runs/outputs for the methods are ignored by.gitignore- training appends
metrics.csvand overwritesmetrics.jpgin each method'soutput-dir, using JPG format anddpi=200 - DDPM, DiT, and Flow Matching use
val_lossfor early stopping; GAN usesval_fake_score. Full-data routes use--patience 8 --min-delta 1e-4, while 10% quick-VAE routes use--patience 3 --min-delta 1e-4
Route A:
# Train DDPM on Route A latent cache.
python ddpm/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune.h5 \
--output-dir ddpm/runs/latent_ddpm_finetune \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route A VAE decoder.
python ddpm/sample.py \
--model-dir ddpm/runs/latent_ddpm_finetune/best_model \
--vae-dir vae/runs/vae_daf_finetune/best_diffusers \
--num-images 128 \
--output-dir ddpm/runs/latent_ddpm_finetune/samples_jpg \
--image-format jpgRoute A 10% quick VAE:
# Train DDPM on the 10% quick-VAE latent cache.
python ddpm/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune_10pct.h5 \
--output-dir ddpm/runs/latent_ddpm_finetune_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the 10% quick-VAE decoder.
python ddpm/sample.py \
--model-dir ddpm/runs/latent_ddpm_finetune_10pct/best_model \
--vae-dir vae/runs/vae_daf_finetune_10pct/best_diffusers \
--num-images 128 \
--output-dir ddpm/runs/latent_ddpm_finetune_10pct/samples_jpg \
--image-format jpgRoute B:
# Train DDPM on Route B latent cache.
python ddpm/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch.h5 \
--output-dir ddpm/runs/latent_ddpm_from_scratch \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B VAE decoder.
python ddpm/sample.py \
--model-dir ddpm/runs/latent_ddpm_from_scratch/best_model \
--vae-dir vae/runs/vae_daf_from_scratch/best_diffusers \
--num-images 128 \
--output-dir ddpm/runs/latent_ddpm_from_scratch/samples_jpg \
--image-format jpgRoute B 10% quick VAE:
# Train DDPM on the Route B 10% quick-VAE latent cache.
python ddpm/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch_10pct.h5 \
--output-dir ddpm/runs/latent_ddpm_from_scratch_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B 10% quick-VAE decoder.
python ddpm/sample.py \
--model-dir ddpm/runs/latent_ddpm_from_scratch_10pct/best_model \
--vae-dir vae/runs/vae_daf_from_scratch_10pct/best_diffusers \
--num-images 128 \
--output-dir ddpm/runs/latent_ddpm_from_scratch_10pct/samples_jpg \
--image-format jpgRoute A:
# Train DiT on Route A latent cache.
python dit/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune.h5 \
--output-dir dit/runs/latent_dit_finetune \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route A VAE decoder.
python dit/sample.py \
--model-dir dit/runs/latent_dit_finetune/best_model \
--vae-dir vae/runs/vae_daf_finetune/best_diffusers \
--num-images 128 \
--output-dir dit/runs/latent_dit_finetune/samples_jpg \
--image-format jpgRoute A 10% quick VAE:
# Train DiT on the 10% quick-VAE latent cache.
python dit/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune_10pct.h5 \
--output-dir dit/runs/latent_dit_finetune_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the 10% quick-VAE decoder.
python dit/sample.py \
--model-dir dit/runs/latent_dit_finetune_10pct/best_model \
--vae-dir vae/runs/vae_daf_finetune_10pct/best_diffusers \
--num-images 128 \
--output-dir dit/runs/latent_dit_finetune_10pct/samples_jpg \
--image-format jpgRoute B:
# Train DiT on Route B latent cache.
python dit/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch.h5 \
--output-dir dit/runs/latent_dit_from_scratch \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B VAE decoder.
python dit/sample.py \
--model-dir dit/runs/latent_dit_from_scratch/best_model \
--vae-dir vae/runs/vae_daf_from_scratch/best_diffusers \
--num-images 128 \
--output-dir dit/runs/latent_dit_from_scratch/samples_jpg \
--image-format jpgRoute B 10% quick VAE:
# Train DiT on the Route B 10% quick-VAE latent cache.
python dit/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch_10pct.h5 \
--output-dir dit/runs/latent_dit_from_scratch_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B 10% quick-VAE decoder.
python dit/sample.py \
--model-dir dit/runs/latent_dit_from_scratch_10pct/best_model \
--vae-dir vae/runs/vae_daf_from_scratch_10pct/best_diffusers \
--num-images 128 \
--output-dir dit/runs/latent_dit_from_scratch_10pct/samples_jpg \
--image-format jpgRoute A:
# Train Flow Matching on Route A latent cache.
python flow_matching/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune.h5 \
--output-dir flow_matching/runs/latent_fm_finetune \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route A VAE decoder.
python flow_matching/sample.py \
--model-dir flow_matching/runs/latent_fm_finetune/best_model \
--vae-dir vae/runs/vae_daf_finetune/best_diffusers \
--num-images 128 \
--output-dir flow_matching/runs/latent_fm_finetune/samples_jpg \
--image-format jpgRoute A 10% quick VAE:
# Train Flow Matching on the 10% quick-VAE latent cache.
python flow_matching/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune_10pct.h5 \
--output-dir flow_matching/runs/latent_fm_finetune_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the 10% quick-VAE decoder.
python flow_matching/sample.py \
--model-dir flow_matching/runs/latent_fm_finetune_10pct/best_model \
--vae-dir vae/runs/vae_daf_finetune_10pct/best_diffusers \
--num-images 128 \
--output-dir flow_matching/runs/latent_fm_finetune_10pct/samples_jpg \
--image-format jpgRoute B:
# Train Flow Matching on Route B latent cache.
python flow_matching/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch.h5 \
--output-dir flow_matching/runs/latent_fm_from_scratch \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B VAE decoder.
python flow_matching/sample.py \
--model-dir flow_matching/runs/latent_fm_from_scratch/best_model \
--vae-dir vae/runs/vae_daf_from_scratch/best_diffusers \
--num-images 128 \
--output-dir flow_matching/runs/latent_fm_from_scratch/samples_jpg \
--image-format jpgRoute B 10% quick VAE:
# Train Flow Matching on the Route B 10% quick-VAE latent cache.
python flow_matching/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch_10pct.h5 \
--output-dir flow_matching/runs/latent_fm_from_scratch_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B 10% quick-VAE decoder.
python flow_matching/sample.py \
--model-dir flow_matching/runs/latent_fm_from_scratch_10pct/best_model \
--vae-dir vae/runs/vae_daf_from_scratch_10pct/best_diffusers \
--num-images 128 \
--output-dir flow_matching/runs/latent_fm_from_scratch_10pct/samples_jpg \
--image-format jpgRoute A:
# Train latent GAN on Route A latent cache.
python gan/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune.h5 \
--output-dir gan/runs/latent_gan_finetune \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route A VAE decoder.
python gan/sample.py \
--checkpoint gan/runs/latent_gan_finetune/best.pt \
--vae-dir vae/runs/vae_daf_finetune/best_diffusers \
--num-images 128 \
--output-dir gan/runs/latent_gan_finetune/samples_jpg \
--image-format jpgRoute A 10% quick VAE:
# Train latent GAN on the 10% quick-VAE latent cache.
python gan/train.py \
--latents-h5 dataset/latents/vae_daf_128_finetune_10pct.h5 \
--output-dir gan/runs/latent_gan_finetune_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the 10% quick-VAE decoder.
python gan/sample.py \
--checkpoint gan/runs/latent_gan_finetune_10pct/best.pt \
--vae-dir vae/runs/vae_daf_finetune_10pct/best_diffusers \
--num-images 128 \
--output-dir gan/runs/latent_gan_finetune_10pct/samples_jpg \
--image-format jpgRoute B:
# Train latent GAN on Route B latent cache.
python gan/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch.h5 \
--output-dir gan/runs/latent_gan_from_scratch \
--patience 8 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B VAE decoder.
python gan/sample.py \
--checkpoint gan/runs/latent_gan_from_scratch/best.pt \
--vae-dir vae/runs/vae_daf_from_scratch/best_diffusers \
--num-images 128 \
--output-dir gan/runs/latent_gan_from_scratch/samples_jpg \
--image-format jpgRoute B 10% quick VAE:
# Train latent GAN on the Route B 10% quick-VAE latent cache.
python gan/train.py \
--latents-h5 dataset/latents/vae_daf_128_from_scratch_10pct.h5 \
--output-dir gan/runs/latent_gan_from_scratch_10pct \
--patience 3 \
--min-delta 1e-4
# Generate 128 jpg images with the Route B 10% quick-VAE decoder.
python gan/sample.py \
--checkpoint gan/runs/latent_gan_from_scratch_10pct/best.pt \
--vae-dir vae/runs/vae_daf_from_scratch_10pct/best_diffusers \
--num-images 128 \
--output-dir gan/runs/latent_gan_from_scratch_10pct/samples_jpg \
--image-format jpgTest scripts live in:
test/
Run:
python -m unittest discover -s test -p "test_*.py"Or:
bash test/run_dry_tests.shThese tests only cover syntax, model forward-pass shape checks, small HDF5 read / write checks, and dry-run level validation. They do not run real training or write large checkpoints.
The VAE itself can create a smooth transition between two images without DDPM or diffusion:
image A -> encoder -> latent A
image B -> encoder -> latent B
interpolate between latent A and latent B
interpolated latent -> decoder -> GIF
The corresponding script:
python vae/interpolate_gif.py \
--image-a dataset/raw/fullMin256/0987/1223987.jpg \
--image-b dataset/raw/fullMin256/0987/3174987.jpg \
--output vae/runs/interpolation_pretrained.gifIf --vae is not provided, the default model is stabilityai/sd-vae-ft-mse. To use your own fine-tuned VAE:
python vae/interpolate_gif.py \
--image-a dataset/raw/fullMin256/0987/1223987.jpg \
--image-b dataset/raw/fullMin256/0987/3174987.jpg \
--vae vae/runs/vae_daf_finetune/best_diffusers \
--output vae/runs/interpolation_finetune.gif- do not prioritize hand-written VAE, U-Net, scheduler, or other core components
- prefer mature implementations such as
diffusers.AutoencoderKL - assume Apple Silicon
MPSby default when available - project code mainly handles paths, data, training entry points, logging, checkpoints, and method-specific sampling scripts
- if a module must be custom-built, the reason and risk should be stated first








