Skip to content

FastGen v0.2.0

Latest

Choose a tag to compare

@juliusberner juliusberner released this 28 Oct 17:04
· 5 commits to main since this release
deb93e6

FastGen v0.2.0

FastGen v0.2.0 is the first major update since v0.1.0 and includes every pull request merged since then (#10, #25–#29, #33, #37 and #38). Some import paths, config fields and inference flags changed; see "Upgrading from v0.1.0" below.

Highlights

  • Methods: AYF, PDD, AnyFlow, Phased DMD, Causal-Forcing and backward simulation.
  • Networks: FLUX.2, Wan 2.2 A14B, Cosmos 3, LingBot-world, and FastGen's first video+audio models: LTX-2, LTX-2.3, LTX-2.5 and MiniMax H3.
  • Training core: LoRA for any network, one guidance API for sampling and training, sharded EMA under FSDP2, and a separate precision for method and sampling state.
  • Tooling: ComfyUI custom nodes, a VLM judge, FlashDreams streaming inference, and 117 experiment configs, up from 73.

New methods

Trajectory matching now groups the consistency-model and knowledge-distillation methods (CM, ECT, sCM, TCM, MeanFlow, KD, CausalKD) and adds:

  • AYF (AYFModel): continuous-time flow map distillation.
  • PDD (PDDModel): parallel decoding distillation for image and video generation.

Distribution matching adds:

  • AnyFlow (AnyFlowModel, #25): any-step video generation with on-policy flow map distillation.
  • Phased DMD: few-step distribution matching with score matching within timestep subintervals.
  • Causal-Forcing: autoregressive diffusion distillation for real-time interactive video.
  • Backward simulation: the student trains on its own sampling trajectories instead of noised data.

Fine-tuning adds teacher forcing for causal models (CausalTeacherForcingSFTModel). Causal video methods share autoregressive student sampling with KV caching.

New networks

Data New in v0.2.0
Image FLUX.2 [dev] and FLUX.2 [klein] base 4B and 9B, each for text-to-image and image editing
Video Wan 2.2 A14B (mixture of experts) for text- and image-to-video, Cosmos 3, and LingBot-world for camera-controlled generation
Video+audio LTX-2, LTX-2.3 and LTX-2.5, with image-to-video recipes for LTX-2 and LTX-2.3; MiniMax H3

fastgen/networks/README.md lists the supported checkpoints.

Training core

  • Module preparation. Modules are prepared once, in the same order on every rank (precision, train mode, gradients, then device placement or FSDP2 sharding), including tied parameters and shared submodules.
  • LoRA. model.net_adapter adds PEFT adapters and extra trainable modules to any network. The teacher reuses the student's frozen base weights instead of keeping a second copy. Adapters train with DDP; FSDP is not supported yet.
  • Guidance. model.guidance_cfg combines classifier-free, mix and skip-layer guidance, each schedulable over time, with optional rescaling. Sampling and teacher passes during training share it.
  • Sampling grids. model.student_grid_cfg sets the student's sampling grid (type, steps and options), replacing student_sample_steps and sample_t_cfg.t_list.
  • Precision. precision now covers only data and method/sampling arithmetic, so it can differ from the model weights (precision_model). Optimizer state, master weights and EMA use precision_opt, which replaces precision_fsdp.
  • FSDP2. EMA copies are sharded like the live network instead of being replicated on every GPU, and validation samples come from the EMA with FSDP too. model.extra_fsdp_modules shards frozen modules such as the teacher or text encoder, and meta-device initialization (model.meta_init) also works without FSDP.
  • torch.compile. model.torch_compile_mode compiles the training networks (#28).
  • Environment. A .env file (see .env.example) sets the output, data, checkpoint and cache roots. Credentials for W&B, Hugging Face, S3 and the VLM endpoint live in one credentials_dir.

Tooling

  • ComfyUI: custom nodes for teacher and student inference, with an example Qwen-Image and LTX-2.3 workflow (comfyui/workflows/QwenImage-LTX2.3.json).
  • VLM judge: the VLMLogger callback and scripts/inference/vlm_infer.py score generated images and videos with any OpenAI-compatible endpoint.
  • Inference: video with audio, image editing, camera-controlled generation from pose files, and FlashDreams streaming inference for Self-Forcing Wan 2.1 T2V (scripts/inference/flashdreams_inference.py).
  • Data: WebDataset loaders for video+audio and for camera-conditioned video. The VipeLogger callback plots camera trajectories and, with the optional ViPE package, reports the pose error of generated clips.
  • Checkpoints and data: FSDP checkpoint conversion (scripts/fsdp_convert.py) and resumable S3 downloads (scripts/resumable_s3_download.py).
  • CI: lint, type checks, tests and package installation run on pull requests (#29).

Bug fixes

Training and checkpoints

  • Run names default to the config file name, so recipes in the same group, such as the EDM CIFAR-10 ones, no longer share a debug directory and resume each other's checkpoints.
  • Resume handles interrupted saves and S3, keeps the FSDP2 optimizer state when some parameters don't match instead of resetting it (#27), and raises an error when a requested checkpoint file is missing.
  • trainer.max_iter=N runs N iterations, not N−1.
  • EMA: bf16 recipes keep their EMA in fp32 so it no longer stalls (a low-precision EMA now warns), and the Wan CausVid recipe updates its EMA.

Methods

  • sCM: correct JVP dropout masks, finite-difference JVP, VP timesteps and loss normalization.
  • DMD2 and LADD: R1 perturbs the noised input, not the clean data.
  • KD: CausalKD no longer crashes, and single-step KD scales its noise correctly.
  • TCM and f-distill: no NaN losses in TCM, and f-distill validation keeps its training statistics.
  • VSD loss uses higher precision in bf16 recipes.
  • Wan MeanFlow: the r pathway trains (#42).

Networks

  • FLUX.1: correct time shift, and no double guidance in FLUX.1-dev.
  • SD1.5 and CogVideoX: an empty caption no longer blanks the whole batch.
  • SDXL micro-conditioning uses the actual resolution.
  • DiT-XL/2 SFT uses DiT's DDPM schedule.
  • Wan: local checkpoints load their own components, I2V uses per-checkpoint defaults, and teachers honor the step count.
  • VACE: the depth annotator no longer runs with random weights, and causal sampling keeps the caller's noise.
  • CogVideoX-5B schedule and RoPE no longer depend on the model path.
  • Cosmos Predict2: video2world timesteps (#26), Predict2.5 SFT inference (#33), reset_parameters, feature extraction and padded-prompt attention masks (#37), and skip-layer guidance and video2world sampling (#38).

Data loading and scripts

  • WebDataset: deterministic training resumes at the right sample, and shard_end and shards_per_worker work as documented.
  • Loaders fail fast on unreadable ignore-index files and all-failing streams, and skip malformed scores.
  • Inference and FID: latent recipes (enable_preprocessors=False) work, SFT checkpoints use their CFG, and a wrong --ckpt_path raises an error.

Smaller fixes

Crashes in DMD2 and Self-Forcing under FSDP2 with precision_fsdp="float32", LambdaLinearScheduler, return_logvar (SD1.5, SDXL, CogVideoX, Cosmos Predict2) and EDM2 with MeanFlow's r; the Adam fallback for an unknown optim_type; EDM2 FID reference statistics; GIF output from --save_high_quality; and download_data.py partial outputs and ImageNet-64 --only-models.

Upgrading from v0.1.0

Imports

  • fastgen.methods.consistency_model and fastgen.methods.knowledge_distillation are now fastgen.methods.trajectory_matching. Imports from fastgen.methods, such as from fastgen.methods import CMModel, still work.
  • fastgen.utils no longer re-exports helpers:
    • LazyCall, instantiate and locate are in fastgen.configs.instantiate, which replaces fastgen.utils.registry. Custom configs should use from fastgen.configs.instantiate import LazyCall as L.
    • expand_like is in fastgen.utils.tensor.
    • set_global_vars and set_temp_global_vars are in fastgen.utils.global_vars.
  • fastgen.networks.inception moved to scripts/fid/inception.py.

Config fields

v0.1.0 v0.2.0
model.guidance_scale model.guidance_cfg.cfg.scale
model.skip_layers, and skip_layers_start_percent in network sample() model.guidance_cfg.cfg.skip_layers, scheduled with cfg.skip_layers_boundary
model.student_sample_steps model.student_grid_cfg.steps
model.sample_t_cfg.t_list model.student_grid_cfg.grid_type="explicit" with grid_kwargs={"timesteps": [...]}
model.fsdp_meta_init model.meta_init
model.precision_fsdp model.precision_opt
model.add_teacher_to_fsdp_dict=False model.extra_fsdp_modules=[]
trainer.tf32_enabled trainer.tf32_matmul and trainer.tf32_conv
log_config.wandb_credential, trainer.checkpointer.s3_credential files in credentials_dir (default ./credentials)
MeanFlow model.guidance_mixture_ratio=κ model.guidance_cfg.mix.scale=κ+1
MeanFlow model.guidance_t_start, model.guidance_t_end per-interval scale lists split at boundary in model.guidance_cfg
MeanFlow model.sample_t_cfg.r_sample_ratio=p model.sample_t_cfg.flow_matching_ratio=1-p
MeanFlow model.sample_r_cfg model.sample_r_overrides, the sample_t_cfg fields that differ for r
CM and sCM sample_t_cfg.quantize, sCM loss_config.divide_x_0_spatial_dim, Self-Forcing enable_gradient_in_rollout removed

TF32 matmuls are now off by default, which is PyTorch's own default; v0.1.0 turned them on. Set trainer.tf32_matmul=True to get the old behavior back.

Network arguments

v0.1.0 v0.2.0
SD1.5, SDXL, FLUX.1 and Qwen-Image model.net.model_id model.net.model_id_or_local_path
EDM model.net.model_type and UNet arguments such as img_resolution, label_dim and dropout a UNet config in model.net.backbone, such as L(SongUNet)(img_resolution=32, ...)
DiT model.net.num_classes model.net.label_dim
Wan model.net.use_wan_official_sinusoidal removed

Networks now reject unknown arguments, so a stale field raises an error instead of being ignored.

Configs and files

  • EDM/config_dmd2_test.py is now EDM/config_dmd2_fast.py.
  • CogVideoX/config_kd.py and CogVideoX/config_sft_5b.py were removed.
  • scripts/inference/prompts/validation_aug_qwen_2_5_14b_seed42.txt is now video_prompts.txt.

Inference scripts

  • --num_steps is now --sample_steps. It counts denoising steps (sampling-grid intervals), not grid points, and defaults to each network's own step count instead of 50.
  • --fps is now the config override model.net.fps; use model.teacher.fps for a separate teacher.
  • --save_as_gif is now --vid_format gif.
  • --num_conditioning_frames and --conditional_frame_timestep were removed:
    • Video2World is configured on the network with model.net.is_video2world and model.net.num_conditioning_frames; see CosmosPredict2/config_*_v2w.py.
    • The conditioning-frame timestep is now the conditional_frame_timestep argument of the Cosmos Predict2 network's forward() and sample(). It defaults to 1e-4; the v0.1.0 flag defaulted to 0.

Requirements

  • Python 3.10–3.12, with NumPy 1.x.
  • PyTorch 2.10, diffusers 0.40.0 and transformers 5.15.0 (previously PyTorch ≥2.6, diffusers 0.35.1 and transformers 4.49.0).
  • New dependencies include torchvision 0.25, torchao 0.16 and peft 0.20.0.
  • The Docker image is based on nvcr.io/nvidia/pytorch:25.11-py3 (previously 25.06).
  • ARM64: install the CUDA build of PyTorch from https://download.pytorch.org/whl/cu130 before pip install -e ., as the README describes; PyPI has only CPU builds for aarch64.
  • Gated models: FLUX.1-dev, FLUX.2-dev, FLUX.2-klein-base-9B and LTX-2.5 are gated on Hugging Face. Accept each model's license there, and provide an access token in HF_TOKEN or credentials/hf_token.txt.

Known issues

  • GPU torch.stft, which the LTX-2 audio path uses, can fail with CUFFT_INTERNAL_ERROR on some driver and container combinations. We saw this with nvcr.io/nvidia/pytorch:25.11-py3 on driver 580.178.04 (GB300); the torch 2.10.0+cu130 wheel from download.pytorch.org works on the same machines.
  • NumPy 2 is not supported yet (#43).

Released Models

  • FastGen-AYF: FLUX.1-dev and Qwen-Image LoRA adapters, distilled with AYF for 4–8-step generation.

Full changelog: v0.1.0...v0.2.0