Skip to content

joliGEN v5.0.0

Latest

Choose a tag to compare

@beniz beniz released this 27 Oct 09:34
· 1 commit to master since this release

Highlights

  • B2B image and video generation, with reference-image and global-context conditioning, mask prediction, LoRA fine-tuning, and configurable temporal conditioning.
  • B2B export and inference workflows for ONNX and TensorRT, including class and mask conditioning and optional fixed temporal noise.
  • Shortcut-model training and image inference.
  • MAT inpainting integration, inference scripts, and motion support.
  • Expanded data controls: multiple datasets, bounding-box context crops, fixed masks, temporal sampling, and camera-color and detail augmentations.
  • Improved video evaluation, masked B2B metrics, and training continuation support.
  • A staged release workflow that validates the GPU server image before publication.

The server image includes the training and HTTP inference interfaces. Optional
deployment backends such as TensorRT require their own runtime dependencies;
the server image is not a TensorRT distribution.

Changes

Features

  • ability to force min border pixels for all online masks (2d7f774)
  • ability to resume multi dataset configuration (058d9a1)
  • ability to use no discriminator in supervised settings (86538be)
  • add optional fixed temporal init noise (0ae671a)
  • added min max border pixels with fixed squared masks (61b93ed)
  • allow fixed temporal noise in TensorRT inference (edb3d0a)
  • allowing largest side to be matched against the input frame size constraint (9be63ea)
  • aug: any angle rotations with online dataloaders (736cc46)
  • aug: in the wild data augmentation (548fb98)
  • aug: online random offset augmentation with square masks (a34d729)
  • b2b bottleneck disabling option (bd51919)
  • b2b global context conditioning with tokens (d7e87d6)
  • b2b lora (4428ed2)
  • b2b loss on masked region only (2b6f9f1)
  • b2b model validation loss (5496f7d)
  • b2b model validation loss (3e5f9e2)
  • b2b multi dataset class token per dataset support (da8551c)
  • b2b option to disable inference time clipping (5b603cf)
  • b2b P_mean, P_std and clamping controls (03d5534)
  • b2b pseudo huber loss option (2d103ff)
  • b2b random fps mixing (94ea29f)
  • b2b support for perceptual losses (1ce4b82)
  • b2b tensorrt inference (d09b48b)
  • b2b timestep uniform mixing (a307677)
  • b2b/jit smartbrush mask prediction (a292b59)
  • bbox context augmentation for b2b (db15334)
  • bbox context augmentation for b2b (2b94855)
  • broaden mask augmentations (f2533ea)
  • camera_color and detail augmentations (ce7000d)
  • cls and mask conditioning for tensorrt inference (1cbd915)
  • cls and mask conditioning support to b2b onnx exports and inference (8373416)
  • configurable motion module layers in b2b model (60b7978)
  • continue cls inference updates (a2a3e38)
  • controled inference for MAT (639bd40)
  • crop delta override in multi-dataset (b6b3d9d)
  • data: fixed size square inpainting mask (5b902dc)
  • data: multi dataset loader support (7df1f98)
  • data: multi-dataset test sets support (be00b9c)
  • finetune directly from another checkpoint dir with --train_continue_from (39d21a4)
  • forcing token conditioning on finetune (6ec6bea)
  • frame step conditioning in video (3059eb7)
  • full frame conditioning for b2b (592cf68)
  • inference scripts for MAT (db33849)
  • inference: allow PTH predicted masks without mandatory bbox (8bbf563)
  • initial data aug for b2b diffusion (04062c9)
  • input frame size control at training time (94ed978)
  • mask coordinates conditioning (a21e4c8)
  • ml: 2 frame video input with 1GT (a94d8a4)
  • ml: add DINOv2 for metric compute (00ece3f)
  • ml: add lpips metric for dilated crop (e2723d7)
  • ml: add mask as channel (297d982)
  • ml: add MAT inpainting model integration (66d34b2)
  • ml: add motion module into vit (0aa5714)
  • ml: add motion module into vit (3efb075)
  • ml: apply dilated-mask PSNR and SSIM metrics for DDPM and CM image generation (b26f312)
  • ml: autoregressive training (455c14e)
  • ml: autoregressive training (8cab76d)
  • ml: check model and NN (a232dfd)
  • ml: clean code (4355642)
  • ml: clearn code (ddf278e)
  • ml: CM model apply to vid (65f4be1)
  • ml: ddim_vid (54cdcea)
  • ml: debug add mask as additional channel for b2b model (c2456d1)
  • ml: debug unite test (36b05ab)
  • ml: enlarger loss calculation (62e80de)
  • ml: enlarger loss calculation (085637c)
  • ml: extend ddpm_vid model to support alg_diffusion_cond_embed mask (9bd2816)
  • ml: fine-tuning consistency model based on the pretrained DDPM (ca96a80)
  • ml: fix bug (d5e78f9)
  • ml: FT img256tovid352 (cc3f8a4)
  • ml: implementation of shortcut model (f1b0035)
  • ml: inference autoregressive (7037b48)
  • ml: inference single image for b2b model (9edb9c4)
  • ml: multi layer motion module application (9a0a53a)
  • ml: offline autoregressive generation (5f332f8)
  • ml: restrict PSNR and SSIM computation to dilated masked zones with option alg_cm_metric_mask (460bbfa)
  • ml: restrict PSNR and SSIM computation to dilated masked zones with option alg_palette_metric_mask (05ac64d)
  • ml: Support DDIM inference schedule in DDPM training (f2ef2d7)
  • ml: training AR with 90% noisy images and 10% one noisy image+one (3c9dac6)
  • ml: unite test (1cf2118)
  • ml: video cls traning and inference (62e6018)
  • ml: vit and vit_vid with registers (e60f065)
  • ml: vit_vid input 4 channel and output 3 channel (6c0baab)
  • models: HAT 2309.05239 (697d454)
  • motion for MAT (bfa78e5)
  • multidataset config from nested dirs (4c6ce05)
  • per denoising step metrics (73f3de4)
  • reference image conditioning for b2b (3bcb0d7)
  • release upgrade (621bb51)
  • script to study corrupted results (9625893)
  • support bbox-context crops in PTH inference (c7e6219)
  • temporal conditioning copy loss + frame degradation (25c8d62)
  • testing: Add unit test for shortcut model (fcd8142)

Bug Fixes

  • added missing true auto-regressive behavior to b2b onnx and tensorrt inference scripts (8af587e)
  • automatic hold out on multi-dataset (d22dae6)
  • b2b adam params, ema and noise scale (cf1d1e0)
  • b2b cfg_scale defaulting to 1.0, and skipping of v_uncond at inference for that value (cdafaf2)
  • b2b label_drop_prob set to 0.0 for single class / unconditioned problems (6b5551e)
  • b2b model in eval mode for validation (eb6a77e)
  • b2b onnx patch integration (b7f1f35)
  • b2b restoration timestep for conditional image (d54a40b)
  • clamp final b2b output (add9f9c)
  • cm test visuals (4721e09)
  • correct image training bug in ect tuning (1b738e4)
  • correct typo (191c7f5)
  • data: multi-dataset generator script (eb757d5)
  • decouple l1/mse loss from perceptual losses in paired training (12867fa)
  • edge clip broaden mask augmentation (5820c60)
  • eval mode for visuals (e0d13e3)
  • evaluate all video test batches for metrics (06e9b15)
  • evaluation noise consistency across steps count and epochs (c88d5c0)
  • global context augmentation with b2b (c536fd2)
  • identical mask semantics at training and inference (c8cbbb4)
  • inference: prevent touching gpu_ids in config when do inference (adbf66a)
  • inference: prevent touching gpu_ids in config when using runtime (58a18c0)
  • json to shema (40aa6c5)
  • load dir with train_continue and train_continue_from (8df9ecd)
  • mask B2B PSNR and cover temporal tests (2a3a6e2)
  • mat get image import (7ec6c8c)
  • mat motion module visualization (a626b03)
  • metric: compute PSNR/SSIM over all frames by flattening 5D tensors to 4D (51d6135)
  • missing b2b_context.py file (384eb36)
  • missing temporal_sampling.py (0b4f1d3)
  • ml: correct else indentation for no-condition training in CM model (c7310fc)
  • multi-classes masks with diff aug activated (3d21638)
  • multi-dataset configurator with no hold outs (44829c8)
  • multidataset for images only (459616e)
  • NCE idt not activated on the right features, #744 (8e3b682)
  • onnx and trt b2b inference (37fecdd)
  • persistent dataloader workers (72e74b0)
  • persistent workers eating too many file handlers at test time (d528820)
  • pin peft version (8bbb313)
  • preserve semantic colors in B2B mask visuals (5d4b921)
  • registers start at the same level as in context tokens (1120346)
  • remove unused cross-attention dimension option (5240e05)
  • removed b2b per-step clamping (27e15e6)
  • Save first-frame noisy inpainting debug output autoregressive (de986e2)
  • Set default sampling method to ddpm instead of ddim (2a97573)
  • temporal motion import (9132c7b)
  • UViT efficient skip connection weight (0e11857)
  • video generation with simpler name and correct crop (efbb303)
  • vision aided loss package with python 3.12 (missing pkg_resources from setuptools) (60b4ed6)
  • warn and exit on invalid command-line arguments (da1fed1)
  • warn user about unknown command line options (e615646)
  • with smartbrush mask prediction, previous frame conditioning for video contains mask when trained with ground truth (32d34cf)
  • with smartbrush-like prediction, the mask class is reported onto the binary predicted mask at inference (4385831)

Docker image

docker pull docker.jolibrain.com/joligen_server:v5.0.0
docker run --rm --gpus all -p 8000:8000 docker.jolibrain.com/joligen_server:v5.0.0