How It Works

How It Works

Technical notes on what the extension does inside Forge Neo. Everything happens at runtime, in memory: no Forge Neo file is changed.

The model

• Pixel space. Iris 3B is a rectified-flow transformer (v-prediction, shift 4, model time = 1000 × sigma) that denoises the RGB image itself in [-1, 1]. The pipeline's VAE slot holds an empty stand-in, and encoding and decoding are the identity, so Forge's RGB live preview shows the real image.
• Loading. The detector reads the transformer's widths from the tensor shapes (rows and norms only, so 4-bit builds with packed columns detect correctly), and the transformer is built through Forge's own Krea 2 loading path, which brings dtype, quantization (INT8 ConvRot, W4A8, GGUF) and offloading with it. INT8 ConvRot zero-pads the trunk's feed-forward width from 6826 to 6912; the text refiner keeps 6826.
• Noise. Forge builds the starting noise as an image latent at 1/8 of the resolution; the extension replaces it with full-resolution noise from the same seed.

Text encoder

• The Qwen3-VL 4B language stack that Forge Neo already loads for Krea 2, with the Iris chat template: the system instruction, the prompt, the assistant turn, the last 300 positions, 12 hidden layers (2, 5, … 35).
• The prompt is cut at 300 tokens minus the template's closing turn (about 295). Surrounding whitespace is dropped, so "", " " and a prompt holding only LoRA tags are all the empty prompt Iris was trained on - its guidance null, and the conditioning of the restoration and depth fine-tunes.
• The text adapter (the layer-pooling blocks and the refiner) runs inside the transformer at every step, on the GPU.

Sampling

• Iris DPM++ 2M is the official FlowDPM-Solver++(2M) (speridlabs/iris-3b, src/iris3b/flow/solver.py): data prediction on the flow schedule alpha = 1 - sigma, first order on the first and last steps, landing exactly on the predicted image. Forge's own DPM++ 2M uses the formula for variance-exploding models, a different integrator on a flow model.
• The Iris schedule is the official grid: 1 - linspace(1, 0.001, n + 1), shifted by 4, ending at 0.
• Iris Turbo is the Turbo LoRA's grid, shift(1 - i/n) from sigma 1, used with LCM (predict the image, add fresh noise at the next sigma): 1, 0.923, 0.8, 0.571 at 4 steps.

Restore x4 and Depth

Both follow the official recipes (src/iris3b/downstream/) and run through Forge's own img2img, so model loading, LoRAs, saving and infotext work as usual:

• The prompt keeps only its LoRA tags, CFG is 1, and the sampling is one Euler step from the task's sigma to 0. Forge mixes noise into an img2img input as sigma × noise + (1 - sigma) × image; the extension passes the image itself as the noise, so the input is exactly the image.
• Restore x4: the input is enlarged 4× by bicubic (and to a short side of at least 1024), passed at sigma 0.5, and the result is x - 0.5 × v. The fine-tune only runs on 1024×1024 tiles: larger images are covered by tiles at stride 512, fused with the official Gaussian window in float32. A wavelet colour fix keeps the restored detail on the colours of the bicubic enlargement.
• Depth: the long side is capped at 1024 and both sides rounded to 16 px; one pass at model time 1000 returns relative log depth (-1 near, +1 far), shown as greyscale scaled to its own 2nd-98th percentiles, near bright. The official 4-channel depth/model.safetensors is not loaded; nynxz's build folds it into the standard layout with the same result.

GPUs without bfloat16

On cards such as the RTX 20 series, Forge runs Iris in float16. The trunk fits easily, but the pixel stage's residual stream does not: in the depth fine-tune its last block reaches about 4.3 × 10⁵, past float16's 65504. Under float16 the extension carries that stream and its norms in float32 while the layers keep Forge's precision; on CPU this matches the bf16 reference with a correlation of 0.99997.

LoRAs

• The Turbo LoRA stores bare layer.lora_down / layer.lora_up keys with alpha and rank in its metadata; the extension converts them for Forge's LoRA loader.
• A LoRA carries the feed-forward width it was extracted at (the upscaler LoRA: 6912). The padded units are always zero, so each LoRA is fitted exactly to the loaded checkpoint, layer by layer, by zero-padding or cropping.
• On INT8 checkpoints Forge always applies LoRAs while generating, so Patch LoRAs on-the-fly gives the same image on or off.

img2img variations

img2img with the base model is Forge's usual mechanism; the official code has no base-model img2img to compare with. At shift 4, denoising 0.3 already starts from sigma 0.67 (67% noise), and the model re-synthesizes fine detail on top of the input's, so variations come out sharper and more contrasted. Denoising 1.0 reproduces the txt2img result for the same seed, and the effect does not depend on CFG or the step count.

Files

File	Status
nynxz iris-3b_bf16, iris-3b_int8_convrot	Tested on an RTX 2070
nynxz iris-3b.safetensors, speridlabs model.safetensors (FP32)	Detected; the same weights at full precision, not tested on a GPU yet
nynxz iris-3b-depth_bf16, iris-3b-depth_int8_convrot	Tested with Depth; refused in txt2img
speridlabs upscaler/model.safetensors	Tested with Restore x4; it has the base layout and no tag, so it cannot be told apart from a base checkpoint
realrebelai Iris-3B-W4A8.safetensors (asym_w4a8_int8) and GGUF	Detected and expected to load; not tested yet
speridlabs depth/model.safetensors	Not supported (4-channel layout)
Aikimi iris-3b-int8 / iris-3b-w4a8	Refused with a message: a format of Aikimi Forge Neo's own loader

Measured on an RTX 2070 (8 GB)

Run	Time
1024×1024, 50 steps, CFG 3, INT8	about 95 s
1024×1024, Turbo LoRA, 4 steps, INT8	about 4 s (bf16: about 6 s)
Restore x4, 256 → 1024	about 10 s
Depth, 1024×1024	about 6 s