Skip to content
github-actions[bot] edited this page Nov 7, 2026 · 1 revision

Open-weight models

Everything formcut can run locally, where each model comes from, what licence it carries, and the code that loads it. Nothing is fetched unless a person asks for it: every download is triggered from the interface or from the API below, and the engine ships exactly one small model inside its build, the 4.6 MB U2NetP at backend/app/bundled_models/u2netp.onnx, so the default cutout works before anything has been downloaded (BG2; the decision is recorded in Licences). A file in the person's model folder always takes precedence over the bundled copy.

All of these run through ONNX Runtime on the engine side, Whisper included since v1.62.0 (backend/app/transcribe.py); a browser whose engine has no ONNX Runtime still runs Whisper in the page through transformers.js. Both are already dependencies; no additional runtime is needed for any model in this list. Every engine-side session takes one provider ordering from model_manager.providers(): the platform accelerator first where present (Core ML on macOS, DirectML on Windows, CUDA on Linux with NVIDIA), the CPU provider always the floor; GET /api/models/runtime reports what is active, and Local models shows it as "Processor".


1. Segmentation and cutout

Used by: Cutout and brushes in the Creator panel, Select subject (auto-masking), and background removal. Registered in backend/app/model_manager.py:

Registry id Model Purpose Licence Size
object-cutout u2netp.onnx general object cutout, the small U²-Net; also the bundled out-of-the-box model Apache-2.0 ~4.6 MB
person-cutout u2net_human_seg.onnx people, trained for human segmentation Apache-2.0 ~176 MB
sharp-cutout isnet-general-use.onnx IS-Net, sharper edges than U²-Net Apache-2.0 ~179 MB
video-matting rvm_mobilenetv3_fp32.onnx Robust Video Matting for people in motion; GPL-3.0-only, offered as an explicit download of upstream's own release, never bundled (see Licences) GPL-3.0-only ~15 MB
click-select SAM 2 Hiera Tiny encoder and decoder point or box prompts: click what you mean Apache-2.0 ~155 MB
click-track SAM 2.1 Tiny video (six files) follows a prompted subject through a clip Apache-2.0 ~190 MB

The model select also offers Subject (Auto), which is an arbitration rather than a model: the person model runs first, its matte is kept when it holds at least one percent of the frame at half threshold, and the object model answers otherwise. Auto needs both models it chooses between, so it is refusable with the same download message until person-cutout is fetched.

The smart brush (Smart restore and Smart erase beside the plain pair) turns a stroke into a prompt: /api/render/smart-brush maps the painted points into the clip's own pixels, spaces up to five of them as positive SAM 2 point prompts on the frame the stroke was made on, and stores the returned region as an outline on the stroke, which the matte shaper fills on every frame. Smart strokes need click-select downloaded.

Upstream research: U²-Net, Qin et al., Apache-2.0 (https://github.com/xuebinqin/U-2-Net); IS-Net from DIS (https://github.com/xuebinqin/DIS); the ONNX exports are redistributed by the rembg project (https://github.com/danielgatis/rembg, MIT). SAM 2 and 2.1 are Meta's, Apache-2.0, as ONNX exports; Robust Video Matting is PeterL1n's own release.

Adding another model needs a registry entry with its pin in the same change (the licence gate in Licences), and an entry in the cutout allow lists:

# backend/app/model_manager.py
MODELS["birefnet"] = {
    "name": "BiRefNet general matting",
    "license": "MIT",
    "files": {"birefnet-general.onnx":
              "https://github.com/danielgatis/rembg/releases/download/v0.0.0/birefnet-general.onnx"},
}
PINNED["birefnet-general.onnx"] = "<sha256 of the file you fetched>"

Then add it to CUTOUT_MODELS in backend/app/segmentation.py with a branch in _infer, and to the cutout model list in backend/app/validation.py. BiRefNet (MIT) and silueta (Apache-2.0, U²-Net quality at a quarter the size) remain the recorded candidates, in the same rembg releases.

The loading code is _session in backend/app/segmentation.py; every session takes the shared provider ordering:

session = onnxruntime.InferenceSession(str(model_file(file)), providers=providers())
# U²-Net family: NCHW float32, 320x320, normalised; one output, the matte
matte = session.run(None, {session.get_inputs()[0].name: tensor})[0][0][0]

model_file() answers model-folder-first, then the bundle. outline() in the same module traces a matte into editable mask anchors, which is what Select subject returns, and what a smart stroke's region is stored as.

Absent-model behaviour: object-cutout is always ready through the bundle, so the default cutout mattes out of the box; any other choice refuses with a download message until fetched, and the renderer reads a cached matte and renders the clip unmatted when there is none, rather than stalling.


2. Speech synthesis

Used by: Studio → Local speech. Registered as local-speech (Kokoro); Pocket TTS (Kyutai, CC-BY-4.0 weights) and Chatterbox (Resemble AI, MIT, with an inaudible watermark on its audio) are installable engines beside it, set up once with backend/setup_local_ai.py and chosen from the same panel.

File Licence Size
kokoro-v1.0.onnx Apache-2.0 ~310 MB
voices-v1.0.bin Apache-2.0 ~27 MB

Links

  • https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.1/kokoro-v1.0.onnx
  • https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.1/voices-v1.0.bin

Upstream weights: https://huggingface.co/hexgrad/Kokoro-82M (Apache-2.0). Runtime wrapper: https://github.com/thewh1teagle/kokoro-onnx (MIT).

Adding other voice models needs no code: the engine takes them at runtime.

POST /api/models/speech
{"id": "piper-en", "name": "Piper English",
 "model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx",
 "voices_url": "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json",
 "license": "MIT"}

Piper voices (https://github.com/rhasspy/piper, MIT) are the usual alternative: around 60 MB each, many languages, listed at https://huggingface.co/rhasspy/piper-voices.


3. Transcription and captions

Used by: Captions. Since v1.62.0 these run in the engine through ONNX Runtime with the shared provider ordering (backend/app/transcribe.py, the job behind POST /api/transcribe), word times taken from the model's alignment heads; a browser whose engine answers 503 runs the first six in the page through transformers.js instead. All are fetched from the Hugging Face hub on first use. The allowlist is WHISPER_MODELS plus ENGINE_WHISPER_MODELS in backend/app/model_manager.py; the last three are the engine's alone and are never prefetched by FORMCUT_MODELS_AUTO=all.

Model Licence Size (quantised)
Xenova/whisper-tiny.en MIT ~40 MB
Xenova/whisper-base.en MIT ~80 MB
Xenova/whisper-small.en MIT ~250 MB
onnx-community/whisper-tiny MIT ~40 MB
onnx-community/whisper-base MIT ~80 MB
onnx-community/whisper-small MIT ~250 MB
Xenova/whisper-medium.en (engine only) MIT ~776 MB
Xenova/whisper-medium (engine only) MIT ~776 MB
onnx-community/whisper-large-v3-turbo (engine only, multilingual) MIT ~1.08 GB

Weights are OpenAI Whisper (MIT, https://github.com/openai/whisper); the repositories above are ONNX conversions. The .en variants are English-only and noticeably better at English than the multilingual model of the same size.

Loading code is in the captions worker:

import { pipeline } from "@huggingface/transformers";
const transcribe = await pipeline("automatic-speech-recognition", modelId, {
  device: navigator.gpu ? "webgpu" : "wasm",
});

4. Upscaling

Used by: AI upscale. Registered as ai-upscale: Real-ESRGAN x4plus as an ONNX export (realesrgan-x4plus.onnx, BSD-3-Clause, ~67 MB), converted from the official weights by a published, reproducible recipe and pinned to the converter's stated SHA-256. The upscaler tiles the frame and runs through the same provider ordering as everything else.


5. Not yet wired, but the obvious next additions

These have no code in formcut today. Each needs a registry entry with its pin, a place in the interface, and a call site; the links are here so that work does not start with a search.

Purpose Model Licence Link
Interpolation RIFE v4.6 ONNX MIT https://github.com/hzwer/Practical-RIFE
Depth, for 3D effects Depth Anything V2 Small Apache-2.0 https://huggingface.co/depth-anything/Depth-Anything-V2-Small
Face detection YuNet ONNX Apache-2.0 https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet
Speech separation Demucs v4 MIT https://github.com/facebookresearch/demucs
Denoise, audio DeepFilterNet 3 Apache-2.0 / MIT https://github.com/Rikorose/DeepFilterNet
Upscaling, animation RealESRGAN-x4plus-anime BSD-3-Clause Real-ESRGAN releases

The earlier entries on this list have landed: Real-ESRGAN is ai-upscale, SAM 2 is click-select and click-track, and Pocket TTS and Chatterbox are installable speech engines.


How downloads work

Every model is fetched by the same path, so nothing above needs special code:

GET    /api/models                  what is registered and what is on disk
GET    /api/models/runtime          the active execution provider and the available list
POST   /api/models/{id}/download    fetch it, as a job with progress
POST   /api/models/{id}/import      supply a file you already have
DELETE /api/models/{id}             remove it
GET    /api/models/speech           speech models added at runtime
POST   /api/models/speech           add one

Three things are true of all of them:

  • Nothing is fetched without being asked. The one weight the engine carries is the bundled U2NetP, inside the build itself, so the default cutout works offline from the first run. FORMCUT_MODELS_AUTO governs startup fetching: 1 (the default) fetches only the default cutout model into the model folder, all fetches every permissively licensed model for a machine that will go offline, 0 fetches nothing.
  • Checksums are pinned. PINNED in model_manager.py holds a SHA-256 for every registry file, taken from real fetches and matching the table in Licences; a mismatch deletes the download rather than caching it. Add a pin in the same change that adds a model.
  • Sources are redirectable. sources.json in the data directory lets a person point any model at a mirror or a local path, for machines that cannot reach GitHub or Hugging Face; the pin still applies, so a mirror cannot substitute a different file.

Local image models on Apple silicon: Krea 2, FLUX.2 [klein] 4B, LoRA stacks

Krea 2 Turbo and FLUX.2 [klein] 4B are optional, installed from their cards in Studio > Local models and run by mflux in their own environment. Neither ships with formcut: their MLX builds come from mflux-community on Hugging Face, pinned file by file, when you install them. FLUX.2 [klein] 4B is Apache-2.0, commercial use allowed. The 9B Klein models are under the FLUX Non-Commercial License and are not offered.

LoRAs live in a folder per model: models/loras/krea-2/ and models/loras/flux2-klein-4b/. Drop .safetensors files in and they appear in the generator's LoRA stack (Look again reads the folder afresh). A <name>.json beside a file keeps its name, strength, trigger words and source. Import a file, or Add from a link:

  • a Hugging Face file link, https://huggingface.co/<repo>/resolve/<revision>/<file>.safetensors, for the model you have chosen;
  • a Civitai model, version or download link (https://civitai.com/models/<id>, ...?modelVersionId=<id>, https://civitai.com/api/download/models/<id>): formcut reads Civitai's record, puts the file in the folder of the model Civitai names (Krea 2, or Flux.2 Klein 4B and its base), refuses any other (Klein 9B, SDXL, Flux.1 ...), checks the download against Civitai's SHA-256 and keeps its trigger words. Civitai asks for an account to download most files: add your Civitai API key in Settings.

Prompts written as Civitai shows them work in the generator for Krea 2, Klein 4B and their checkpoints: <lora:name:strength> tags anywhere in the prompt (the name is the LoRA's file name without .safetensors, found the way Automatic1111 finds it, then its name in formcut or the file name it was downloaded as; a strength may be negative, as sliders use), and a pasted picture's generation data: Negative prompt: and the Steps: ..., Seed: ..., Model: ..., width/height or Size: WxH line. Steps, Seed and size are used (the size rounded to a multiple of 16); Model: chooses a checkpoint of that name in models/checkpoints/<model>/; CFG other than 1, the sampler, the schedule, clip skip and a negative prompt are not taken by these distilled models, and the result says so. A tag whose LoRA is not in the model's folder stops the picture and names the folder to put it in.

Prompt wraps

A prompt wrap is its own kind of run, beside Generate. With Krea 2 or one of its checkpoints chosen, the image generator shows a Prompt wrap section under the settings: Load prompt wrap… takes a JSON file of three prompts or more (examples/prompt-wrap.json shows one) and lists its entries as the engine read them; Run wrap runs them in turn, and with Round and round ticked, from the first again after the last until stopped (unticking it lets the picture in hand finish, then stops; Stop stops at once). Nothing of the settings above reaches a wrap: not the prompt box (which may be empty), the size, steps, seed, sampler, CFG, negative prompt, LoRA stack or Start from picture. Each entry brings its own; what an entry leaves out comes from the file's "defaults", then from the model's own (1024 × 1024, its steps, its sampler, CFG off, a random seed). Generate, for its part, never runs the wrap.

The file is {"defaults": {...}, "prompts": [...]}, {"prompts": [...]} or the list alone; an entry, or the defaults, may say {"prompt": "...", "width": 832, "height": 1248, "steps": 9, "seed": -1, "loras": [{"name": "Emotions V1", "scale": 1.0}], "sampler": "euler", "cfg": 1, "negative": "..."}, and a string entry is its prompt. LoRAs are named as the library names them (its file names, as a <lora:> tag does, which a prompt may carry too, with Civitai's generation data); scales run from -2 to 2, eight at most; sizes are made multiples of 16 from 256 to 2048. Every entry passes the same prompt guard a single prompt does. The file is checked when it is loaded, and an entry that cannot be read is named. The model is loaded once for the whole wrap, however its entries differ: formcut's mflux worker (backend/app/mflux_worker.py, run in formcut's mflux environment) builds the model the way mflux-generate-krea2 does, with every LoRA the wrap uses applied unbaked, then makes each picture with mflux's own generate_image, its prompt, size, steps, seed, sampler and CFG, each LoRA at that entry's scale and the LoRAs it does not use set aside. One load carries only as many LoRAs as the memory holds beside the model (four fifths of the memory, less the model and 3 GB); a wrap naming more is split into as few loads as fit. Should the worker not run with the installed mflux, the wrap goes on a setup at a time (a load where an entry's LoRAs, size or steps change) and says so. (Until 2.3.3 every change of setup was a new load: with a LoRA taking turns, every entry, and the owner's run stalled at loading and took the browser down.) Each picture records which entry made it. The engine's endpoint is POST /api/generate/wrap-run {generator, wrap, continuous}.

The generative library

What the image and video generator makes is kept apart from the media library: files in generative/, thumbnails in generative-thumbs/, records (prompt, seed, steps, size, LoRAs, model, time) in generative.json, in the data folder. The Studio's results show it, latest first, as a gallery: the picture fills its tile, its seed, steps, size and wrap position show on hover with Add to media (a copy into the media library, for the timeline; the generation stays) and Delete; a click opens it full screen with its prompt. Image-to-image can start from a generation. Audio the generator makes (speech, music) still goes to the media library, as it is laid on the timeline at once.

LoRAs from OneTrainer, Civitai and other trainers

mflux matches a LoRA's keys to its model by exact patterns, and trainers name the same weights differently. formcut checks each LoRA in the library against the installed mflux's own mapping for its model (it runs in formcut's mflux environment, so it is right for that mflux), once per file and mflux version, and shows the result under the LoRA's row in the generator: how many keys apply, what was renamed, and what is left and why. Where renaming makes keys apply, a copy with the renamed header (the tensors' bytes as they were) is written to .converted/ beside the file, and mflux is given that copy. Renamed: OneTrainer's historical layout (lora_transformer_…, where mflux reads lora_unet_…), LyCORIS's lycoris_… and flat LoKr names (read into the dotted names mflux's Krea 2 mapping knows), and ComfyUI's model.diffusion_model.. Said, not fixable by renaming: text-encoder weights (mflux applies LoRAs to the image model only), a DoRA's magnitude (mflux applies only its LoRA part, so it will not look as trained), LoHa (hada_) and full-weight differences, which mflux does not load. A LoRA none of whose keys apply is marked unusable with the reason (made for another model, a LoHa, text-encoder weights only), rather than stopping a run. OneTrainer's DIFFUSERS, KOHYA, ORIGINAL and COMFY LoRA outputs all load as they are.

Steps, seed, continuous runs, and what each picture carries

For Krea 2, Klein 4B and their checkpoints the image generator has a Steps slider (1 to 50, starting at the model's own: 8 for Krea 2, 4 for Klein) and a Seed slider with its number beside it: -1 (Random) draws a new seed each run, any other seed repeats a picture. The seed used is shown on each result and in the note, and written into the picture's file as Automatic1111 writes it (the PNG's parameters: the prompt, the LoRAs as <lora:file:strength> tags, then Steps, Sampler, CFG scale, Seed, Size, Model), beside mflux's own record; the library keeps the same. Pasted back into the generator, that text asks for the same picture. With a random seed, Continuous starts the next run as each one finishes, until it is unticked, cancelled, a run fails, or Studio is closed. With Krea 2 and its checkpoints, Continuous is a session in the engine: one run of mflux over hundreds of random seeds, so the model and its LoRAs are loaded once and stay in memory between pictures; each picture joins the results as soon as it is made (through the content filter, with its recipe written into it); the prompt can be changed while it runs, and the next picture reads it (mflux's --prompt-file), while the LoRAs, size and steps stay as the session started. Unticking Continuous lets the picture in hand finish and be saved, then ends the session and frees the memory; Cancel stops at once, and nothing starts again until Generate is pressed. Each result's time is in the note.

A result opens full screen with a click (or Enter): Escape, a click or Close returns; the arrow keys move through the results. While a picture is made, the terminal formcut was started from shows mflux's own output as it comes, its tqdm progress bar included (the steps, the time elapsed and left, its/s), redrawn in place and sized to the terminal, between a line from formcut when the run starts and one when it ends; FORMCUT_QUIET_PROGRESS=1 keeps it out. Under the generator's progress bar, the same reading as tqdm gives it: the step and [elapsed<left, rate], or while the model loads, how long loading has taken.

If Krea 2's content filter was removed, its warning goes with each picture until it is acknowledged (I understand) on the Krea 2 card in Studio > Local models; after that the pictures no longer carry it.

What formcut asks of mflux for Krea 2

mflux-generate-krea2 as mflux 0.20.0 parses it: --model the build's or checkpoint's folder (no --base-model, which the Krea 2 command does not need); --prompt; --width/--height in multiples of 16; --seed with every seed of a request, so several pictures come from one load of the model; --steps (8 unless asked); --scheduler er_sde (the default) or euler, the generator's Sampler (a Civitai "Euler ..." sampler becomes euler; mflux has no ancestral Euler); --guidance only when a CFG scale other than 1 is asked for (it turns CFG on, a second pass a step, slower), and then the negative prompt with --negative-prompt; each LoRA as --lora PATH SCALE, with --no-bake-lora for a 4-bit model (the 4-bit build, or a checkpoint whose folder name says q4 or 4bit) or when memory is short, as baking would turn a 4-bit model's LoRA layers to 8-bit; and for memory --low-ram (which tiles the decode), --mlx-cache-limit-gb, or --vae-tiling alone when the model fits but its untiled decode (about 7 GB at 1024 px) would not. NVIDIA PiD decoding (--pid-decode) is not used.

Image-to-image (--image PATH STRENGTH): Krea 2 and its checkpoints can start from a picture. In the image generator, Start from takes the frame at the playhead (rendered from the project as it stands, as an export would draw it) or an image or video in the media library (a video at the playhead's time); Image influence runs from 0 (the picture has no say) to 1, mflux's default 0.4; the size follows the source (Match the source picture, fitted within 2048 pixels, in multiples of 16) unless another is chosen. A pasted Civitai prompt's Denoising strength is read as 1 minus the influence (A1111's denoising is how much is repainted). FLUX.2 [klein] 4B does not take a source here.

Your own checkpoints

A Krea 2 or Klein 4B fine-tune runs as a model of its own once it is in mflux's MLX format, in a folder under models/checkpoints/krea-2/<name>/ or models/checkpoints/flux2-klein-4b/<name>/. mflux's format is the folder mflux-save writes: text_encoder/, tokenizer/, vae/, and the transformer's weights in transformer/ (Klein) or at the top (Krea 2). Studio > Local models lists what it found, and what a folder lacks. To make one from a fine-tune published in the diffusers layout (a Hugging Face repository or a folder with transformer/, text_encoder/, ...), use the mflux formcut installs:

models/krea-2/runtime/bin/mflux-save --model <repo or folder> --base-model krea-2 \
    --path models/checkpoints/krea-2/<name> --quantize 8

(--base-model flux2-klein-4b for Klein; --lora-paths and --lora-scales bake LoRAs in.) A single-file checkpoint as Civitai serves it for ComfyUI or Automatic1111 has to be put into the diffusers layout first; formcut does not do that conversion. A Krea 2 fine-tune is under Krea 2's licence, so it runs once that licence is accepted on the Krea 2 card, with the content filter as Krea 2 has it.

Every file is checked from its header: LoRA and LyCORIS LoKr load; DoRA in the PEFT layout does not (mflux refuses it); a file that fails is listed with its reason and cannot be stacked. In the stack, each LoRA has a strength from -2 to 2 and a switch; up to eight at once.

Memory: Automatic runs mflux as it runs from the command line, tiling the decode (--vae-tiling) when the model and an untiled decode would not fit beside each other, and uses --low-ram with a capped MLX cache only when the model itself would not fit (both cost a great deal of time); Low memory always uses them (slower); Normal uses none. Before a model runs, the preview's cached frames are let go (drawn again when asked for), and one local image model runs at a time. The terminal's first line for a run is the exact mflux command, to set beside mflux run by hand. The generator shows the model's own progress (loading, each step, finishing), and Cancel ends the model's process.

formcut

Pages

In the repository


These pages are published from .docs/wiki/ in the repository. Edit them there; a push to main updates this wiki. Editing a page here is overwritten by the next publish.

Clone this wiki locally