Repository navigation
Models
Everything formcut can run locally, where each model comes from, what licence it
carries, and the code that loads it. Nothing is fetched unless a person asks for
it: every download is triggered from the interface or from the API below, and
the engine ships exactly one small model inside its build, the 4.6 MB U2NetP at
backend/app/bundled_models/u2netp.onnx, so the default cutout works before
anything has been downloaded (BG2; the decision is recorded in
Licences). A file in the person's model folder always takes
precedence over the bundled copy.
All of these run through ONNX Runtime on the engine side, Whisper included
since v1.62.0 (backend/app/transcribe.py); a browser whose engine has no ONNX
Runtime still runs Whisper in the page through transformers.js. Both are already
dependencies; no additional runtime is needed for any model in this list. Every
engine-side session takes one provider ordering from
model_manager.providers(): the platform accelerator first where present
(Core ML on macOS, DirectML on Windows, CUDA on Linux with NVIDIA), the CPU
provider always the floor; GET /api/models/runtime reports what is active,
and Local models shows it as "Processor".
Used by: Cutout and brushes in the Creator panel, Select subject
(auto-masking), and background removal. Registered in
backend/app/model_manager.py:
| Registry id | Model | Purpose | Licence | Size |
|---|---|---|---|---|
object-cutout |
u2netp.onnx |
general object cutout, the small U²-Net; also the bundled out-of-the-box model | Apache-2.0 | ~4.6 MB |
person-cutout |
u2net_human_seg.onnx |
people, trained for human segmentation | Apache-2.0 | ~176 MB |
sharp-cutout |
isnet-general-use.onnx |
IS-Net, sharper edges than U²-Net | Apache-2.0 | ~179 MB |
video-matting |
rvm_mobilenetv3_fp32.onnx |
Robust Video Matting for people in motion; GPL-3.0-only, offered as an explicit download of upstream's own release, never bundled (see Licences) | GPL-3.0-only | ~15 MB |
click-select |
SAM 2 Hiera Tiny encoder and decoder | point or box prompts: click what you mean | Apache-2.0 | ~155 MB |
click-track |
SAM 2.1 Tiny video (six files) | follows a prompted subject through a clip | Apache-2.0 | ~190 MB |
The model select also offers Subject (Auto), which is an arbitration rather
than a model: the person model runs first, its matte is kept when it holds at
least one percent of the frame at half threshold, and the object model answers
otherwise. Auto needs both models it chooses between, so it is refusable with
the same download message until person-cutout is fetched.
The smart brush (Smart restore and Smart erase beside the plain pair) turns
a stroke into a prompt: /api/render/smart-brush maps the painted points into
the clip's own pixels, spaces up to five of them as positive SAM 2 point
prompts on the frame the stroke was made on, and stores the returned region as
an outline on the stroke, which the matte shaper fills on every frame. Smart
strokes need click-select downloaded.
Upstream research: U²-Net, Qin et al., Apache-2.0
(https://github.com/xuebinqin/U-2-Net); IS-Net from DIS
(https://github.com/xuebinqin/DIS); the ONNX exports are redistributed by the
rembg project (https://github.com/danielgatis/rembg, MIT). SAM 2 and 2.1 are
Meta's, Apache-2.0, as ONNX exports; Robust Video Matting is PeterL1n's own
release.
Adding another model needs a registry entry with its pin in the same change (the licence gate in Licences), and an entry in the cutout allow lists:
# backend/app/model_manager.py
MODELS["birefnet"] = {
"name": "BiRefNet general matting",
"license": "MIT",
"files": {"birefnet-general.onnx":
"https://github.com/danielgatis/rembg/releases/download/v0.0.0/birefnet-general.onnx"},
}
PINNED["birefnet-general.onnx"] = "<sha256 of the file you fetched>"Then add it to CUTOUT_MODELS in backend/app/segmentation.py with a branch
in _infer, and to the cutout model list in backend/app/validation.py.
BiRefNet (MIT) and silueta (Apache-2.0, U²-Net quality at a quarter the size)
remain the recorded candidates, in the same rembg releases.
The loading code is _session in backend/app/segmentation.py; every
session takes the shared provider ordering:
session = onnxruntime.InferenceSession(str(model_file(file)), providers=providers())
# U²-Net family: NCHW float32, 320x320, normalised; one output, the matte
matte = session.run(None, {session.get_inputs()[0].name: tensor})[0][0][0]model_file() answers model-folder-first, then the bundle. outline() in the
same module traces a matte into editable mask anchors, which is what Select
subject returns, and what a smart stroke's region is stored as.
Absent-model behaviour: object-cutout is always ready through the bundle,
so the default cutout mattes out of the box; any other choice refuses with a
download message until fetched, and the renderer reads a cached matte and
renders the clip unmatted when there is none, rather than stalling.
Used by: Studio → Local speech. Registered as local-speech (Kokoro);
Pocket TTS (Kyutai, CC-BY-4.0 weights) and Chatterbox (Resemble AI, MIT, with
an inaudible watermark on its audio) are installable engines beside it, set up
once with backend/setup_local_ai.py and chosen from the same panel.
| File | Licence | Size |
|---|---|---|
kokoro-v1.0.onnx |
Apache-2.0 | ~310 MB |
voices-v1.0.bin |
Apache-2.0 | ~27 MB |
Links
https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.1/kokoro-v1.0.onnxhttps://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.1/voices-v1.0.bin
Upstream weights: https://huggingface.co/hexgrad/Kokoro-82M (Apache-2.0).
Runtime wrapper: https://github.com/thewh1teagle/kokoro-onnx (MIT).
Adding other voice models needs no code: the engine takes them at runtime.
POST /api/models/speech
{"id": "piper-en", "name": "Piper English",
"model_url": "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx",
"voices_url": "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json",
"license": "MIT"}
Piper voices (https://github.com/rhasspy/piper, MIT) are the usual alternative:
around 60 MB each, many languages, listed at
https://huggingface.co/rhasspy/piper-voices.
Used by: Captions. Since v1.62.0 these run in the engine through ONNX
Runtime with the shared provider ordering (backend/app/transcribe.py, the job
behind POST /api/transcribe), word times taken from the model's alignment
heads; a browser whose engine answers 503 runs the first six in the page through
transformers.js instead. All are fetched from the Hugging Face hub on first use.
The allowlist is WHISPER_MODELS plus ENGINE_WHISPER_MODELS in
backend/app/model_manager.py; the last three are the engine's alone and are
never prefetched by FORMCUT_MODELS_AUTO=all.
| Model | Licence | Size (quantised) |
|---|---|---|
Xenova/whisper-tiny.en |
MIT | ~40 MB |
Xenova/whisper-base.en |
MIT | ~80 MB |
Xenova/whisper-small.en |
MIT | ~250 MB |
onnx-community/whisper-tiny |
MIT | ~40 MB |
onnx-community/whisper-base |
MIT | ~80 MB |
onnx-community/whisper-small |
MIT | ~250 MB |
Xenova/whisper-medium.en (engine only) |
MIT | ~776 MB |
Xenova/whisper-medium (engine only) |
MIT | ~776 MB |
onnx-community/whisper-large-v3-turbo (engine only, multilingual) |
MIT | ~1.08 GB |
Weights are OpenAI Whisper (MIT, https://github.com/openai/whisper); the
repositories above are ONNX conversions. The .en variants are English-only and
noticeably better at English than the multilingual model of the same size.
Loading code is in the captions worker:
import { pipeline } from "@huggingface/transformers";
const transcribe = await pipeline("automatic-speech-recognition", modelId, {
device: navigator.gpu ? "webgpu" : "wasm",
});Used by: AI upscale. Registered as ai-upscale: Real-ESRGAN x4plus as an
ONNX export (realesrgan-x4plus.onnx, BSD-3-Clause, ~67 MB), converted from
the official weights by a published, reproducible recipe and pinned to the
converter's stated SHA-256. The upscaler tiles the frame and runs through the
same provider ordering as everything else.
These have no code in formcut today. Each needs a registry entry with its pin, a place in the interface, and a call site; the links are here so that work does not start with a search.
| Purpose | Model | Licence | Link |
|---|---|---|---|
| Interpolation | RIFE v4.6 ONNX | MIT | https://github.com/hzwer/Practical-RIFE |
| Depth, for 3D effects | Depth Anything V2 Small | Apache-2.0 | https://huggingface.co/depth-anything/Depth-Anything-V2-Small |
| Face detection | YuNet ONNX | Apache-2.0 | https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet |
| Speech separation | Demucs v4 | MIT | https://github.com/facebookresearch/demucs |
| Denoise, audio | DeepFilterNet 3 | Apache-2.0 / MIT | https://github.com/Rikorose/DeepFilterNet |
| Upscaling, animation | RealESRGAN-x4plus-anime | BSD-3-Clause | Real-ESRGAN releases |
The earlier entries on this list have landed: Real-ESRGAN is ai-upscale,
SAM 2 is click-select and click-track, and Pocket TTS and Chatterbox are
installable speech engines.
Every model is fetched by the same path, so nothing above needs special code:
GET /api/models what is registered and what is on disk
GET /api/models/runtime the active execution provider and the available list
POST /api/models/{id}/download fetch it, as a job with progress
POST /api/models/{id}/import supply a file you already have
DELETE /api/models/{id} remove it
GET /api/models/speech speech models added at runtime
POST /api/models/speech add one
Three things are true of all of them:
-
Nothing is fetched without being asked. The one weight the engine
carries is the bundled U2NetP, inside the build itself, so the default
cutout works offline from the first run.
FORMCUT_MODELS_AUTOgoverns startup fetching:1(the default) fetches only the default cutout model into the model folder,allfetches every permissively licensed model for a machine that will go offline,0fetches nothing. -
Checksums are pinned.
PINNEDinmodel_manager.pyholds a SHA-256 for every registry file, taken from real fetches and matching the table in Licences; a mismatch deletes the download rather than caching it. Add a pin in the same change that adds a model. -
Sources are redirectable.
sources.jsonin the data directory lets a person point any model at a mirror or a local path, for machines that cannot reach GitHub or Hugging Face; the pin still applies, so a mirror cannot substitute a different file.
Krea 2 Turbo and FLUX.2 [klein] 4B are optional, installed from their cards in Studio > Local models and run by mflux in their own environment. Neither ships with formcut: their MLX builds come from mflux-community on Hugging Face, pinned file by file, when you install them. FLUX.2 [klein] 4B is Apache-2.0, commercial use allowed. The 9B Klein models are under the FLUX Non-Commercial License and are not offered.
LoRAs live in a folder per model: models/loras/krea-2/ and models/loras/flux2-klein-4b/. Drop .safetensors
files in and they appear in the generator's LoRA stack (Look again reads the folder afresh). A <name>.json beside a
file keeps its name, strength, trigger words and source. Import a file, or Add from a link:
- a Hugging Face file link,
https://huggingface.co/<repo>/resolve/<revision>/<file>.safetensors, for the model you have chosen; - a Civitai model, version or download link (
https://civitai.com/models/<id>,...?modelVersionId=<id>,https://civitai.com/api/download/models/<id>): formcut reads Civitai's record, puts the file in the folder of the model Civitai names (Krea 2, or Flux.2 Klein 4B and its base), refuses any other (Klein 9B, SDXL, Flux.1 ...), checks the download against Civitai's SHA-256 and keeps its trigger words. Civitai asks for an account to download most files: add your Civitai API key in Settings.
Prompts written as Civitai shows them work in the generator for Krea 2, Klein 4B and their checkpoints:
<lora:name:strength> tags anywhere in the prompt (the name is the LoRA's file name without .safetensors, found
the way Automatic1111 finds it, then its name in formcut or the file name it was downloaded as; a strength may be
negative, as sliders use), and a pasted picture's generation data: Negative prompt: and the Steps: ..., Seed: ..., Model: ..., width/height or Size: WxH line. Steps, Seed and size are used (the size rounded to a multiple of 16);
Model: chooses a checkpoint of that name in models/checkpoints/<model>/; CFG other than 1, the sampler, the
schedule, clip skip and a negative prompt are not taken by these distilled models, and the result says so. A tag whose
LoRA is not in the model's folder stops the picture and names the folder to put it in.
A prompt wrap is its own kind of run, beside Generate. With Krea 2 or one of its checkpoints chosen, the image
generator shows a Prompt wrap section under the settings: Load prompt wrap… takes a JSON file of three prompts
or more (examples/prompt-wrap.json shows one) and lists its entries as the engine read them; Run wrap runs them in
turn, and with Round and round ticked, from the first again after the last until stopped (unticking it lets the
picture in hand finish, then stops; Stop stops at once). Nothing of the settings above reaches a wrap: not the prompt
box (which may be empty), the size, steps, seed, sampler, CFG, negative prompt, LoRA stack or Start from picture. Each
entry brings its own; what an entry leaves out comes from the file's "defaults", then from the model's own (1024 × 1024,
its steps, its sampler, CFG off, a random seed). Generate, for its part, never runs the wrap.
The file is {"defaults": {...}, "prompts": [...]}, {"prompts": [...]} or the list alone; an entry, or the
defaults, may say {"prompt": "...", "width": 832, "height": 1248, "steps": 9, "seed": -1, "loras": [{"name": "Emotions V1", "scale": 1.0}], "sampler": "euler", "cfg": 1, "negative": "..."}, and a string entry is its prompt.
LoRAs are named as the library names them (its file names, as a <lora:> tag does, which a prompt may carry too, with
Civitai's generation data); scales run from -2 to 2, eight at most; sizes are made multiples of 16 from 256 to 2048.
Every entry passes the same prompt guard a single prompt does. The file is checked when it is loaded, and an entry that
cannot be read is named. The model is loaded once for the whole wrap, however its entries differ: formcut's mflux
worker (backend/app/mflux_worker.py, run in formcut's mflux environment) builds the model the way
mflux-generate-krea2 does, with every LoRA the wrap uses applied unbaked, then makes each picture with mflux's own
generate_image, its prompt, size, steps, seed, sampler and CFG, each LoRA at that entry's scale and the LoRAs it
does not use set aside. One load carries only as many LoRAs as the memory holds beside the model (four fifths of the
memory, less the model and 3 GB); a wrap naming more is split into as few loads as fit. Should the worker not run with
the installed mflux, the wrap goes on a setup at a time (a load where an entry's LoRAs, size or steps change) and says
so. (Until 2.3.3 every change of setup was a new load: with a LoRA taking turns, every entry, and the owner's run stalled
at loading and took the browser down.) Each
picture records which entry made it. The engine's endpoint is POST /api/generate/wrap-run {generator, wrap, continuous}.
What the image and video generator makes is kept apart from the media library: files in generative/, thumbnails in
generative-thumbs/, records (prompt, seed, steps, size, LoRAs, model, time) in generative.json, in the data folder.
The Studio's results show it, latest first, as a gallery: the picture fills its tile, its seed, steps, size and wrap
position show on hover with Add to media (a copy into the media library, for the timeline; the generation stays)
and Delete; a click opens it full screen with its prompt. Image-to-image can start from a generation. Audio the
generator makes (speech, music) still goes to the media library, as it is laid on the timeline at once.
mflux matches a LoRA's keys to its model by exact patterns, and trainers name the same weights differently. formcut
checks each LoRA in the library against the installed mflux's own mapping for its model (it runs in formcut's mflux
environment, so it is right for that mflux), once per file and mflux version, and shows the result under the LoRA's
row in the generator: how many keys apply, what was renamed, and what is left and why. Where renaming makes keys
apply, a copy with the renamed header (the tensors' bytes as they were) is written to .converted/ beside the file,
and mflux is given that copy. Renamed: OneTrainer's historical layout (lora_transformer_…, where mflux reads
lora_unet_…), LyCORIS's lycoris_… and flat LoKr names (read into the dotted names mflux's Krea 2 mapping knows),
and ComfyUI's model.diffusion_model.. Said, not fixable by renaming: text-encoder weights (mflux applies LoRAs to
the image model only), a DoRA's magnitude (mflux applies only its LoRA part, so it will not look as trained), LoHa
(hada_) and full-weight differences, which mflux does not load. A LoRA none of whose keys apply is marked unusable
with the reason (made for another model, a LoHa, text-encoder weights only), rather than stopping a run. OneTrainer's
DIFFUSERS, KOHYA, ORIGINAL and COMFY LoRA outputs all load as they are.
For Krea 2, Klein 4B and their checkpoints the image generator has a Steps slider (1 to 50, starting at the model's
own: 8 for Krea 2, 4 for Klein) and a Seed slider with its number beside it: -1 (Random) draws a new seed each run,
any other seed repeats a picture. The seed used is shown on each result and in the note, and written into the
picture's file as Automatic1111 writes it (the PNG's parameters: the prompt, the LoRAs as <lora:file:strength>
tags, then Steps, Sampler, CFG scale, Seed, Size, Model), beside mflux's own record; the library keeps the same.
Pasted back into the generator, that text asks for the same picture. With a random seed, Continuous starts the next
run as each one finishes, until it is unticked, cancelled, a run fails, or Studio is closed. With Krea 2 and its
checkpoints, Continuous is a session in the engine: one run of mflux over hundreds of random seeds, so the model and
its LoRAs are loaded once and stay in memory between pictures; each picture joins the results as soon as it is made
(through the content filter, with its recipe written into it); the prompt can be changed while it runs, and the next
picture reads it (mflux's --prompt-file), while the LoRAs, size and steps stay as the session started. Unticking
Continuous lets the picture in hand finish and be saved, then ends the session and frees the memory; Cancel stops at
once, and nothing starts again until Generate is pressed. Each result's time is in the note.
A result opens full screen with a click (or Enter): Escape, a click or Close returns; the arrow keys move through
the results. While a picture is made, the terminal formcut was started from shows mflux's own output as it comes,
its tqdm progress bar included (the steps, the time elapsed and left, its/s), redrawn in place and sized to the
terminal, between a line from formcut when the run starts and one when it ends; FORMCUT_QUIET_PROGRESS=1 keeps it
out. Under the generator's progress bar, the same reading as tqdm gives it: the step and [elapsed<left, rate], or
while the model loads, how long loading has taken.
If Krea 2's content filter was removed, its warning goes with each picture until it is acknowledged (I understand) on the Krea 2 card in Studio > Local models; after that the pictures no longer carry it.
mflux-generate-krea2 as mflux 0.20.0 parses it: --model the build's or checkpoint's folder (no --base-model,
which the Krea 2 command does not need); --prompt; --width/--height in multiples of 16; --seed with every
seed of a request, so several pictures come from one load of the model; --steps (8 unless asked); --scheduler
er_sde (the default) or euler, the generator's Sampler (a Civitai "Euler ..." sampler becomes euler; mflux has
no ancestral Euler); --guidance only when a CFG scale other than 1 is asked for (it turns CFG on, a second pass a
step, slower), and then the negative prompt with --negative-prompt; each LoRA as --lora PATH SCALE, with
--no-bake-lora for a 4-bit model (the 4-bit build, or a checkpoint whose folder name says q4 or 4bit) or when
memory is short, as baking would turn a 4-bit model's LoRA layers to 8-bit; and for memory --low-ram (which tiles
the decode), --mlx-cache-limit-gb, or --vae-tiling alone when the model fits but its untiled decode (about 7 GB
at 1024 px) would not. NVIDIA PiD decoding (--pid-decode) is not used.
Image-to-image (--image PATH STRENGTH): Krea 2 and its checkpoints can start from a picture. In the image
generator, Start from takes the frame at the playhead (rendered from the project as it stands, as an export would draw
it) or an image or video in the media library (a video at the playhead's time); Image influence runs from 0 (the
picture has no say) to 1, mflux's default 0.4; the size follows the source (Match the source picture, fitted within
2048 pixels, in multiples of 16) unless another is chosen. A pasted Civitai prompt's Denoising strength is read as
1 minus the influence (A1111's denoising is how much is repainted). FLUX.2 [klein] 4B does not take a source here.
A Krea 2 or Klein 4B fine-tune runs as a model of its own once it is in mflux's MLX format, in a folder under
models/checkpoints/krea-2/<name>/ or models/checkpoints/flux2-klein-4b/<name>/. mflux's format is the folder
mflux-save writes: text_encoder/, tokenizer/, vae/, and the transformer's weights in transformer/ (Klein) or
at the top (Krea 2). Studio > Local models lists what it found, and what a folder lacks. To make one from a fine-tune
published in the diffusers layout (a Hugging Face repository or a folder with transformer/, text_encoder/, ...),
use the mflux formcut installs:
models/krea-2/runtime/bin/mflux-save --model <repo or folder> --base-model krea-2 \
--path models/checkpoints/krea-2/<name> --quantize 8
(--base-model flux2-klein-4b for Klein; --lora-paths and --lora-scales bake LoRAs in.) A single-file checkpoint
as Civitai serves it for ComfyUI or Automatic1111 has to be put into the diffusers layout first; formcut does not do
that conversion. A Krea 2 fine-tune is under Krea 2's licence, so it runs once that licence is accepted on the Krea 2
card, with the content filter as Krea 2 has it.
Every file is checked from its header: LoRA and LyCORIS LoKr load; DoRA in the PEFT layout does not (mflux refuses it); a file that fails is listed with its reason and cannot be stacked. In the stack, each LoRA has a strength from -2 to 2 and a switch; up to eight at once.
Memory: Automatic runs mflux as it runs from the command line, tiling the decode (--vae-tiling) when the model
and an untiled decode would not fit beside each other, and uses --low-ram with a capped MLX cache only when the
model itself would not fit (both cost a great deal of time); Low memory always uses them (slower); Normal uses none.
Before a model runs, the preview's cached frames are let go (drawn again when asked for), and one local image model
runs at a time. The terminal's first line for a run is the exact mflux command, to set beside mflux run by hand. The generator shows the model's own progress (loading, each step, finishing), and
Cancel ends the model's process.
Pages
- Home
- Architecture
- Creator workflow
- Automation
- Extensions
- Packaging
- Translation
- Mobile notes
- Models
- Licences
In the repository
These pages are published from .docs/wiki/ in the repository. Edit them there;
a push to main updates this wiki. Editing a page here is overwritten by the
next publish.