Text-to-video and image-to-video that adapts to the machine it is on — from a 24 GB workstation down to an 8 GB consumer GPU, and ultimately to CPU-only.
Built on diffusers with Wan 2.1 / 2.2, LTX-Video and LTX-2.3.
Running these models is rarely a question of calling the right pipeline. The same model needs a different execution strategy on different hardware — different dtype, different weight format, different component placement, different decode path. Most guidance online assumes a 24 GB Ampere-or-newer card and silently misfires elsewhere:
bfloat16is wrong on an RTX 20-series card. Turing has no bf16 tensor cores, so every upstream example's dtype falls back to unaccelerated kernels.torch.cuda.is_bf16_supported()returnsTrueanyway. videogen uses fp16 there.float16is wrong on a typical x86 CPU. Without AVX-512/AMX there is no 16-bit compute path, so 16-bit only adds conversions. videogen computes in fp32 and saves memory on the weights instead.- FlashAttention-2 requires sm_80+. Having it installed does not mean it can run.
- The text encoder is often the real bottleneck. UMT5-XXL is 11.4 GB at fp16 — larger than an entire RTX 2070. Encoding on the CPU once and caching the embedding is most of what makes Wan 2.2 fit in 8 GB.
videogen makes each of those a recorded, overridable decision instead of a footgun.
Measured in the CPU container against real weights, not taken from documentation:
| Finding | Consequence |
|---|---|
diffusers keeps GGUF weights quantised. After loading Wan 2.1-1.3B Q4_K_M, 300 tensors were still GGUFParameter (uint8), dequantised per forward pass. |
A 4.2 GB Q6_K denoiser costs 4.2 GB of memory. Peak RSS for a full run was 3.5 GiB. |
| transformers dequantises GGUF at load. | A 3.4 GB Q4 UMT5-XXL still costs ~10 GiB at fp16. Quantised text encoders are a download saving only — planning against file size would promise a fit that then OOMs. |
AutoencoderKLWan does support enable_tiling()/enable_slicing(), contrary to the diffusers memory guide. |
Tiled decode is a real rung on the fallback ladder. |
fp8 (float8_e4m3fn) works as pure storage on CPU, with no fp8 hardware. |
Viable weight-halving fallback for models with no GGUF release — and it works on Turing. |
Docker Desktop's VM on this host exposes avx/fma/f16c but not avx2. |
CPU throughput measured there (~8 GFLOPS/core) is a floor; native Linux on the same silicon should be ~2× faster. |
hf-xet dies with SIGILL on that VM — no traceback, process just vanishes mid-download. |
HF_HUB_DISABLE_XET=1 is set in the CPU image. |
| Measured 56.2 s/step (Wan 2.1-1.3B, 768 tokens, 4 container CPUs) vs a 3×-optimistic first estimate. | The built-in FLOP model was recalibrated; videogen bench overrides it per machine. |
| A quantised file is not always a quantised model. A GGUF denoiser stays compressed in memory; an fp8 safetensors checkpoint is loaded at the compute dtype, so it saves download size only. | On CPU, Wan 2.2 5B runs while LTX 2B does not — the GGUF weights stay at 4.2 GB while LTX's 1.9B parameters cost 7.6 GB at fp32. Parameter count is a poor guide; the planner sizes by resident bytes. |
LTX bundles the transformer and VAE in one checkpoint, and returns packed, normalised latents from output_type="latent" (Wan does not). |
The planner picks a VAE variant sharing the transformer's file, and the backend unpacks and denormalises before decoding — otherwise the VAE gets garbage. |
Image-to-video pipelines need the VAE during denoising. WanImageToVideoPipeline sizes its latents from self.vae.config.z_dim; the text-to-video pipeline never touches it. |
Passing vae=None — correct and worth 1.5–2.8 GiB for t2v — crashes i2v after the download and pipeline build. The VAE is now loaded up front when needed, shared with the decode stage, and counted in the memory estimate. |
Wan 2.2 TI2V-5B Q3_K_M dies with SIGILL on the AVX2-less Docker VM. The file is byte-exact, 1.3B Q3_K_M loads fine here, and 5B Q4_K_M works — so it is specific to that one combination. |
Prefer --variant q4_k_m or higher for the 5B on CPU. Unexplained; not reproduced on a native machine. |
ram_available must not be cached. A cached probe kept refusing work after Docker was given more memory, and a leftover container silently ate 6 GiB. |
Only the static hardware description is cached; free RAM, free disk and free VRAM are re-read on every use. |
| Freeing a model on CPU does not lower RSS. glibc keeps the freed arenas, so the next component is allocated on top of memory that is free but not returned. | free_memory() calls malloc_trim(0) on Linux. Staged execution bounds allocation, not resident set, without it. |
LTX's single-file checkpoints are version-specific. Loading 0.9.8 weights against the repo-root 0.9.1 config fails: decoder.conv_in is 512 wide there and 1024 in 0.9.8. |
Config is inferred from the checkpoint for safetensors, and only passed explicitly for GGUF (which carries no diffusers metadata). |
| A CPU plan needs headroom. A run predicted at 9.7 GiB against 10.6 GiB available was OOM-killed anyway — loading transiently holds both the on-disk buffer and the materialised tensors. | CPU jobs are planned into 85% of available RAM, turning a late kill into an up-front refusal. |
Working and verified. With real Wan weights on CPU, both modes end to end: hardware probe, planner, model registry, disk-aware downloads, staged execution, GGUF loading, chunked VAE decode, H.264 encoding, and the measurement feedback loop.
- text-to-video, Wan 2.1-1.3B, 256×256×9 — ~4 min
- image-to-video, Wan 2.2 TI2V-5B Q4_K_M, 256×256×9 — ~12 min, 9 distinct frames
Both recorded their own s/step, so subsequent plans for the same workload report
(measured) rather than an estimate. With the mock backend: the SQLite job queue, REST API,
SSE progress, web UI, both generation modes, and the image-upload endpoint.
144 tests pass where torch and diffusers are installed; 118 on a machine with neither, which is the point — the service layers do not depend on them.
Partly verified — LTX-Video. Against real weights, the download, single-file
loading of transformer and VAE, pipeline construction and the denoise loop all
work (2 steps at 83.8 s/step on CPU). Decode and encode are not verified: LTX 2B
needs 7.6 GiB for weights alone at fp32, and on a 13 GiB Docker VM the run dies
somewhere between denoise and decode — OOM on one attempt, SIGILL on the next, which
is what a machine at its limit looks like. Adding malloc_trim (glibc keeps freed
arenas, so dropping the transformer did not lower RSS) helped but was not enough.
It should work on the RTX 2070, where fp16 halves the weights to 3.8 GiB. The other
route is a GGUF release: city96 publishes LTX-Video 0.9.6 GGUFs (~1.4 GiB at
Q4_K_M) which would stay compressed in memory the way Wan's do. There is no 0.9.8
GGUF yet.
Implemented but unverified on hardware: the CUDA paths — fp16-on-Turing, group
offload, VRAM accounting, and the Wan2.2-Turbo LoRA. These need the RTX 2070. Run
videogen doctor, then videogen bench, then a --preset draft generation there.
Written but never executed — LTX-2.3. The spec is real (verified sizes and
configs) and the call contract is checked against the live diffusers signature, but
no frame has been generated: it needs the ltx2 package, which is not on PyPI,
and ~64 GiB of weights for a model wanting 24 GB of VRAM. videogen doctor reports
the missing runtime rather than failing after the download. Treat its first real run
as bring-up.
Resumable long jobs. Denoising latents are checkpointed every few steps, so a crash, an OOM kill or a restart resumes rather than starting over — which matters when a CPU render is measured in tens of minutes. A checkpoint is discarded if anything that would change the result differs (model, variant, dtype, geometry, step count, seed), because resuming onto a different computation produces a plausible but wrong video. Disabled below 8 steps, where it would cost more than it saves.
GET /api/hardware probe results, incl. what is *actually* usable
GET /api/models registered models + whether each fits this machine
POST /api/plan preview an execution plan without queueing
POST /api/jobs enqueue (422 + suggestions if it cannot be satisfied)
POST /api/jobs/upload enqueue an image-to-video job with a file
GET /api/jobs queue listing + counts
GET /api/jobs/{id} one job, with progress and observed ETA
GET /api/jobs/{id}/events SSE progress stream, closes on terminal state
GET /api/jobs/{id}/video the finished MP4
DELETE /api/jobs/{id} cancel (pending: immediate; running: clean unwind)
Impossible requests are rejected at submit time with concrete suggestions, rather than failing twenty minutes into a job.
python3 -m venv .venv && source .venv/bin/activate
pip install -e . # core: CLI, planner, registry, mock backend
pip install -e '.[cuda]' # add the real inference backendsvideogen doctor # what this machine can run, and why
videogen models ls # registered models and install sizes
videogen models pull wan22 # fetch weights ahead of the first render
videogen models gc # show reclaimable disk (add --yes to delete)
videogen plan "a fox in snow" # the full execution plan, without running it
videogen t2v "a fox in snow" -o fox.mp4
videogen i2v photo.jpg "slow pan across the valley"
videogen bench # measure this machine's real s/step
videogen serve # REST API + web UI on :8000videogen doctor is the first thing to run on any new machine. It reports the GPU
architecture, whether bf16/fp8/flash-attn are actually usable, and the plan each
registered model would receive.
| Tier | Runs on | Models | Install |
|---|---|---|---|
| 0 | anything, including CPU-only | Wan 2.1 T2V-1.3B (t2v) | 4.8 GiB |
| 0 | anything | LTX-Video 2B 0.9.8 distilled (t2v + i2v) | 6.3 GiB |
| 1 | 8 GB VRAM | Wan 2.2 TI2V-5B (t2v + i2v) | 8.4 GiB |
| 2 | 16 GB+ | LTX-Video 13B 0.9.8 distilled (t2v + i2v) | 17.3 GiB |
| 3 | 24 GB+ or cloud | LTX-2.3 22B (video + synchronised audio, up to 4K) | ~64 GiB |
Two defaults worth knowing:
- Wan 2.2 TI2V-5B is the quality target for an 8 GB card — one dense 5B model doing both modes, with GGUF quantisations from 3.4 GB and a 4-step Turbo LoRA.
- LTX-Video 2B distilled is the fast one. It is distilled out of the box (8 steps, no CFG) and its VAE compresses 8× temporally against Wan's 4×, so it costs about half the tokens of Wan 2.2 for the same output. Note the two models' VAEs are equally aggressive spatially once Wan's 2×2 patchification is counted — the whole advantage is on the time axis, which is not what the headline VAE ratios suggest.
LTX ships all-in-one checkpoints where the denoiser and VAE are the same file, so the planner picks a VAE variant that shares the transformer's file — otherwise the download silently doubles.
The core library deliberately does not depend on torch, so the CLI, planner, job
queue and web UI can be developed on machines where torch cannot be installed (Intel
macOS caps out at torch 2.2.2). The mock backend implements the same protocol as the
real ones and produces a real, playable MP4:
videogen t2v "test" --backend mock -o /tmp/test.mp4
pytestFor CPU-tier inference on such a machine, use the linux/amd64 container in docker/,
where current CPU torch wheels are available.
Apache-2.0. Model weights carry their own licenses — Wan is Apache-2.0, LTX-Video is
OpenRAIL-M, LTX-2.3 is under the LTX-2 Community License. videogen models show <id>
prints the license for each.