Skip to content

feat(fp8): enable FP8 storage for Anima - #9415

Open
Pfannkuchensack wants to merge 4 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/fp8_anima
Open

feat(fp8): enable FP8 storage for Anima#9415
Pfannkuchensack wants to merge 4 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/fp8_anima

Conversation

@Pfannkuchensack

Copy link
Copy Markdown
Collaborator

Summary

Stacked on # — depends on the extra_skip_patterns mechanism introduced there. Merge that one first.

The fp8_storage toggle was shown for Anima main models but did nothing: AnimaCheckpointModel never called _apply_fp8_layerwise_casting. This wires it in — the state dict is already cast to a single model_dtype before load_state_dict, so the layerwise cast has one unambiguous compute dtype to restore to.

Wiring alone renders a heavily dithered image with no fine detail at all. The cause is t_embedder: it produces the adaln_lora conditioning consumed by every block, so casting it to FP8 corrupts every token everywhere. None of the generic skip patterns reach it — they target diffusers' module names (norm, pos_embed, patch_embed, proj_in/out) and this architecture names the equivalent modules differently (x_embedder, final_layer, adaln_modulation_*).

AnimaTransformer now declares _skip_layerwise_casting_patterns, the same attribute diffusers models use, so the loader needs no special-casing.

Note this is the same module as Z-Image's t_embedder, broken through a different mechanism: there diffusers read weight.dtype and cast the input to float8; here the plain precision loss is enough.

Related Issues / Discussions

Follow-up to #8945 (FP8 storage), #9231 (hook-based casting) and the Z-Image PR this is stacked on.

QA Instructions

Needs a CUDA GPU. Model Manager → an Anima main model → Default Settings → enable FP8 Storage → Save.

  1. Generate. Expect in the log:

    FP8 layerwise casting enabled for anima-preview (storage=float8_e4m3fn, compute=torch.bfloat16, param_size=2012MB)
    

    and the transformer resident at ~2012MB instead of 3988MB. On main there is no FP8 line at all — that is the dead toggle this PR fixes.

  2. Quality vs. bf16. Note a fixed seed, then run once with FP8 on and once off. Two gotchas that will otherwise give you a false result:

    • Disable the invocation cache (PUT /api/v1/app/invocation_cache/disable), or the second run returns the first run's image unchanged and the two look pixel-identical.
    • Anima needs 35 steps / CFG 4.5; at 9 steps / CFG 1.0 both runs look mushy and the comparison says nothing.

    Expect the same composition with slightly coarser fine structure under FP8 — not a different image, and definitely not a dithered mess. A dithered result means the skip patterns are not being applied.

  3. Regression: with FP8 off, output must be unchanged from before this PR.

Measured during development, same seed/steps/CFG each run — this is what pins the skip list down:

skip list param_size result
none 1994 MB dithered, no fine detail ❌
t_embedder 2010 MB clean ✅
+ x_embedder, final_layer 2012 MB clean ✅
+ adaln_modulation 2180 MB clean ✅

So t_embedder is necessary and sufficient. The two I/O layers are kept as ~2MB of margin, matching what diffusers skips by default for comparable DiTs. adaln_modulation is deliberately not listed — it costs 168MB and made no difference.

Unit tests:

uv run --extra cuda --extra test pytest tests/backend/model_manager/load/test_load_default_fp8.py tests/backend/anima -q --no-cov

Merge Plan

Merge after the Z-Image PR — _apply_fp8_to_nn_module(..., extra_skip_patterns=...) does not exist without it. No DB schema, no redux slice, no API schema change otherwise.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration — n/a, no redux changes
  • Documentation added / updated (if applicable) — n/a
  • Updated What's New copy (if doing a release after this PR) — n/a

Z-Image was excluded from FP8 storage in invoke-ai#8945 because diffusers'
enable_layerwise_casting() was called with the global torch dtype (fp16) while
Z-Image loads in bf16: skipped modules stayed bf16, hooked ones produced fp16,
and attention crashed. That root cause was fixed later in the same PR — the
compute dtype now comes from the model's own parameters — so the exclusion is
obsolete.

Removing it alone is not enough. Our hook-based cast (invoke-ai#9231) dropped one thing
diffusers' enable_layerwise_casting() did: honoring the model's declared
_skip_layerwise_casting_patterns. Z-Image needs it, and not for quality —
TimestepEmbedder.forward reads self.mlp[0].weight.dtype and casts its *input*
to it. With an fp8 weight the input becomes float8 before our pre-hook restores
the weight, and F.linear dies with:

    RuntimeError: "addmm_cuda" not implemented for 'Float8_e4m3fn'

which is why ZImageTransformer2DModel declares ['t_embedder', 'cap_embedder'].
_apply_fp8_to_nn_module now takes extra_skip_patterns and the caller passes the
model's list. For other models this is a strict superset of our defaults
(FLUX/SD3 pos_embed+norm, UNet norm, CogView4 also proj_out), so it only ever
skips more.

Also wire the cast into ZImageCheckpointModel: only the diffusers loader called
it, so the toggle was a silent no-op for single-file Z-Image models even though
both paths build the same ZImageTransformer2DModel.

Tested end to end on CUDA: transformer resident VRAM drops from ~11.5GB to
5880MB for both Z-Image-Turbo (diffusers) and Z-Image-Turbo
(checkpoint, 14.37GB file), with clean output images in both cases.
The fp8_storage toggle was shown for Anima main models but did nothing:
AnimaCheckpointModel never called _apply_fp8_layerwise_casting. Wire it in — the
state dict is cast to a single model_dtype before load_state_dict, so the
layerwise cast has one unambiguous compute dtype to restore to.

Wiring alone renders a heavily dithered image with no fine detail. The cause is
t_embedder: it produces the adaln_lora conditioning consumed by every block, so
casting it to FP8 corrupts every token everywhere. None of the generic skip
patterns match it — they target diffusers' module names (norm, pos_embed,
patch_embed, proj_in/out) and this architecture names things differently.

AnimaTransformer now declares _skip_layerwise_casting_patterns, the same
attribute diffusers models use, so the loader needs no special-casing.

Measured on CUDA, same seed/steps/CFG each run: casting nothing = broken at
1994MB; t_embedder alone = clean at 2010MB; adding x_embedder and final_layer
changes nothing further (2012MB) and is kept as margin on the I/O layers;
adaln_modulation was tested too and is deliberately not listed — it costs 168MB
and made no difference. Against a bf16 reference (3988MB) the FP8 result keeps
the same composition and loses only a little micro-detail.
@github-actions github-actions Bot added python PRs that change python files backend PRs that change backend files frontend PRs that change frontend files python-tests PRs that change python tests labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files frontend PRs that change frontend files python PRs that change python files python-tests PRs that change python tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant