feat(fp8): enable FP8 storage for Anima - #9415
Open
Pfannkuchensack wants to merge 4 commits into
Open
Conversation
Z-Image was excluded from FP8 storage in invoke-ai#8945 because diffusers' enable_layerwise_casting() was called with the global torch dtype (fp16) while Z-Image loads in bf16: skipped modules stayed bf16, hooked ones produced fp16, and attention crashed. That root cause was fixed later in the same PR — the compute dtype now comes from the model's own parameters — so the exclusion is obsolete. Removing it alone is not enough. Our hook-based cast (invoke-ai#9231) dropped one thing diffusers' enable_layerwise_casting() did: honoring the model's declared _skip_layerwise_casting_patterns. Z-Image needs it, and not for quality — TimestepEmbedder.forward reads self.mlp[0].weight.dtype and casts its *input* to it. With an fp8 weight the input becomes float8 before our pre-hook restores the weight, and F.linear dies with: RuntimeError: "addmm_cuda" not implemented for 'Float8_e4m3fn' which is why ZImageTransformer2DModel declares ['t_embedder', 'cap_embedder']. _apply_fp8_to_nn_module now takes extra_skip_patterns and the caller passes the model's list. For other models this is a strict superset of our defaults (FLUX/SD3 pos_embed+norm, UNet norm, CogView4 also proj_out), so it only ever skips more. Also wire the cast into ZImageCheckpointModel: only the diffusers loader called it, so the toggle was a silent no-op for single-file Z-Image models even though both paths build the same ZImageTransformer2DModel. Tested end to end on CUDA: transformer resident VRAM drops from ~11.5GB to 5880MB for both Z-Image-Turbo (diffusers) and Z-Image-Turbo (checkpoint, 14.37GB file), with clean output images in both cases.
The fp8_storage toggle was shown for Anima main models but did nothing: AnimaCheckpointModel never called _apply_fp8_layerwise_casting. Wire it in — the state dict is cast to a single model_dtype before load_state_dict, so the layerwise cast has one unambiguous compute dtype to restore to. Wiring alone renders a heavily dithered image with no fine detail. The cause is t_embedder: it produces the adaln_lora conditioning consumed by every block, so casting it to FP8 corrupts every token everywhere. None of the generic skip patterns match it — they target diffusers' module names (norm, pos_embed, patch_embed, proj_in/out) and this architecture names things differently. AnimaTransformer now declares _skip_layerwise_casting_patterns, the same attribute diffusers models use, so the loader needs no special-casing. Measured on CUDA, same seed/steps/CFG each run: casting nothing = broken at 1994MB; t_embedder alone = clean at 2010MB; adding x_embedder and final_layer changes nothing further (2012MB) and is kept as margin on the I/O layers; adaln_modulation was tested too and is deliberately not listed — it costs 168MB and made no difference. Against a bf16 reference (3988MB) the FP8 result keeps the same composition and loses only a little micro-detail.
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant,
dunkeroni and
lstein
as code owners
July 31, 2026 08:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
fp8_storagetoggle was shown for Anima main models but did nothing:AnimaCheckpointModelnever called_apply_fp8_layerwise_casting. This wires it in — the state dict is already cast to a singlemodel_dtypebeforeload_state_dict, so the layerwise cast has one unambiguous compute dtype to restore to.Wiring alone renders a heavily dithered image with no fine detail at all. The cause is
t_embedder: it produces theadaln_loraconditioning consumed by every block, so casting it to FP8 corrupts every token everywhere. None of the generic skip patterns reach it — they target diffusers' module names (norm,pos_embed,patch_embed,proj_in/out) and this architecture names the equivalent modules differently (x_embedder,final_layer,adaln_modulation_*).AnimaTransformernow declares_skip_layerwise_casting_patterns, the same attribute diffusers models use, so the loader needs no special-casing.Note this is the same module as Z-Image's
t_embedder, broken through a different mechanism: there diffusers readweight.dtypeand cast the input to float8; here the plain precision loss is enough.Related Issues / Discussions
Follow-up to #8945 (FP8 storage), #9231 (hook-based casting) and the Z-Image PR this is stacked on.
QA Instructions
Needs a CUDA GPU. Model Manager → an Anima main model → Default Settings → enable FP8 Storage → Save.
Generate. Expect in the log:
and the transformer resident at ~2012MB instead of 3988MB. On
mainthere is no FP8 line at all — that is the dead toggle this PR fixes.Quality vs. bf16. Note a fixed seed, then run once with FP8 on and once off. Two gotchas that will otherwise give you a false result:
PUT /api/v1/app/invocation_cache/disable), or the second run returns the first run's image unchanged and the two look pixel-identical.Expect the same composition with slightly coarser fine structure under FP8 — not a different image, and definitely not a dithered mess. A dithered result means the skip patterns are not being applied.
Regression: with FP8 off, output must be unchanged from before this PR.
Measured during development, same seed/steps/CFG each run — this is what pins the skip list down:
t_embedderx_embedder,final_layeradaln_modulationSo
t_embedderis necessary and sufficient. The two I/O layers are kept as ~2MB of margin, matching what diffusers skips by default for comparable DiTs.adaln_modulationis deliberately not listed — it costs 168MB and made no difference.Unit tests:
Merge Plan
Merge after the Z-Image PR —
_apply_fp8_to_nn_module(..., extra_skip_patterns=...)does not exist without it. No DB schema, no redux slice, no API schema change otherwise.Checklist
What's Newcopy (if doing a release after this PR) — n/a