Skip to content

mage-flow: add FFN checkpointing logic - #2907

Merged
bghira merged 6 commits into
feature/ffn-checkpointing-modefrom
feature/mageflow-ffn-checkpointing
Jul 27, 2026
Merged

mage-flow: add FFN checkpointing logic#2907
bghira merged 6 commits into
feature/ffn-checkpointing-modefrom
feature/mageflow-ffn-checkpointing

Conversation

@bghira

@bghira bghira commented Jul 27, 2026

Copy link
Copy Markdown
Owner

This pull request introduces several improvements and new tests for the MageFlow Transformer model, focusing on enhanced gradient checkpointing flexibility, support for feed-forward network (FFN) checkpointing, and increased test coverage for attention and packing logic.

MageFlow Transformer Model Enhancements

  • Added support for FFN-specific gradient checkpointing by introducing the _supports_ffn_gradient_checkpointing flag and handling a new gradient_checkpointing_scope parameter. This allows selective checkpointing of feed-forward layers, improving memory efficiency during training. [1] [2] [3] [4]
  • Updated imports and initialization to include get_checkpoint_backend_scope, ensuring the correct checkpointing scope is used throughout the model. [1] [2] [3]

Test Coverage Improvements

  • Added a test (test_mageflow_attention_joint_pack_uses_single_flash_dtype) to verify that the MageFlow attention processor correctly enforces a single dtype (bfloat16) for all projections and outputs, ensuring consistency and preventing dtype mismatches in flash attention. [1] [2]
  • Added a test (test_pack_latents_tracks_rectangular_target_shape) to validate that the model's latent packing logic correctly tracks and returns the expected shapes, indices, and lengths for rectangular targets, increasing reliability of input handling.

@bghira
bghira changed the base branch from main to feature/ffn-checkpointing-mode July 27, 2026 02:18
@bghira
bghira merged commit d0d954e into feature/ffn-checkpointing-mode Jul 27, 2026
@bghira
bghira deleted the feature/mageflow-ffn-checkpointing branch July 27, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant