add torch-ffn and unsloth-ffn gradient checkpointing modes for speed-up - #2911
Merged
Conversation
mage-flow: add FFN checkpointing logic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request expands and clarifies the documentation for the
--gradient_checkpointing_backendoption across all supported languages, and introduces new experimental guides explaining the tradeoffs and usage of Unsloth-style checkpointing. The main changes include adding new backend options, updating descriptions to reflect these, and providing detailed, language-specific guides for advanced checkpointing strategies.Documentation improvements:
--gradient_checkpointing_backendoption now documents four choices:torch,torch-ffn,unsloth, andunsloth-ffn, with clear explanations of each backend's behavior, requirements, and supported model scopes. [1] [2] [3] [4] [5] [6]Experimental guides:
Internationalization:
--gradient_checkpointing_backendoption and the new guides are provided in English, Spanish, Hindi, Japanese, Portuguese, and Chinese, ensuring consistent and clear guidance for users in all supported languages. [1] [2] [3] [4] [5] [6] [7] [8] [9]These updates ensure that users understand the available checkpointing backends, their tradeoffs, and how to apply them effectively in different training scenarios.