fix: preserve torch_dtype in get_engine_config and add it to CodeFormulaV2#3117
Merged
cau-git merged 2 commits intodocling-project:mainfrom Mar 17, 2026
Merged
Conversation
…ulaV2 get_engine_config() was silently dropping torch_dtype from engine overrides when constructing EngineModelConfig. This caused CodeFormulaV2 to load in fp32 while using Flash Attention 2, which only supports fp16/bf16. Also adds torch_dtype="bfloat16" to the CodeFormulaV2 preset's TRANSFORMERS engine override. Fixes docling-project#3026 Signed-off-by: majiayu000 <1835304752@qq.com>
Contributor
|
✅ DCO Check Passed Thanks @majiayu000, all your commits are properly signed off. 🎉 |
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
dolfim-ibm
requested changes
Mar 13, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Signed-off-by: majiayu000 <1835304752@qq.com>
Contributor
Author
|
Hi @dolfim-ibm, done! I've moved |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3026
The CodeFormulaV2 preset was missing
torch_dtypein its TRANSFORMERS engine override, causing the model to load in fp32 which is incompatible with Flash Attention 2 (requires fp16/bf16). Additionally,get_engine_config()was not preserving thetorch_dtypefield from engine overrides, so even presets that specified it would lose the value.Changes
torch_dtype="bfloat16"toCodeFormulaV2preset'sTRANSFORMERSengine override instage_model_specs.pyget_engine_config()to preserve and pass throughtorch_dtypefrom engine overrides toEngineModelConfigtest_get_engine_config_preserves_torch_dtypeto verify torch_dtype propagationTest plan
pre-commit run --all-filespasses (Ruff + MyPy)uv run pytest tests/test_vlm_presets_and_runtime_options.py— all 32 tests pass