Skip to content

[Offloading] [Bugfix] Fix fully offloaded model saving - #47336

Merged
SunMarc merged 6 commits into
huggingface:mainfrom
kylesayrs:kylesayrs/fix-disk-offloading-saving
Jul 23, 2026
Merged

[Offloading] [Bugfix] Fix fully offloaded model saving#47336
SunMarc merged 6 commits into
huggingface:mainfrom
kylesayrs:kylesayrs/fix-disk-offloading-saving

Conversation

@kylesayrs

@kylesayrs kylesayrs commented Jul 15, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

CI

Purpose

  • Fix full model saving for fully-disk-offloaded models

Changes

Testing

Suggested Reviewers

kylesayrs added 2 commits July 15, 2026 03:13
@kylesayrs
fix
5c5eb9a 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@kylesayrs
remove breakpoint
9ad3c80 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@HDCharles

HDCharles commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

looks good

This was referenced Jul 15, 2026
Merged
Merged
@Rocketknight1 Rocketknight1 mentioned this pull request Jul 16, 2026
Closed
4 tasks
@kylesayrs kylesayrs changed the title [Offloading] Fix fully offloaded model saving [Offloading] [Bugfix] Fix fully offloaded model saving Jul 18, 2026
@kylesayrs

kylesayrs commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@Cyrilvallez PTAL

SunMarc
SunMarc reviewed Jul 21, 2026
View reviewed changes

@SunMarc SunMarc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! Left a comment

Comment thread src/transformers/modeling_utils.py Outdated
Comment on lines +3447 to +3448
if hasattr(self, "hf_device_map") and (
len(set(self.hf_device_map.values())) > 1 or "disk" in self.hf_device_map.values()

@SunMarc SunMarc Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure about this ? this will always be true in multi-gpu setup. len(set(self.hf_device_map.values())) > 1

@kylesayrs kylesayrs Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this condition is meant to be equivalent with this accelerate logic, which signifies when hooks are forced on the model.

Accelerate couples the concepts of "attaching hooks" and "offloading the weights".

@SunMarc SunMarc Jul 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed, can you fix this condition or revert the changes here ?

kylesayrs added a commit to vllm-project/llm-compressor that referenced this pull request Jul 22, 2026
@kylesayrs
[Example] [MoE] HY_V3 (#2928)
42a1b83 
## Purpose ##
* Support `HY_V3` model

## Prerequisites ##
### Transformers ###
- [ ] huggingface/transformers#47336

## Testing ##
* Used to generate https://huggingface.co/RedHatAI/Hy3-NVFP4-FP8

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@kylesayrs

kylesayrs commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Tested locally

@pytest.mark.parametrize("device_map", [
    {'model.visual': 0, 'model.language_model': 1, 'lm_head': 1},
    {"model.visual":0,"model.language_model":"cpu" ,"lm_head": "cpu"},
    {"model.visual":0,"model.language_model":"disk" ,"lm_head": "disk"},
    {'model.visual': 0, 'model.language_model': 0, 'lm_head': 0},
    {"model.visual":"cpu","model.language_model":"cpu" ,"lm_head": "cpu"},
    {"model.visual":"disk","model.language_model":"disk" ,"lm_head": "disk"},
])
def test_save(device_map):
    model = Qwen3VLMoeForConditionalGeneration.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct",device_map=device_map, offload_folder="offload_folder")
    model.save_pretrained("tmp")

The full disk offloading case fails without the current changes, which means these changes are effective and working

SunMarc
SunMarc approved these changes Jul 23, 2026
View reviewed changes

@SunMarc SunMarc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@SunMarc

SunMarc commented Jul 23, 2026

Copy link
Copy Markdown
Member

@bot /style

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Style fix is beginning .... View the workflow run here.

@SunMarc
SunMarc enabled auto-merge July 23, 2026 16:18
@kylesayrs
fix format
d5b2e71 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
auto-merge was automatically disabled July 23, 2026 16:21

Head branch was pushed to by a user without write access

@SunMarc
SunMarc enabled auto-merge July 23, 2026 16:27
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Jul 23, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 30017737207:2
Result: success | Jobs: 2 | Tests: 22 | Failures: 1 | Duration: 1m 56s

@SunMarc
SunMarc added this pull request to the merge queue Jul 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 23, 2026
@SunMarc
SunMarc added this pull request to the merge queue Jul 23, 2026
Merged via the queue into huggingface:main with commit e9c987a Jul 23, 2026
114 checks passed
stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026
@kylesayrs @stevhliu
[Offloading] [Bugfix] Fix fully offloaded model saving (huggingface#4…
b3a47eb 
…7336)

* fix

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* remove breakpoint

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* fix offloading condition

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* fix format

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Sainava pushed a commit to Sainava/Sai-transformers that referenced this pull request Aug 3, 2026
@kylesayrs @Sainava
[Offloading] [Bugfix] Fix fully offloaded model saving (huggingface#4…
389c945 
…7336)

* fix

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* remove breakpoint

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* fix offloading condition

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* fix format

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SunMarc SunMarc SunMarc approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Offloading] Cannot save disk-offloaded Qwen/Qwen3-30B-A3B

4 participants

@kylesayrs @HDCharles @SunMarc @HuggingFaceDocBuilderDev