Flip default on weights_only #137602

mikaylagawarecki · 2024-10-09T16:16:17Z

Stack from ghstack (oldest at bottom):

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @rec

[ghstack-poisoned]

pytorch-bot · 2024-10-09T16:16:21Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137602

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 27c7380 with merge base 73fde0d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e75d417 Pull Request resolved: #137602

[ghstack-poisoned]

ghstack-source-id: f9d26b5 Pull Request resolved: #137602

[ghstack-poisoned]

mikaylagawarecki · 2024-10-16T16:15:42Z

.github/ci_commit_pins/xla.txt

@@ -1 +1 @@
-2eb4a60ed14a38260b85b0c765161f0ce45be6d1
+f71c02d1f457d58371e013632efb016c01bd1866


This was intentional to include pytorch/xla#8259

kit1980 · 2024-10-16T17:11:34Z

I'm not sure if it's OK to use different default in fbcode/OSS.
cc @malfet

kit1980 · 2024-10-16T17:13:04Z

Also this will break a lot of people.
Maybe it's time to break.

malfet

Tentatively LGTM, but please change the signatures a bit further. And not IS_FBCODE feels a bit suspicious, perhaps better behavior would be to add weights_only=False to respective callsights?

malfet · 2024-10-16T17:19:04Z

.github/ci_commit_pins/xla.txt

@@ -1 +1 @@
-2eb4a60ed14a38260b85b0c765161f0ce45be6d1
+f71c02d1f457d58371e013632efb016c01bd1866


This is not intended, is it?

This is intended, see #137602 (comment)

malfet · 2024-10-16T17:22:07Z

torch/serialization.py

@@ -1159,7 +1160,7 @@ def load(
    map_location: MAP_LOCATION = None,
    pickle_module: Any = None,
    *,
-    weights_only: Optional[bool] = None,
+    weights_only: Optional[bool] = not IS_FBCODE,


if you are changing it to hard boolean, please remove Optional and delete the logic associated with opt-in checks

Suggested change

weights_only: Optional[bool] = not IS_FBCODE,

weights_only: bool = not IS_FBCODE,

kit1980 · 2024-10-16T17:30:15Z

And just FYI users can use TorchFix to find and update call sites where weights_only is not explicitly set: https://github.com/pytorch-labs/torchfix?tab=readme-ov-file#tor102-torchload-without-weights_only-parameter-is-unsafe

albanD

A couple more pieces need to be added to the CI and we should announce this one as a bc-breaking change broadly for sure!
But yes let's do it!

Summary: Some fixes for pytorch/pytorch#137602 Reviewed By: xuzhao9 Differential Revision: D64628614 Pulled By: mikaylagawarecki

Summary: Some fixes for pytorch/pytorch#137602 Pull Request resolved: #2514 Reviewed By: xuzhao9 Differential Revision: D64628614 Pulled By: mikaylagawarecki fbshipit-source-id: edebf25cc6648919d5673a3baeaffdac26e5b91f

[ghstack-poisoned]

ghstack-source-id: 97e1430 Pull Request resolved: #137602

Prevent same global from being added multiple times Pull Request resolved: pytorch#139303 Approved by: https://github.com/janeyx99 ghstack dependencies: pytorch#138936, pytorch#139221, pytorch#139433, pytorch#139541, pytorch#137602

…139810) After pytorch#137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: pytorch#139810 Approved by: https://github.com/kit1980

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434 Pull Request resolved: #2558

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434 Pull Request resolved: #2558 (cherry picked from commit 165163a)

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: huggingface#34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: #34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

…139810) After pytorch#137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: pytorch#139810 Approved by: https://github.com/kit1980

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: huggingface#34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

Flip default on weights_only

4839d5a

[ghstack-poisoned]

This was referenced Oct 9, 2024

Remove dependency on numpy for serialization for XLA/open registration devices without numpy #137444

Closed

Remove numpy dependency for maia serialization #137600

Closed

mikaylagawarecki added a commit that referenced this pull request Oct 9, 2024

Flip default on weights_only

25fcbad

ghstack-source-id: e75d417 Pull Request resolved: #137602

mikaylagawarecki added release notes: python_frontend python frontend release notes category topic: bc breaking topic category labels Oct 9, 2024

Update on "Flip default on weights_only"

a141fb5

[ghstack-poisoned]

mikaylagawarecki added a commit that referenced this pull request Oct 9, 2024

Flip default on weights_only

6bd4ece

ghstack-source-id: f9d26b5 Pull Request resolved: #137602

JackCaoG mentioned this pull request Oct 15, 2024

Set weights_only to False for xser.load pytorch/xla#8259

Merged

Update on "Flip default on weights_only"

12ac158

[ghstack-poisoned]

mikaylagawarecki requested a review from a team as a code owner October 16, 2024 16:08

pytorch-bot bot added the ciflow/inductor label Oct 16, 2024

mikaylagawarecki requested a review from albanD October 16, 2024 16:15

mikaylagawarecki commented Oct 16, 2024

View reviewed changes

malfet approved these changes Oct 16, 2024

View reviewed changes

albanD approved these changes Oct 16, 2024

View reviewed changes

mikaylagawarecki added ciflow/trunk Trigger trunk jobs on your pull request ci-no-td Do not run TD on this PR labels Oct 17, 2024

mikaylagawarecki mentioned this pull request Oct 18, 2024

Fixes to prep for weights_only default flip pytorch/benchmark#2514

Closed

Update on "Flip default on weights_only"

9062584

[ghstack-poisoned]

mikaylagawarecki requested a review from kulinseth as a code owner October 22, 2024 16:14

mikaylagawarecki added a commit that referenced this pull request Oct 22, 2024

Flip default on weights_only

bcb59c6

ghstack-source-id: 97e1430 Pull Request resolved: #137602

vmoens pushed a commit to pytorch/rl that referenced this pull request Nov 13, 2024

[BugFix] Fix tests failing because of pytorch/pytorch#137602

98ae8c0

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434 Pull Request resolved: #2558

vmoens pushed a commit to pytorch/rl that referenced this pull request Nov 13, 2024

[BugFix] Fix tests failing because of pytorch/pytorch#137602

165163a

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434 Pull Request resolved: #2558

vmoens pushed a commit to pytorch/rl that referenced this pull request Nov 14, 2024

[BugFix] Fix tests failing because of pytorch/pytorch#137602

a647231

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434 Pull Request resolved: #2558 (cherry picked from commit 165163a)

ClowDragon mentioned this pull request Nov 26, 2024

torch.load error pytorch/opacus#690

Closed

github-actions bot deleted the gh/mikaylagawarecki/269/head branch December 5, 2024 02:14

mikaylagawarecki mentioned this pull request Dec 5, 2024

I think it is necessary to set torch.load with wetigh-only=True by default #142126

Closed

dvrogozh mentioned this pull request Dec 7, 2024

[XPU] model works with 2.5.1 while break with nightly build #142123

Closed

chang-l mentioned this pull request Jan 6, 2025

[TestFailure] Resolving exceptions thrown across multiple GraphBolt tests. dmlc/dgl#7852

Merged

4 tasks

joshuacwnewton mentioned this pull request Jan 28, 2025

PyTorch 2.6 has changed the default value of weights_only to True, breaking the loading of some nnunetv2 models MIC-DKFZ/nnUNet#2681

Open

This was referenced Feb 9, 2025

feat: add LoRA adapter layer and Mixtral LoRA training ROCm/Megatron-LM#53

Merged

fix: explicitly set weights_only to False during checkpoint loading to support PyTorch 2.6 ROCm/Megatron-LM#56

Merged

Archana-Shinde1 mentioned this pull request May 5, 2025

[REQUEST] Backport fix for Critical CVE-2025-32434 (9.8) to PyTorch v2.1.2 open-ce/open-ce#1362

Open

Phil26AT mentioned this pull request Jun 10, 2025

Make repo compatible with latest torch cvg/glue-factory#130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Flip default on weights_only #137602

Flip default on weights_only #137602

Uh oh!

mikaylagawarecki commented Oct 9, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 9, 2024 •

edited

Loading

Uh oh!

mikaylagawarecki Oct 16, 2024

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

malfet left a comment

Uh oh!

malfet Oct 16, 2024

Uh oh!

mikaylagawarecki Oct 16, 2024

Uh oh!

malfet Oct 16, 2024

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

albanD left a comment

Uh oh!

Uh oh!

		@@ -1 +1 @@
		2eb4a60ed14a38260b85b0c765161f0ce45be6d1
		f71c02d1f457d58371e013632efb016c01bd1866

	weights_only: Optional[bool] = not IS_FBCODE,
	weights_only: bool = not IS_FBCODE,

Flip default on weights_only #137602

Flip default on weights_only #137602

Uh oh!

Conversation

mikaylagawarecki commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137602

✅ No Failures

Uh oh!

mikaylagawarecki Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

mikaylagawarecki Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

malfet Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

kit1980 commented Oct 16, 2024

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikaylagawarecki commented Oct 9, 2024 •

edited

Loading

pytorch-bot bot commented Oct 9, 2024 •

edited

Loading