[v2.2.1] Release Tracker

[atalman]

🐛 Describe the bug

This issue is for tracking cherry-picks to the release branch. Following is release branch for the 2.2.1 release.

Our plan from this point from this point is roughly:

• Phase 1 (until 2/14): Cherry-pick post deadline (End of day 5PM PST)
• Phase 2 (after 2/14): Perform extended integration/stability/performance testing based on Release Candidate builds.

Only issues that have ‘cherry-picks’ in this tracker will be considered for the release.

Cherry-Pick Criteria

Phase 1 (until 2/14):

The Releng team relies on the cherry pick process to manage risk to release quality, i.e. by porting a small set of commit from trunk that are "must-have" into the release branch, we limit the change to the minimal to address pressing issues. Thus, not everything a developer land into the trunk will make it into the release. So, please consider the criteria below and follow the cherry picking process. Only low-risk changes may be cherry-picked from master:

1. Fixes to regressions against the most recent release (e.g. 2.2.0 for 2.2.1 release; see module: regression issue list)
2. Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
3. Fixes to new features being introduced in 2.2.0 release
4. Documentation improvements
5. Release branch specific changes (e.g. blocking ci fixes, change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @huydhn, @osalpekar, @malfet). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 2/14):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.2
   git checkout release/2.2
   git cherry-pick abcdef12345
   
   # Submit a PR based against 'release/2.2' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.2

3. Make a request below with the following format:

Link to landed trunk PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

See HUD 2.2

Versions

2.2.1

[huydhn]

Link to landed master PR (if applicable):

• Remove custom PyTorch build dependency logic on 3.11 builder#1697

Link to release branch PR:

• Remove custom PyTorch build dependency logic on 3.11 builder#1699

Criteria Category:

• CI fixes AppleSilicon binaries are build without OpenMP support #114721 to enable OpenMP backend for MacOS 3.11 wheel.

---

@atalman merged

[antoniojkim]

Link to landed trunk PR (if applicable):

• Replaced CHECK with TORCH_CHECK in order to not abort, but throw a Ru… #117653

Link to release branch PR:

• Replaced CHECK with TORCH_CHECK in order to not abort, but throw a RuntimeError instead #119301

Criteria Category:

• Low risk critical fix for unrecoverable crash via abort

---

@malfet: merged

[snadampal]

Link to master PR (if applicable):

• ci: aarch64 linux: fix torch performance regression with conda openblas package builder#1696

Link to 2.2 release branch PR:

• ci: aarch64 linux: fix torch performance regressions with conda openblas package builder#1700

Criteria Category:
Fixes this performance regression issue##119374
This PR fixes the regression introduced when the aarch64 linux wheel builder was moved from local openblas builds to conda package during PyTorch 2.1.

---

@malfet: needs some discussion, as technically its not a 2.1->2.2 regression, but rather 2.0->2.2 one

---

@malfet merged

[tringwald]

Link to landed trunk PR (if applicable):

• Added missing CircularPad*d references so the docs are actually built. #118465
• Clarified sampling process of torch.randn for complex dtypes. #118315
• Updated docs for deprecated torch.set_default_tensor_type #115041

Link to release branch PR:

• Missing docs for CircularPad2d #119313
• Clarified sampling process of torch.randn for complex dtypes. (#118315) #119315
• Updated docs for deprecated torch.set_default_tensor_type (#115041) #119316

Criteria Category:

• Doc fixes

---

@atalman merged

[mvpatel2000]

Dtensor / device mesh is currently broken on 2.2 and has serious silent correctness errors. These were fixed by teh following PRs:

• [2d] unflatten_tensor on compute stream for DTensorExtension #116559
• [2d] Ensure gradient clear out pending AsyncCollectiveTensor in FSDP Extension #116122 (required in conjunction with next PR)
• [reland] unflatten_tensor on compute stream for DTensorExtension #117020
• [FSDP] enable autograd in forward prefetching #116792

Related to distributed checkpointing:

• [DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

Related dtensor sync_module_states bug:

• [fsdp][2d] FSDP sync module states handle tensor subclass #117336

These are all low-risk PRs and fix various issues, eg silent correctness.

CC: @chauhang

---

@mvpatel2000 could you please post cherry-picks to release/2.2 branch for these PRs ?

Edit: skylion included PR below

[mdevino]

Link to landed trunk PR (if applicable):

• Fix typo on torch.frombuffer() documentation #119214

Link to release branch PR:

• Fix typo on torch.frombuffer() documentation #119388

Criteria Category:
Documentation Improvements - this PR fixes a typo in the torch.frombuffer() function documentation.

---

@atalman merged

[mdevino]

Link to landed trunk PR (if applicable):

• Fix typo on Contribution Guide #119428

Link to release branch PR:

• Fix typo on Contribution Guide (#119428) #119505

Criteria Category:
Documentation Improvements - this PR fixes a typo in the contribution guide

---

@atalman merged

[huydhn]

Link to landed master PR (if applicable):

• Pin conda-build to 3.28.4 builder#1698

Link to release branch PR:

• Pin conda-build to 3.28.4 builder#1701

Criteria Category:

• CI fixes to build conda MacOS due to conda build has non-zero exit status despite successful build conda/conda-build#5167. We might not need this as the issue has been fixed in conda 24.1.1 patch release. This is standing by in case RC build is failing.

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Switch nightly binaries to oidc. Remove aws keys #117416

Link to release branch PR:

• Switch nightly binaries to oidc. Remove aws keys (#117416) #119560

Criteria Category:

• CI fixes use oidc when uploading binaries

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• [oidc] Migrate Triton wheel upload to oidc #117648
• Fix trition wheels build (take 2) #117706

Link to release branch PR:

• [oidc] Migrate Triton wheel upload to oidc (#117648) #119564

Criteria Category:

• CI fixes use oidc when uploading triton wheels

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Migrate rocm test to using oidc #117160
• The permission id-token write needs to be set on rocm-test callers #117422

Link to release branch PR:

• Migrate rocm test to using oidc (#117160) #119565

Criteria Category:

• CI fixes use oidc for rocm

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• [CI] Explicitly specify read-all permissions on the token #117290
• [CI] Set permissions for stale workflow #117371

Link to release branch PR:

• [CI] Explicitly specify read-all permissions on the token (#117290) #119568

Criteria Category:

• CI fixes use oidc

---

@atalman merged

[mvpatel2000]

Fixes sharded checkpoint bug which causes OOM on large GPU counts:

#117799

CC: @chauhang @fegin

Edit: skylion included PR below

Edit: See #119295 (comment). #117799 was replaced by #118197 in main

---

@atalman wrote: @mvpatel2000 I don't see this PR being included as a cherry-pick

@atalman wrote: As discussion here: #119295 (comment) this PR is not required

[Skylion007]

Link to landed trunk PR (if applicable):

• [dtensor] fix dtensor _to_copy op for mix precision #116426

Link to release branch PR:

• [dtensor] fix dtensor _to_copy op for mix precision (#116426) #119687

Criteria Category:

• Silent Correctness

---

@Skylion007 The Cherry PIck introduce number of breakages. Could you please resolve these ? Resolved

---

@atalman merged

[Skylion007]

Link to landed trunk PR (if applicable):

• [FSDP] enable autograd in forward prefetching #116792

Link to release branch PR:

• [FSDP] enable autograd in forward prefetching (#116792) #119688

Criteria Category:

• Silent Correctness

---

@atalman merged

[Skylion007]

Link to landed trunk PR (if applicable):

• [2d] Ensure gradient clear out pending AsyncCollectiveTensor in FSDP Extension #116122
• [reland] unflatten_tensor on compute stream for DTensorExtension #117020
• [fsdp][2d] FSDP sync module states handle tensor subclass #117336

Link to release branch PR:

• [FSDP][2D] Fix DTensor Extension Bugs #119690

Criteria Category:

• Silent Correctness

---

@atalman merged

[Skylion007]

Link to landed trunk PR (if applicable):

• [DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

Link to release branch PR:

• [DCP][state_dict] DCP state_dict cannot correctly find FQN when the l… #119691

Criteria Category:

• Checkpoint loading bug in DCP: Fix to new feature / regression fix.

---

@atalman merged

[Skylion007]

Link to landed trunk PR (if applicable):

• [state_dict] Calls wait() for the DTensor to_local() result #118197
• [DTensor] Only wait on AsyncCollectiveTensor after DTensor-based state dict loading #119716

Link to release branch PR:

• [state_dict] Calls wait() for the DTensor to_local() result (#118197) #119692

Criteria Category:

• Crash (low risk fix)

---

@atalman wrote: @Skylion007 this cherry-pick is missing unit tests, we require unit tests for all cherry-picks. Also this one mentions #117799 do we need this cherry pick as well here ?

---

@fegin wrote: @atalman
#117799 is never landed. The actual fixes are the #118197 and #119716. The reason why these 2 do not have unit tests is because these 2 PRs are actually fixing the existing unit tests. Before these 2 PRs are landed, the tests (e.g., "testdistributed/checkpoint/state_dict.py") are marked as flaky. We tested locally before the PRs are landed and the errors could be reproduced roughly 3 out of 10 runs. The PRs fixes the issue that the state_dict may have garbage values.

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Specific PYTORCH_VERSION when building RC for test channel test-infra#4803
• Add constraint for version to conda build when in test channel test-infra#4805

Link to release branch PR:

• Specific PYTORCH_VERSION when building RC for test channel # test-infra#4944

Criteria Category:

• CI cherry-picks originally in 2.1.2

---

@atalman merged

[huydhn]

Link to landed trunk PR (if applicable):

• Grant test build wheel workflows with correct permissions test-infra#4870
• Refactor wheel upload job to a separate job running on GH ephemeral runner test-infra#4877
• Correctly checkout test-infra on Nova _binary_upload job test-infra#4882
• Refactor conda upload job to a separate job running on GH ephemeral runner test-infra#4886
• Fix anaconda path in conda environment on GH ephemeral runner test-infra#4891
• Fix the wrong upload of vs2019 conda package on Windows test-infra#4893
• Move upload step to github after post-script test-infra#4894
• Set channel before setup-binary-upload on upload jobs test-infra#4901

Link to release branch PR:

• Cherry pick a bunch of OIDC changes for 2.2.1 release test-infra#4945

Criteria Category:

• CI cherry-picks to use OIDC in the release workflows

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• NA

Link to release branch PR:

• Update generate_binary_build_matrix for release 2.2.1 test-infra#4946

Criteria Category:

• CI release only, bump versions

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Fix TCP Store Windows #118860

Link to release branch PR:

• Fix TCP Store Windows (#118860) #119769

Criteria Category:

• Critical FIx Windows

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• [EZ] Set maximum supported version of Python as 3.12 #119743

Link to release branch PR:

• [EZ] Set maximum supported version of Python as 3.12 (#119743) #119770

Criteria Category:

• PyPi Metadata fix

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Update oneDNN build option for older systems #118057

Link to release branch PR:

• Update oneDNN build option for older systems (#118057) #119773

Criteria Category:

• Critical Regression

---

@atalman merged

[fegin]

Link to landed trunk PR (if applicable):

• Correctly formatting the example in get_state_dict #119532

Link to release branch PR:

• Correctly formatting the example in get_state_dict (#119532) #119804

Criteria Category:

• Documentation improvements

---

@atalman merged

[fegin]

Link to landed trunk PR (if applicable):

• [DCP][state_dict] Fix the issue that get_state_dict/set_state_dict ignore the buffer #119573

Link to release branch PR:

• [DCP][state_dict] Fix the issue that get_state_dict/set_state_dict ig… #119807

Criteria Category:

• Low risk critical fixes

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Add triton version for nightly and release builder#1703

Link to release branch PR:

• Add triton version for nightly and release  builder#1704
• [RELEASE-ONLY CHANGES] Update triton package name builder#1707

Criteria Category:

• Critical CI fix - Triton

---

@huydhn merged

[mvpatel2000]

Link to landed trunk PR (if applicable):

• Remove HSDP validation check  #112435
• Remove parent device mesh check #118620
• [FSDP] Fixed device_mesh and auto wrap #119064
• Patch all_gather to support HSDP + TP  #118638
• Update chunk_dtensor to support HYBRID_SHARD #119481

Link to release branch PR:

• HSDP + TP integration bug fixes  #119819

Criteria Category:

• Fixes to new features being introduced in 2.2.0 release

---

@atalman merged

[wanchaol]

Link to landed trunk PR (if applicable):

• Fix ColwiseParallel typo #116151

Link to release branch PR:

• Fix ColwiseParallel typo (#116151) #119821

Criteria Category:

• Documentation Improvements

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• [Inductor] Skip triton templates for mixedmm on SM70- #118591

Link to release branch PR:

• [Inductor] Skip triton templates for mixedmm on SM70- (#118591) #119894

Criteria Category:

• Critical FIx on V100, regression against 2.1.0 release

---

@huydhn merged

[atalman]

Link to landed master PR (if applicable):
Properly preserve SymInt input invariant when splitting graphs #117406

Link to release branch PR:
Properly preserve SymInt input invariant when splitting graphs (#117406)

Criteria Category:
Crashes - Functional failure in case of DDPOptimizer and dynamic shape.

---

@atalman merged

[mvpatel2000]

Link to landed master PR:
#117453

Link to release branch PR:
#119916

Criteria Category:
Fixes to new features being introduced in 2.2.0 release -- minor fix for argument not propagated in new API introduced in 2.2

--
@fegin wrote: @mvpatel2000 Can you confirm this is required to avoid OOM? distributed_state_dictwill perform cpu_offloading in a later stage after state_dict() even without this PR. This PR just make the offloading done in state_dict() call. This is specifically for model state_dict and, iirc, your experiments were OOM when doing optimizer state_dict, not model state_dict. This PR does not have any unit tests and our existing cpu offload unit tests only test the final results, and does not test when cpu offloading happens. So it seems that this PR may not be included in 2.2.1 due to lack of unit test.

cc., @atalman

---

@atalman wrote: @mvpatel2000 reverting this PR: #119995 . Will not be merging this cherry-pick

[wanchaol]

Link to landed trunk PR (if applicable):

• fix compile DTensor.from_local in trace_rule_look up #119659

Link to release branch PR:

• fix compile DTensor.from_local in trace_rule_look up (#119659) #119941

Criteria Category:

• regressions against the most recent release, silent correctness

---

@atalman merged

[Vins33]

Hi guys, could you tell me if the next release will support cuda12.4?

@atalman wrote: @Vins33 We will have concrete dates once Cuda 12.4 is released. Please stay tuned

[atalman]

Link to landed trunk PR (if applicable):

• [RelEng] Define BUILD_BUNDLE_PTXAS #119750
• pytorch/builder@13a5d23

Link to release branch PR:

• [RelEng] Define BUILD_BUNDLE_PTXAS (#119750) #119988
• Bundle PTXAS into 11.8 wheel builder#1705

Criteria Category:

• Critical regression

---

@atalman merged

[atalman]

Please Note We don't accept cherry-picks anymore. We are currently in Phase 2 (after 2/14).

[huydhn]

Link to landed trunk PR (if applicable):

• Always run binary upload test-infra#4958

Link to release branch PR:

• Always run binary upload test-infra#4962

Criteria Category:

• CI fix to ignore 3.12 data binary which doesn't exist atm

---

@huydhn merged

[atalman]

Closing this. Completed