[v2.0.1] Release Tracker

[atalman]

This issue is for tracking cherry-picks to the release branch. Following is release branch for the 2.0.1 release.

Our plan from this point from this point is roughly:

• Phase 1 (until 04/21): Cherry-pick post deadline (End of day 5PM PST)
• Phase 2 (after 04/21): Perform extended integration/stability/performance testing based on Release Candidate builds.

Only issues that have ‘cherry-picks’ in this tracker will be considered for the release.

Cherry-Pick Criteria

Phase 1 (until 04/21):

The Releng team relies on the cherry pick process to manage risk to release quality, i.e. by porting a small set of commit from trunk that are "must-have" into the release branch, we limit the change to the minimal to address pressing issues. Thus, not everything a developer land into the trunk will make it into the release. So, please consider the criteria below and follow the cherry picking process. Only low-risk changes may be cherry-picked from master:

1. Fixes to regressions against the most recent release (e.g. 2.0.0 for 2.0.1 release; see module: regression issue list)
2. Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
3. Fixes to new features being introduced in 2.0.0 release
4. Documentation improvements
5. Release branch specific changes (e.g. blocking ci fixes, change version identifiers)

Any other change requires special dispensation from the release managers (currently @malfet, @atalman, @weiwangmeta , @izaitsevfb @osalpekar ). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 04/21):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.0
   git checkout release/2.0
   git cherry-pick abcdef12345
   
   # Submit a PR based against 'release/2.0' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.0

3. Make a request below with the following format:

Link to landed master PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

See HUD 2.0

Versions

2.0.1

[kshitij12345]

Link to landed master PR (if applicable):

• [fix] vmap: fix segfault on data access #97237

Link to release branch PR:

• [release 2.0.1] [fix] vmap: fix segfault on data access #97857

Criteria Category:

• 2: Low risk critical fixes for crashes

---

@atalman merged

[XuehaiPan]

Link to landed master PR (if applicable):

• Fix type hint for torch.Tensor.grad_fn #96804

Link to release branch PR:

• [release 2.0.1] Fix type hint for torch.Tensor.grad_fn (#96804) #97867

Criteria Category:

• 1: Fixes to regressions against the most recent release

---

@atalman merged

[crcrpar]

Link to landed master PR (if applicable):

• Fix wrong handling of grad_scale & found_inf in fused optimizers #95847

Link to release branch PR:

• [v2.0.1] Handle grad_scale and found_inf independently #97885

Criteria Category:

• 1: Fixes to regressions against the most recent release

---

@atalman merged

[crcrpar]

Link to landed master PR (if applicable):

• GradScaler recomputes optimizer_state["found_inf_per_device"] before optimizer.step #97415

Link to release branch PR:

• [v2.0.1] GradScaler recomputes found_inf before optimizer.step #97886

Criteria Category:

• 2: Low risk critical fixes

---

@atalman merged

[kshitij12345]

Link to landed master PR (if applicable):

• [fix] jacrev and jacfwd : support non-tensor args again #97746

Link to release branch PR:

• [release 2.0.1] [fix] jacrev and jacfwd : support non-tensor args again #97908

Criteria Category:

• 1: Fixes to regressions against the most recent release

---

@atalman merged

[soulitzer]

Link to landed master PR (if applicable):

• Fix module backward pre-hooks to actually update gradient #97983

Link to release branch PR:

• Fix module backward pre-hooks to actually update gradient (#97983) #98017

Criteria Category:

• 3: Fixes to new features being introduced in 2.0.0 release

---

@atalman merged

[zou3519]

Link to landed master PR (if applicable):

• [vmap] Fix index_select support when dim is negative #97916

Link to release branch PR:

• [vmap] Fix index_select support when dim is negative (#97916) #98019

Criteria Category:

• Low risk critical fixes for: crashes

---

@atalman merged

[XuehaiPan]

Link to landed master PR (if applicable):

• [DataLoader] Short circuit pin_memory recursion when operating on bytes #97737
• [DataLoader] Fix collation logic #97789
• Revisit torch._six.string_classes removal (#94709) #97863

Link to release branch PR:

• [release v2.0.1] Revisit torch._six.string_classes removal (#97737, #97789, #97863) #98055

Criteria Category:

• 1: Fixes to regressions against the most recent release

---

@atalman merged

[zou3519]

Link to landed master PR (if applicable):

• Clarify the saving of intermediates in the "extending torch.func" docs #98020

Link to release branch PR:

• Clarify the saving of intermediates in the "extending torch.func" docs (#98020) #98081

Criteria Category:

• 4. Documentation improvements

---

@atalman merged

[snadampal]

Link to landed pytorch/builder repo main:

• [aarch64] Adding scripts for aarch64 CI builder#1293
• [aarch64] update readme with the "--enable-mkldnn" option builder#1362
• pytorch/builder@0157db5
• pytorch/builder@85075c1
• pytorch/builder@a007fbc

Link to pytorch/builder repo release/2.0

• [aarch64] Adding scripts for aarch64 CI builder#1378

---

@malfet: Builder cherry-picks are not needed as packages are build from main branch

[snadampal]

torchtext wheel are missing for aarch64 for PyTorch 2.0.0 release

• [aarch64] torchtext 0.15.1 wheels are missing for aarch64_linux platform builder#1375

Link to release branch PR:

• [aarch64] Adding scripts for aarch64 CI builder#1378

Criteria Category:

1. Fixes to regressions against the most recent release (e.g. 2.0.0 for 2.0.1 release; see module: regression issue list)

This fixes the issue:

• [aarch64] torchtext 0.15.1 wheels are missing for aarch64_linux platform builder#1375

[snadampal]

Link to landed master PR (not yet merged, but approved as part of the below release branch PR)

• [aarch64] add runtime enable/disable for mkldnn matmul kernel #96141

Link to release branch PR:

• add runtime enable/disable for mkldnn matmul kernel #96169

Criteria Category:

1. Fixes to regressions against the most recent release (e.g. 2.0.0 for 2.0.1 release; see module: regression issue list)

This fixes the issue

• mkldnn matmul kernel may be slower than openblas kernel for very small tensor shapes #96140

---

@atalman this was not merged for release

[snadampal]

Link to landed master PR(under review)

• [aarch64] acl build script updates for multi isa build with armv8-a base architecture builder#1370

Link to release branch PR:

• [aarch64] wheel builder scripts and acl build system fixes for multi_isa builds builder#1379

Criteria Category:

1. Fixes to regressions against the most recent release (e.g. 2.0.0 for 2.0.1 release; see module: regression issue list)

This fixes the issue:

• Illegal instruction (core dumped) : PyTorch 2.0 on Raspberry Pi 4.0 8gb  #97226

[NivekT]

Link to landed master PR (if applicable):

• [DataLoader] Short circuit pin_memory recursion when operating on bytes #97737
• [DataLoader] Fix collation logic #97789

Link to release branch PR:

• [v2.0.1][DataLoader] Short circuit pin_memory recursion when operating on bytes #98349
• [v2.0.1][DataLoader] Fix collation logic  #98348

Criteria Category:

1. Fixes to regressions against the most recent release

---

@malfet: This is already merged as part of #98055

[ptrblck]

Link to landed master PR (if applicable):

• Back out "GradScaler recomputes optimizer_state["found_inf_per_device"] before optimizer.step (#97415)" #98613

Link to release branch PR:

• Revert "[v2.0.1] GradScaler recomputes found_inf before optimizer.step" #98624

Criteria Category:

• Reverts PR causing hangs via Back out "GradScaler recomputes optimizer_state["found_inf_per_device"] before optimizer.step (#97415)" #98613 (comment)

---

@atalman merged

[kulinseth]

Link to master PR:

• [MPS] Fix the MacOS 13.3 selector check. #96733
  Link to release branch PR:
• [MPS] Fix the MacOS 13.3 selector check. #98910

Criteria Category:
Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

---

@atalman merged

[sean-xiang-applovin]

Link to landed master PR:

• fix the use of incomplete vector<T> for C++20 compatibilities #93978

Link to release branch PR:

• fix the use of incomplete vector<T> for C++20 compatibilities (#93978) #99013

Criteria Category:

Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks.
It will help getting pytorch to build with XCode 14.2.3 which is the latest release

---

@malfet: This PR does not meet the specified criteria, as it does not fix any of the abovementioned issues.

[eracah]

Link to landed master PR (if applicable):

• fix load_sharded_optimizer_state_dict error on multi node #98063

Link to release branch PR:

• [release 2.0.1] [fix] fix load_sharded_optimizer_state_dict error on multi node #99103

Criteria Category:

• Fixes to new features being introduced in 2.0.0 release

---

@atalman merged

[jbschlosser]

Link to landed master PR:

• Adds 3D attn_mask support to merge_masks() for Multihead Attention fast path #98991
  Link to release branch PR:
• Adds 3D attn_mask support to merge_masks() for Multihead Attention fa… #99092

Criteria Category: Fixes to regressions against the most recent release

---

@atalman merged

[mlazos]

Link to landed master PR:

• [Dynamo] Fix for tuple construction from tuple iterators #97862

Link to release branch PR:

• Fix tuple iterator issue #99443

Criteria Category:
Critical fix for torch.compile to not crash on stable diffusion.

---

@atalman merged

[malfet]

Link to landed master PR:

• Workaround for CuDNN-8.7+ load bug #98644

Link to release branch PR:

• Workaround for CuDNN-8.7+ load bug (#98644) #99451

Criteria Category:

• Fixes to regressions against the most recent release (loading cudnn-8.7 on ubuntu-22.04+)

---

@atalman merged

[malfet]

Linkt to landed master PR:

• [MPS] Allow float16 input to float32 LayerNorm #96430

Link to release branch PR:

• [MPS] Allow float16 input to float32 LayerNorm (#96430) #99454

Criteria Category:

• Regression introduced in most recent release: unblocks float16 LayerNorm on MPS

---

@malfet: Yes, it enables non-standard behavior, but it also fixes float16 for LayerNorm

[malfet]

Link to landed master PR:

• [MPS] Fix batch_norm_backwards key #98794

Link to release branch PR:

• [MPS] Fix batch_norm_backwards key (#98794) #99456

Criteria Category:

• Low risk critical fixes for crashes in MPS

---

@atalman merged

[peri044]

Link to landed master PR:

• Fix flaky Dynamo export tests #96488

Link to release branch PR:

• [release/2.0.1] Fix flaky Dynamo export tests (#96488) #99459

Criteria Category:

Critical fixes for torch dynamo related tests in FX2TRT (Test failures)

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• NA

Link to release branch PR:

• [Release only change] mitigate issue caused by deprecated pip option #99487

Criteria Category:

• Critical fix to CI health, turn off failing test

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• Remove rocm python 3.11 restriction #97818

Link to release branch PR:

• Remove rocm python 3.11 restriction #99552

Criteria Category:

• Critical CI fix, enable py3.11 for rocm builds

---

@atalman merged

[crcrpar]

Link to landed master PR (if applicable):

• Remove redundant found_inf recompute from _step_supports_amp_unscaling path #98620

Link to release branch PR:

• [2.0.1] Remove redundant found_inf recompute #99666

Criteria Category:

• Fixes to new features being introduced in 2.0.0 release. The new features had a different behavior

---

@atalman merged

[mlazos]

Link to landed master PR:

• [Dynamo] Fix for tuple construction from tuple iterators #97862

Link to release branch PR:

• Fix tuple iterator issue #99443

Criteria Category: Critical fix for torch.compile to not crash on stable diffusion.

Hi all, will this get merged? This is a critical fix. The two test failures in the run are related to a missing "no_torchaudio" package, so unrelated to the change.

---

@mlazos this is merged

[malfet]

Link to landed master PR:

• Fix NumPy scalar arrays to tensor conversion #97696

Link to release branch PR:

• Fix NumPy scalar arrays to tensor conversion (#97696) #99732

Criteria Category: Regression from the most recent release

---

@atalman merged

[cpuhrsch]

Link to landed master PR (if applicable):

• Disable SDPA FlashAttention backward and mem eff attention on sm86+ for head_dim above 64 #99105

Link to release branch PR:

• [2.0.1] Disable SDPA FlashAttention backward and mem eff attention on sm86+ for head_dim above 64 #99736

Criteria Category:

• 2: Low risk critical fixes for crashes

---

@atalman merged

[bdhirsh]

Link to landed master PR (if applicable):

• aot autograd: handle detach() and no_grad() mutations on input #95980

Link to release branch PR:

• aot autograd: handle detach() and no_grad() mutations on input (#95980) #99740

Criteria Category:

Low risk critical fixes for AOTAutograd

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• Update MHA doc string #97046

Link to release branch PR:

• Update MHA doc string (#97046) #99746

Criteria Category:

• Documentation improvements

---

@malfet: merged

[voznesenskym]

Link to landed master PR (if applicable):

• Always get attr static out #95771

Link to release branch PR:

• Cherrypick - 95771 #99772

Criteria Category:

• Low risk critical fixes for crashes

[lezcano]

Link to landed master PR (if applicable):

• Add device_asserts before indirect loads and stores #98590

Link to release branch PR:

• Cherrypick 98590 #99795

Criteria Category:
Low risk critical fixes for crashes

[atalman]

closing this as finished