[v2.1.1] Release Tracker

[atalman]

🐛 Describe the bug

This issue is for tracking cherry-picks to the release branch. Following is release branch for the 2.1.1 release.

Our plan from this point from this point is roughly:

• Phase 1 (until 11/3): Cherry-pick post deadline (End of day 5PM PST)
• Phase 2 (after 11/3): Perform extended integration/stability/performance testing based on Release Candidate builds.

Only issues that have ‘cherry-picks’ in this tracker will be considered for the release.

Cherry-Pick Criteria

Phase 1 (until 11/3):

The Releng team relies on the cherry pick process to manage risk to release quality, i.e. by porting a small set of commit from trunk that are "must-have" into the release branch, we limit the change to the minimal to address pressing issues. Thus, not everything a developer land into the trunk will make it into the release. So, please consider the criteria below and follow the cherry picking process. Only low-risk changes may be cherry-picked from master:

1. Fixes to regressions against the most recent release (e.g. 2.1.0 for 2.1.1 release; see module: regression issue list)
2. Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
3. Fixes to new features being introduced in 2.1.0 release
4. Documentation improvements
5. Release branch specific changes (e.g. blocking ci fixes, change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @huydhn, @osalpekar, @malfet). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 11/3):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.1
   git checkout release/2.1
   git cherry-pick abcdef12345
   
   # Submit a PR based against 'release/2.1' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.1

3. Make a request below with the following format:

Link to landed master PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

See HUD 2.1

Versions

2.1.1

[tringwald]

Link to landed master PR (if applicable):

• Improved the docs for torch.std, torch.var, torch.std_mean, torch.var_mean and torch.cov #109326

Link to release branch PR:

• Improved the docs for torch.std, torch.var, torch.std_mean, torch.var… #110969

Criteria Category:

• Documentation fixes

---

@atalman merged

[Skylion007]

Link to landed master PR (if applicable):

• Update fully_sharded_data_parallel to fix typing #110545

Link to release branch PR:

• Update fully_sharded_data_parallel to fix typing (#110545) #111036

Criteria Category:

• Documentation fixes

---

@atalman merged. However, will be reverted since does not fall into documentation fixes

Revert:

• Revert "Update fully_sharded_data_parallel to fix typing (#110545)" #111683

---

@atalman merged
@malfet: Criteria category is misleading, as #109326 is not a documentation fix

[soumith]

got request from Mosaic to include these two fixes:

Link to landed master PR (if applicable):

• [FSDP] continue if param not exist in sharded load #109116
• Update chunk_sharding_spec.py #108915

Link to release branch PR:

• [FSDP] continue if param not exist in sharded load (#109116) #111149
• Update chunk_sharding_spec.py (#108915) #111151

Criteria Category:

• Critical fixes

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• Try to use linux.arm64.2xlarge runners #107672

Link to release branch PR:

• Try to use linux.arm64.2xlarge runners (#107672) #111039

Criteria Category:

• Required CI fix for linux Arm 64

---

@atalman merged

[Skylion007]

A minor NCCL version dropped that has a bug fix. Should we open a PR onto 2.1 tree to update it?
Link to the trunk PR:

• N/A (trunk is already on 2.19.3, see [BE]: Update NCCL submodule to v2.19.3 #110827)

Link to release branch PR:

• [2.1.1] Update NCCL to 2.18.6 for upstream bugfix #111677

Criteria Category:

• Low risk critical fixes for: crashes (see NCCL 2.18.3: Socket recv failed while polling for opId=... / Cuda failure 'invalid argument' NVIDIA/nccl#957 )

---

@malfet: merged

[atalman]

Link to landed master PR (if applicable):

• Add pypi required metadata to all wheels except linux #111042

Link to release branch PR:

• Add pypi required metadata to all wheels except linux #111578

Criteria Category:

• Critical CI fix for Pytorch 2.0.1 pypi wheel does not install dependent cuda libraries #100974

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• Fix concurrency limits for Create Release #110759

Link to release branch PR:

• [Release/2.1] Fix concurrency limits for Create Release #111597

Criteria Category:

• Critical CI fix

---

@malfet merged

[jataylo]

Link to landed master PR (if applicable):

• Introduce is_big_gpu condition for test_max_autotune #111467

Link to release branch PR:

• [Release/2.1] Introduce is_big_gpu condition for test_max_autotune #111667

Criteria Category:

• Unit test fixes

---

@huydhn: Do we need to cherry pick this change? From what I see, this fixes the issue introduced by #111381, but that change isn't picked to 2.1.1 (cc @shunting314) Confirmed with @shunting314 that we don't need to cherry pick #111381 or this one

[atalman]

Link to landed master PR (if applicable):

• NA

Link to release branch PR:

• [release only] Pin disabled and unstable jobs. keep CI green #111675

Criteria Category:

• CI Fixes

---

@atalman merged

[wz337]

Link to landed master PR (if applicable):

• [Test][ShardedTensor] Add test for corner case for chunk sharding spec #109626

Link to release branch PR:

• [Release/2.1.1] [Test][ShardedTensor] Add test for corner case for chunk sharding spec #109626 #111679

Criteria Category:

• Test for the critical fix(Update chunk_sharding_spec.py #108915) that is cherry-picked into Release2.1 above

---

@atalman merged

[wz337]

Comments from user:

without this, resuming training at scale goes from minutes to hours

Link to landed master PR (if applicable):

• [DCP] Remove _shard_tensor() call in load_sharded_optimizer_state_dict in optimizer.py #111096
• [DCP][test] Make dim_0 size of params scale with world_size in torch/distributed/checkpoint/test_fsdp_optim_state.py #112825

Link to release branch PR:

• [Release/2.1.1][DCP] Remove _shard_tensor() call in load_sharded_optimizer_state_dict in optimizer.py #111096 #111687
• [DCP][test] Make dim_0 size of params scale with world_size in torch/distributed/checkpoint/test_fsdp_optim_state.py #112825 #112894

Criteria Category:

• Special dispensation: this is a low risk performance fix.
  The context is that without the fix, distributed checkpointing load scale very very poorly with number of GPUs. While it is fast on 1-2 nodes, it slows down to nearly an hour at scale (480 GPUs), because of the collective communication happening behind the scenes. This is flagged by MosaicML folks.

---

@malfet: Looks fine, need to do some manual testing before merging
@wz337: I have added more context in the cherry-pick PR and included a unit test update that could potentially capture the performance improvement.

---

@atalman merged

[thiagocrepaldi]

Link to landed master PR (if applicable):

• [ONNX] Fix aten::new_zeros due to TorchScript behavior change on Pytorch 2.1 Fix #110935 #110956

Link to release branch PR:

• [Release/2.1.1][ONNX] Fix aten::new_zeros due to TorchScript behavior change on Pytorch 2.1 Fix #110935 #111694

Criteria Category:

• Fix regression to TorchScript

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• Fix create source distribution step for release #111697

Link to release branch PR:

• Fix create source distribution step for release (#111697) #111801

Criteria Category:

• Critical CI fix

---

@atalman merged

[Skylion007]

#111703 is a good backport candidate and fixes a repeatable and problematic mem leak according to the issue.

@Skylion007 please follow cherry-pick process in the header to post a cherry-pick for review: #110961 (comment)

Closed my PR for @tringwald's issue below

[tringwald]

Link to landed master PR (if applicable):

• Fixed a memory leak in PyTorchFileReader #111703

Link to release branch PR:

• Fixed a memory leak in PyTorchFileReader #111814

Criteria Category:

• Memory Leak

---

@malfet: merged, as change is very localized, but we need to have better guidance to distinguish between large and small memory leaks going forward

[jcaip]

Link to landed master PR (if applicable):

• [sparse] Fix semi-structured sparse shape mismatch bug #110420

Link to release branch PR:

• [sparse] Fix semi-structured sparse shape mismatch bug (#110420) #111845

Criteria Category:

• Critical Fixes

---

@atalman moving it to 2.1.2 release

[eqy]

Link to landed master PR (if applicable):

• [NCCL][CUDA][CUDA Graphs] Flush enqueued work before starting a graph capture 2 #110665

Link to release branch PR:

• [NCCL][CUDA][CUDA Graphs] Flush enqueued work before starting a graph capture 2 #111875

Criteria Category:

• Critical fixes

---

@malfet: this does not look like a regression between 2.0 and 2.1 is it?
@eqy: not sure if it's 2.0, this fixes hopefully the remaining outstanding issues that began after #98793

---

@huydhn merged

[eqy]

Link to landed master PR (if applicable):

• [CUDA][cuFFT] Initialize CUDA context for cuFFT before execute is called #110326

Link to release branch PR:

• [CUDA][cuFFT] Initialize CUDA context for cuFFT before execute is called #111877

Criteria Category:

• Critical fixes

---

@atalman merged

[BowenBao]

Link to landed master PR (if applicable):

• Ignore beartype if its version is 0.16.0 #111859

Link to release branch PR:

• [Release/2.1.1] Ignore beartype if its version is 0.16.0 #111861

Criteria Category:

• Critical fixes: regression, crash

---

@atalman merged

[malfet]

Link to the landed main commit:

• pytorch/builder@c5e331c

Lint to release branch PR:

• [aarch64_wheel] Fix readdir crash builder#1574

Criteria Category:

• Regression from the previous release, crash

---

@atalman merged

[huydhn]

Link to landed master PR (if applicable):

• fix TEST_ROCM definition to disable test_jit_cudnn_extension on rocm #110385

Link to release branch PR:

• fix TEST_ROCM definition to disable test_jit_cudnn_extension on rocm (#110385) #111942

Criteria Category:

• Critical CI fix

---

@huydhn merged

[atalman]

Link to landed master PR (if applicable):

• Add continue-on-error if ssh step is failing #111916

Link to release branch PR:

• Add continue-on-error if ssh step is failing (#111916) #112026

Criteria Category:

• Critical CI fix

---

@malfet merged

[atalman]

Link to landed master PR (if applicable):

• Docker release build don't include build suffix in the release #112046

Link to release branch PR:

• Docker release build don't include build suffix in the release #112040

Criteria Category:

• Critical CI docker release fix

---

@atalman merged

[atalman]

Link to landed master PR (if applicable):

• [dynamo] Register einops functions lazily #110575
• [dynamo] Add specialized variable tracker for sys.modules #110990

Link to release branch PR:

• [release/2.1.1][dynamo] Fix circular import with einops #111835

Criteria Category:

• Critical Fix

---

@malfet merged

[atalman]

Link to landed master PR (if applicable):

• Reland "AOTAutograd: Go down inference path if no outputs require grad (#111011)" #111347

Link to release branch PR:

• Reland "AOTAutograd: Go down inference path if no outputs require gra… #112163

Criteria Category:

• Performance issue Eval torch compile slower than eager for Torch 2.1.0 #110666

---

@bdhirsh could you please take a look at the cherry pick, there are multiple failures

[atalman]

Link to landed master PR (if applicable):

• Verify flatbuffer module fields are initialized #109794

Link to release branch PR:

• Verify flatbuffer module fields are initialized (#109794) #112165

Criteria Category:

• Critical fix, Segmentation fault Segmentation fault in 'flatbuffer_loader.cpp:298' #109793

---

@malfet: merged

[kit1980]

Link to landed master PR (if applicable):

• 3a284da

Link to release branch PR:

• Revert "Do not materialize entire randperm in RandomSampler (#103339)" #112187

Criteria Category:

• Low risk critical fixes for backwards compatibility,

Alban: This is actually a regression where it used to be possible to have indices on the mps device and it is not possible anymore.

---

@huydhn merged

[huydhn]

Link to landed master PR (if applicable):

• [MPS] Skip virtualized devices #111576

Link to release branch PR:

• [MPS] Skip virtualized devices (#111576) #112265

Criteria Category:

• Critical CI fix, GitHub MacOS runner

---

@huydhn merged

[malfet]

Link to the landed PR:

• [MPS] Fix mps to cpu copy with storage offset #109557

Link to the release branch PR:

• [MPS] Fix mps to cpu copy with storage offset #112432

Criteria Category:

• Low risk critical fixes for silent correctness on MPS->CPU copy

---

@malfet merged

[huydhn]

Link to the landed PR:

• Remove http://repo.okay.com.mx builder#1575

Link to the release branch PR:

• pytorch/builder@7790132

Criteria Category:

• Critical CI fixes

---

@huydhn merged. Learn from @malfet: We don't have separation of the Docker image from main and release branch on builder atm. So I shouldn't have merged the change into the release branch because it would override the correct image from main. @malfet fixes this by rebuilding the image in pytorch/builder@617327e

[wz337]

Got request from MosaicML to include this 1 fix.

Comments from users:

without this, we can't do training resumption because the model gets loaded without the optimizer

Link to the landed PR:

• Enable planner to be used for loading sharded optimizer state dict #112259
• [DCP] Add test for planner option for load_sharded_optimizer_state_dict #112891

Link to the release branch PR:

• [CherryPick] Enable planner to be used for loading sharded optimizer state dict #112259 #112520
• [DCP] Add test for planner option for load_sharded_optimizer_state_dict #112891 #112930

Criteria Category:

• Special dispensation: this is a low-risk fix for an existing feature. Requested by MosaicML to include.

---

@malfet: I think category is wrong, this feels like a feature work (adding new argument to the existing method), isn't it?
@huydhn: To summarize the decision going forward, for patch release, we will require regression tests to be part of the cherry pick. Without some tests, it's hard to say if the cherry pick is needed. In this specific case, this looks like the change doesn't fit for patch release and would better be as part of 2.2 release instead.
@malfet: merged

[huydhn]

Link to the landed PR:

• N/A

Link to the release branch PR:

• [Release-only] Pin Docker images to 2.1 for release #112665

Criteria Category:

• Critical CI fixes. This is to update the pinned Docker image to -2.1 to match the release branch

---

@huydhn merged

[huydhn]

Link to the landed PR:

• Upload to docker.io when pushing release tag builder#1576

Link to the release branch PR:

• Upload to docker.io when pushing release tag builder#1577

Criteria Category:

• Critical CI fixes. This allows the build Docker images to be updated when cutting RC

---

@malfet: We never had release tags in builder repo, had we? And may be it's fine to start doing it going forward, but not sure why it needs to be cherry-picked into release branch for 2.1.1 release)
@huydhn: The cherry pick is just to make sure that we have 2.1 Docker images pushed to Docker hub. After chatting with Nikita, we decide to close this one as it doesn't fit the cherry pick criteria. See comments in pytorch/builder#1576 (comment)
@atalman not merged to 2.1.1

[malfet]

Link to the landed trunk PR:

• Prevent OOB access in foreach_list variants #112349

Link to the release branch PR:

• Prevent OOB access in foreach_list variants (#112349) #112756

Criteria Category:

• Fixes to regressions against the most recent release (didn't crash in 2.0.1, crashes in 2.1.0)

---

@malfet: merged

[malfet]

Link to the landed trunk PR:

• Fix buffer overflow in torch.sort #111672

Link to the release branch PR:

• Fix buffer overflow in torch.sort #112784

Criteria Category:

• Fixes to regressions against the most recent release (didn't crash in 2.0.1, crashes in 2.1.0)

---

@malfet merged

[huydhn]

Link to the landed trunk PR:

• Revert "Fix bug: not creating empty tensor with correct sizes and device. (#106734)" #112170

Link to the release branch PR:

• Revert "Fix bug: not creating empty tensor with correct sizes and device. (#106734)" (#112170) #112790

Criteria Category:

• Fixes to regressions against the most recent release Noisy warnings from broadcasting in boolean operations #110941

---

@huydhn merged

[drisspg]

Link to landed master PR (if applicable):

• Fix the meta func for mem_eff_backward #110893

Link to release branch PR:

• Fix the meta func for mem_eff_backward (#110893) #112792

Criteria Category:

• Fixes to regression when trying to compile scaled dot product attention

---

@huydhn merged, confirm that the issue on #110832 (reproducible on 2.1.0) is fixed

[drisspg]

Link to landed master PR (if applicable):

• Mirror of Xformers Fix #112267

Link to release branch PR:

• Mirror of Xformers Fix (#112267) #112795

Criteria Category:

• Silent correctness regression for scaled dot product attention

---

@huydhn merged

[drisspg]

Link to landed master PR (if applicable):

• Fix mem eff bias bug #112673

Link to release branch PR:

• Fix mem eff bias bug (#112673) #112796

Criteria Category:

• Fixes a silent correctness issue with scaled dot product attention reported in: [Breaking change 2.1] Passing non-contiguous inputs to SDPA on CUDA device with the mem-efficient attention backend returns garbage #112577

---

@huydhn merged confirm that the issue on #112577 (reproducible on 2.1.0) is fixed

[malfet]

Link to landed trunk PR:

• Better errors in c10::DriverAPI on dl failure #112995
• c10::DriverAPI Try opening libcuda.so.1 #112996

Link to release branch PR:

• c10::DriverAPI Try opening libcuda.so.1 #113096

Criteria Category:

• Fixes to new features being introduced in 2.1.0 release

---

@malfet merged

[malfet]

Closing and unpinning, as final RC is out