Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v.1.12.0] Release Tracker #78005

Closed
atalman opened this issue May 20, 2022 · 58 comments
Closed

[v.1.12.0] Release Tracker #78005

atalman opened this issue May 20, 2022 · 58 comments
Labels
oncall: releng In support of CI and Release Engineering triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone
1.12.0

Comments

@atalman
Copy link
Contributor

atalman commented May 20, 2022 •
edited
Loading

We cut a release branch for the 1.12.0 release.

Our plan from this point from this point is roughly:

  • Phase 1 (until 06/13): work on finalizing the release branch
  • Phase 2 (after 06/13): perform extended integration/stability/performance testing based on Release Candidate builds.

This issue is for tracking cherry-picks to the release branch.

Cherry-Pick Criteria

Phase 1 (until 06/13/22):

Only low-risk changes may be cherry-picked from master:

  1. Fixes to regressions against the most recent minor release (e.g. 1.11 for 1.12 release; see module: regression issue list)
  2. Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
  3. Fixes to new features introduced in the most recent minor release (e.g. 1.11.0 for 1.12 release)
  4. Test/CI fixes
  5. Documentation improvements
  6. Compilation fixes or ifdefs required for different versions of the compilers or third-party libraries
  7. Release branch specific changes (e.g. change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @seemethere ). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 06/13/22):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

  1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

  2. Create (but do not land) a PR against the release branch.

    # Find the hash of the commit you want to cherry pick
# (for example, abcdef12345)
git log

git fetch origin release/1.12
git checkout release/1.12
git cherry-pick abcdef12345

# Submit a PR based against 'release/1.12' either:
# via the GitHub UI
git push my-fork

# via the GitHub CLI
gh pr create --base release/1.12

  3. Make a request below with the following format:

Link to landed master PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
*
  1. Someone from the release team will reply with approved / denied or ask for more information.
  2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

Please note HUD Link with branch CI status and link to the HUD to be provided here.
HUD

Versions

1.12.0
@atalman atalman added oncall: releng In support of CI and Release Engineering triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels May 20, 2022
@atalman atalman added this to the 1.12.0 milestone May 20, 2022
@atalman atalman pinned this issue May 20, 2022
@zou3519
Copy link
Contributor

zou3519 commented May 20, 2022 •
edited by seemethere
Loading

Link to landed master PR (if applicable):

  • n/a

Link to release branch PR:

Criteria Category:

  • Release branch specific changes (removing prototype feature that should not be in release)

@seemethere: Merged

@pritamdamania87
Copy link
Contributor

pritamdamania87 commented May 22, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Test/CI fixes

@atalman merged

@robieta
Copy link

robieta commented May 23, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:
Critical fix: profiler segfault
It's a one line fix and should be effectively zero risk.

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented May 24, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:
Critical CI fix

@atalman Merged

@atalman
Copy link
Contributor Author

atalman commented May 24, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical Fixes

@atalman Merged

@atalman
Copy link
Contributor Author

atalman commented May 24, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical Fixes

@atalman merged

@zou3519
Copy link
Contributor

zou3519 commented May 25, 2022 •
edited
Loading

EDIT: Nevermind, please ignore

@JackCaoG
Copy link
Collaborator

JackCaoG commented May 25, 2022 •
edited by malfet
Loading

Link to landed master PR:

  • None

Link to release branch PR:

Criteria Category:
Release branch specific changes : updating pytorh/xla 1.12 branch for CI testing.

@malfet: edited and merged

@wonjoolee95 wonjoolee95 mentioned this issue May 25, 2022
Merged
@atalman
Copy link
Contributor Author

atalman commented May 26, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical Fix

@malfet merged

@atalman
Copy link
Contributor Author

atalman commented May 30, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI fix

@atalman merged

@NivekT
Copy link
Contributor

NivekT commented May 31, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Documentation improvements

@atalman merged

@mcarilli
Copy link
Collaborator

mcarilli commented Jun 1, 2022 •
edited by atalman
Loading

The regression described in #78549 IS present in the 1.12 release branch (https://github.com/pytorch/pytorch/commits/release/1.12/test/distributed/test_c10d_common.py).

@ngimel advises we should fix it or at least allow a workaround in 1.12, and @azrael417 and I won't argue with that.

cc @mrshenli

See @mrshenli fix below:

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

Critical fixes for: backwards compatibility. See #78549

@atalman merged

@kshitij12345
Copy link
Collaborator

kshitij12345 commented Jun 2, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category: Documentation improvements

cc: @anjali411

@atalman merged

@NivekT
Copy link
Contributor

NivekT commented Jun 2, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Fixes to regressions - we introduced a change that causes noticeable performance issue in the data loading process (specifically IterDataPipe). This cherry-pick addresses that issue.

@atalman merged

@asl3 asl3 unpinned this issue Jun 2, 2022
@atalman atalman pinned this issue Jun 2, 2022
@yuguo68
Copy link
Contributor

yuguo68 commented Jun 2, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:
silent correctness, one liner fix to a missing gradient info issue

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 2, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fixes

@atalman merged

@VitalyFedyunin
Copy link
Contributor

VitalyFedyunin commented Jun 2, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • DataLoader backward compatibility correctness

@atalman merged

@kshitij12345
Copy link
Collaborator

kshitij12345 commented Jun 3, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • chalf support is still experimental. The PR warns user when creating chalf tensor.

cc: @anjali411

@atalman megred

@mrshenli
Copy link
Contributor

mrshenli commented Jun 3, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

Critical fixes for: backwards compatibility. See #78549

@atalman merged

@ejguan
Copy link
Contributor

ejguan commented Jun 6, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • DataLoader backward compatibility correctness

@atalman merged

@NivekT
Copy link
Contributor

NivekT commented Jun 6, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Fixes to new features introduced in the most recent minor release - we introduced new functional API names for various DataPipes in torchdata, and this batch correctly aligns the API names.

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 6, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fixes

@atalman merged

@VitalyFedyunin
Copy link
Contributor

VitalyFedyunin commented Jun 6, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • DataLoader backward compatibility correctness

@atalman merged

@eqy
Copy link
Collaborator

eqy commented Jun 8, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical fix for NCCL

CC @ptrblck @ngimel @pritamdamania87

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 9, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Doc changes

@atalman merged

@yuguo68
Copy link
Contributor

yuguo68 commented Jun 9, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • silent correctness, fix a jit wrong operator schema issue

@atalman merged

@kulinseth
Copy link
Collaborator

kulinseth commented Jun 10, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical feature fix for running hf_Bert and many other Transformer networks on MPS backend.

@malfet: merged

@atalman
Copy link
Contributor Author

atalman commented Jun 10, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical FIX

@atalman merged it (athough I don't know if we need to track builder changes here)

@NivekT
Copy link
Contributor

NivekT commented Jun 10, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Typo fix to regressions against the most recent minor release

@malfet: merged

@malfet
Copy link
Contributor

malfet commented Jun 10, 2022

Link to landed master PR:

Link to release branch PR:

Criteria Category:

  • Fixes regression from previous release

@malfet: merged

@atalman
Copy link
Contributor Author

atalman commented Jun 13, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fixes

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 13, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fixes

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 13, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI fixes

@atalman merged

@atalman
Copy link
Contributor Author

atalman commented Jun 13, 2022

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI fixes

@malfet merged

@atalman
Copy link
Contributor Author

atalman commented Jun 13, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fixes

@atalman merged

@ngimel
Copy link
Collaborator

ngimel commented Jun 13, 2022 •
edited by atalman
Loading

Landed master PR #79401
Cherry pick #79455
Criteria: documentation improvements

@atalman merged

@mcarilli
Copy link
Collaborator

mcarilli commented Jun 13, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical fix (without this PR, CUDA graph capture of Adam and AdamW produce silently incorrect results)

@atalman merged

@anjali411
Copy link
Contributor

anjali411 commented Jun 14, 2022 •
edited by atalman
Loading

Link to landed master PR: #78753

Link to release branch PR: #79481

Criteria Category: docs

@atalman merged

@rohan-varma
Copy link
Member

rohan-varma commented Jun 14, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical fix (silent correctness)

@malfet: merged after additional discussion assessing risks(low, as issue requires opt in to be accessed)

@atalman
Copy link
Contributor Author

atalman commented Jun 14, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI Fix

@malfet merged

@atalman
Copy link
Contributor Author

atalman commented Jun 14, 2022

Please note we are currently in Phase 2 (after 06/13): perform extended integration/stability/performance testing based on Release Candidate builds. Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

@eqy
Copy link
Collaborator

eqy commented Jun 14, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

documentation improvements

@malfet: merged

@ejguan
Copy link
Contributor

ejguan commented Jun 14, 2022 •
edited by malfet
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

--
@malfet: merged

@malfet
Copy link
Contributor

malfet commented Jun 15, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical FIX

Cherry-picked #78326 (as 1d5ecdb) into release 1.12, which should have been done as related change

@malfet merged

@jbschlosser
Copy link
Contributor

jbschlosser commented Jun 15, 2022 •
edited by atalman
Loading

Link to landed master PRs: #79545 and #79549
Link to release branch PRs: #79610 and #79611
Criteria Category: Critical fix for silent BC-breaking behavior

@atalman merged

@erichan1
Copy link
Member

erichan1 commented Jun 16, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

@malfet merged

@ejguan
Copy link
Contributor

ejguan commented Jun 20, 2022 •
edited
Loading

Link to landed master PR (if applicable):

cc: @atalman

@seemethere
Copy link
Member

seemethere commented Jun 20, 2022 •
edited by atalman
Loading

Link to landed master PR (if applicable):

  • n/a

Link to release branch PR:

Criteria Category:

@atalman merged

@malfet
Copy link
Contributor

malfet commented Jun 20, 2022 •
edited by atalman
Loading

Link to landed master PR:

Link to release branch PR:

Criteria Category:

  • Critical infra fixes: restores tagged doc builds on release branch

@malfet merged

@atalman
Copy link
Contributor Author

atalman commented Jun 21, 2022 •
edited
Loading

Link to landed master PR (if applicable):

Link to release branch PR:

Criteria Category:

  • Critical CI fix

@atalman merged

@abhudev abhudev unpinned this issue Jun 27, 2022
@atalman
Copy link
Contributor Author

atalman commented Jun 30, 2022

Release is completed, closing this

@atalman atalman closed this as completed Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
oncall: releng In support of CI and Release Engineering triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
1.12.0
Development

No branches or pull requests