Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v1.7 patch] Prioritize raising error message about unused parameters when rebuild_buckets fails #46099

Merged
merged 1 commit into from
Oct 12, 2020

Conversation

rohan-varma
Copy link
Member

Summary:

Note: This PR has been merged into master at 62554a3 after the 1.7 branch cut
(see original PR: #45933). This PR is to merge it into the 1.7 branch.

---- Original Commit Description Follows ---

#45933 Prioritize raising error message about unused parameters when rebuild_buckets fails

Occasionally users run DDP with models with unused params, in this
case we would like to surface an error message telling them to run with
find_unused_params=True. However, a recent change to rebuild_buckets logic (#44798) made
it so that we raise a size mismatch error when this happens, but the
information about unused parameters is likely to be more useful and likely to
be the most common case of failure. Prefer raising this error over the
subsequent size mismatch errors.

Differential Revision: D24151256

@facebook-github-bot facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 9, 2020
@dr-ci
Copy link

dr-ci bot commented Oct 9, 2020

💊 CI failures summary and remediations

As of commit 786b9e3 (more details on the Dr. CI page):



❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_ios_12_0_0_x86_64_build (1/1)

Step: "Update Homebrew" (full log | diagnosis details | 🔁 rerun) ❄️

fatal: Could not read from remote repository.
remote: Total 60 (delta 51), reused 21 (delta 17), pack-reused 0         
Unpacking objects:  96% (58/60) Unpacking objects:  98% (59/60) Unpacking objects: 100% (60/60) Unpacking objects: 100% (60/60), 13.00 KiB | 302.00 KiB/s, done. 
From ssh://github.com/Homebrew/homebrew-cask-versions 
 + f127a2be3...39b210e60 master     -> origin/master  (forced update) 
+ git reset --hard origin/master 
HEAD is now at 39b210e60 Update brave-browser-dev from 86.1.16.55,116.55 to 86.1.16.57,116.57 (#9744) 
+ for path in '$(find /usr/local/Homebrew -type d -name .git)' 
+ cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/.. 
+ git fetch --depth=1 origin 
Connection to github.com closed by remote host.  
fatal: Could not read from remote repository. 
 
Please make sure you have the correct access rights 
and the repository exists. 

ci.pytorch.org: 1 failed


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 2 times.

@malfet malfet merged commit a0a65a9 into release/1.7 Oct 12, 2020
@facebook-github-bot facebook-github-bot deleted the cherry_pick_reducer_error_fix branch January 27, 2021 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
oncall: distributed Add this issue/PR to distributed oncall triage queue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants