Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[functorch] add new ensembling api, demonstrate in example #88850

Closed
wants to merge 16 commits into from
Closed

[functorch] add new ensembling api, demonstrate in example #88850

wants to merge 16 commits into from

Conversation

samdow
Copy link
Contributor

@samdow samdow commented Nov 10, 2022 •
edited by pytorchmergebot

@pytorch-bot
Copy link

pytorch-bot bot commented Nov 10, 2022 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88850

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit fd510c3:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@samdow samdow mentioned this pull request Nov 10, 2022
Closed
@samdow samdow changed the title add new ensembling api, demonstrate in example [functorch] add new ensembling api, demonstrate in example Nov 10, 2022
@samdow samdow added topic: docs topic category release notes: functorch release notes category; Pertaining to torch.func or pytorch/functorch labels Nov 10, 2022
@samdow samdow requested a review from zou3519 November 15, 2022 17:29
@samdow samdow mentioned this pull request Nov 17, 2022
Closed
zou3519
zou3519 reviewed Dec 16, 2022
View reviewed changes
Copy link
Contributor

@zou3519 zou3519 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directionally this looks good to me and the code looks correct. We should bikeshed the name of the API before this goes in

torch/_functorch/make_functional.py Outdated Show resolved Hide resolved
torch/_functorch/make_functional.py Outdated Show resolved Hide resolved
torch/_functorch/make_functional.py Outdated Show resolved Hide resolved
functorch/examples/ensembling/parallel_train.py Outdated Show resolved Hide resolved
functorch/examples/ensembling/parallel_train.py
Comment on lines +85 to +86
for key in grad_weights:
new_weights[key] = weights[key] - grad_weights[key] * lr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(no action required) Honestly should just look like the following:

def sgd_update(weight, grad_weight):
    return weight - grad_weight * lr

new_weights = pytree.tree_map(sgd_update, weights, grad_weights)

but our tree_map doesn't have those semantics yet (JAX's does).

functorch/examples/ensembling/parallel_train.py Outdated Show resolved Hide resolved
test/functorch/test_eager_transforms.py Outdated Show resolved Hide resolved
test/functorch/test_eager_transforms.py Outdated Show resolved Hide resolved
test/functorch/test_eager_transforms.py Show resolved Hide resolved
torch/_functorch/make_functional.py Outdated Show resolved Hide resolved
@samdow samdow mentioned this pull request Dec 21, 2022
Closed
zou3519
zou3519 reviewed Dec 22, 2022
View reviewed changes
Copy link
Contributor

@zou3519 zou3519 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had two last two points, then this should be good to go!

  • My first reaction at stack_ensembled_state was some confusion, but I can be convinced
  • Are there tests for the error cases?

torch/_functorch/functional_call.py Outdated Show resolved Hide resolved
torch/_functorch/functional_call.py Outdated Show resolved Hide resolved
torch/_functorch/functional_call.py Outdated Show resolved Hide resolved
@samdow samdow mentioned this pull request Dec 27, 2022
Closed
@zou3519 zou3519 self-requested a review December 28, 2022 14:27
@zou3519
Copy link
Contributor

zou3519 commented Dec 28, 2022

Looks like there are some erroneous submodule changes

zou3519
zou3519 approved these changes Dec 28, 2022
View reviewed changes
Copy link
Contributor

@zou3519 zou3519 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

torch/_functorch/functional_call.py Outdated Show resolved Hide resolved
torch/_functorch/functional_call.py Show resolved Hide resolved
@samdow
Copy link
Contributor Author

samdow commented Dec 29, 2022

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 29, 2022
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: The following mandatory check(s) failed (Rule superuser):

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

@samdow
Copy link
Contributor Author

samdow commented Jan 3, 2023

@pytorchmergebot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased gh/samdow/46/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/88850)

pytorchmergebot pushed a commit that referenced this pull request Jan 3, 2023
add new ensembling api, demonstrate in example
53189e4 
ghstack-source-id: bf8db67fa3b8591d2339466883a7ad00107b47a8
Pull Request resolved: #88850
@samdow
Copy link
Contributor Author

samdow commented Jan 4, 2023

@pytorchmergebot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@facebook-github-bot facebook-github-bot deleted the gh/samdow/46/head branch June 8, 2023 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zou3519 zou3519 zou3519 approved these changes

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: functorch release notes category; Pertaining to torch.func or pytorch/functorch topic: docs topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@samdow @zou3519 @pytorchmergebot