Skip to content

Dropout op elimination - enable for ORT training #5588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ytaous merged 4 commits into from
Oct 27, 2020
Merged

Dropout op elimination - enable for ORT training #5588

ytaous merged 4 commits into from
Oct 27, 2020
+122 −0

Conversation

@ytaous
Copy link
Contributor

@ytaous ytaous commented Oct 26, 2020

Eliminate Dropout op when ratio is 0 for ORT training.

Motivation and Context

  • In the experiment for Mainz model, when ratio = 0, it introduce some overhead with Cuda DtoD memcpy.
  • Allow Dropout elimination when:
  1. ratio input is an initializer of 0
  2. ratio input is not a graph input, so it cannot be overridden

@ytaous ytaous added the training issues related to ONNX Runtime training; typically submitted using template label Oct 26, 2020
@ytaous ytaous requested a review from SherlockNoMad October 26, 2020 01:28
@ytaous ytaous requested a review from a team as a code owner October 26, 2020 01:28
SherlockNoMad
SherlockNoMad reviewed Oct 26, 2020
View reviewed changes
SherlockNoMad
SherlockNoMad previously approved these changes Oct 26, 2020
View reviewed changes
@ytaous ytaous requested a review from SherlockNoMad October 27, 2020 17:35
@ytaous ytaous merged commit 6f824c2 into master Oct 27, 2020
@ytaous ytaous deleted the ettao/dropout branch October 27, 2020 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@SherlockNoMad SherlockNoMad SherlockNoMad approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

training issues related to ONNX Runtime training; typically submitted using template

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ytaous @SherlockNoMad