[doc][hackathon] To add Adadelta Optimizer to the documentation #63255
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iramazanli
added
module: optimizer
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit 05b6943 (more details on the Dr. CI page):
🕵️ 2 new failures recognized by patternsThe following CI failures do not appear to be due to upstream breakages:
|
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
4 times, most recently
from
7499cbf
to
0e157af
Compare
iramazanli
changed the title
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
2 times, most recently
from
9fbbcd5
to
8cf072a
Compare
Codecov Report
@@ Coverage Diff @@
## master #63255 +/- ##
==========================================
- Coverage 66.81% 63.72% -3.10%
==========================================
Files 695 698 +3
Lines 90845 90881 +36
==========================================
- Hits 60701 57912 -2789
- Misses 30144 32969 +2825 |
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
6 times, most recently
from
990ec96
to
23fbbb6
Compare
albanD
reviewed
Sep 7, 2021
torch/optim/adadelta.py
Outdated
| &\rule{110mm}{0.4pt} \\ | ||
| &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, | ||
| \: f(\theta) \text{ (objective)}, \: \rho \text{ (decay)}, \: weightdecay \\ | ||
| &\textbf{initialize} : E[g^2]_0 \leftarrow 0, \: E[\Delta x^2]_0 \leftarrow 0 |
There was a problem hiding this comment.
Why not name these like the other optimizers (like exp_avg) I feel like this would be easier to read?
There was a problem hiding this comment.
sounds good ! and done
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
2 times, most recently
from
800e4c5
to
cdd0cf2
Compare
albanD
reviewed
Sep 9, 2021
torch/optim/adadelta.py
Outdated
| &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, | ||
| \: f(\theta) \text{ (objective)}, \: \rho \text{ (decay)}, | ||
| \: \lambda \text{ (weight decay)} \\ | ||
| &\textbf{initialize} : square\_avg_0 \leftarrow 0, |
There was a problem hiding this comment.
Why not use v_0 like the other algorithms: https://pytorch.org/docs/master/generated/torch.optim.RMSprop.html#torch.optim.RMSprop ?
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
2 times, most recently
from
3cd6c2f
to
3ed039c
Compare
albanD
approved these changes
Sep 10, 2021
|
@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
from
3ed039c
to
79cf319
Compare
|
@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
from
79cf319
to
88dddf9
Compare
|
@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
iramazanli
force-pushed
the
adadelta_algorithm_doc
branch
from
88dddf9
to
05b6943
Compare
|
@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@iramazanli merged this pull request in dafa0a5. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It has been discussed before that adding description of Optimization algorithms to PyTorch Core documentation may result in a nice Optimization research tutorial. In the following tracking issue we mentioned about all the necessary algorithms and links to the originally published paper #63236.
In this PR we are adding description of AdaDelta Algorithm to the documentation. For more details, we refer to the paper here https://arxiv.org/abs/1212.5701
cc @vincentqb @iramazanli