Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BYOL moving average update #574

Merged
merged 1 commit into from Mar 4, 2021

Conversation

wjn0
Copy link
Contributor

@wjn0 wjn0 commented Mar 4, 2021

No longer exclude biases from the moving average update in BYOL. See discussion on #440

What does this PR do?

Fixes the BYOL moving average callback to update all parameters, no longer excluding biases, to better match the original paper

Fixes #440

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests? [not needed for typos/docs]
  • Did you verify new and existing tests pass locally with your changes?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

  • Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 馃檭

No longer exclude biases from the moving average update in BYOL.

Fixes Lightning-Universe#440
@codecov
Copy link

codecov bot commented Mar 4, 2021

Codecov Report

Merging #574 (592d28f) into master (1bec122) will decrease coverage by 51.79%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master     #574       +/-   ##
===========================================
- Coverage   78.03%   26.23%   -51.80%     
===========================================
  Files         115      115               
  Lines        6701     6674       -27     
===========================================
- Hits         5229     1751     -3478     
- Misses       1472     4923     +3451     
Flag Coverage 螖
cpu 26.23% <0.00%> (+<0.01%) 猬嗭笍
pytest 26.23% <0.00%> (+<0.01%) 猬嗭笍
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage 螖
pl_bolts/callbacks/byol_updates.py 45.45% <0.00%> (-54.55%) 猬囷笍
pl_bolts/models/rl/__init__.py 0.00% <0.00%> (-100.00%) 猬囷笍
pl_bolts/models/rl/common/agents.py 0.00% <0.00%> (-100.00%) 猬囷笍
pl_bolts/models/rl/dueling_dqn_model.py 0.00% <0.00%> (-100.00%) 猬囷笍
pl_bolts/models/rl/double_dqn_model.py 0.00% <0.00%> (-95.66%) 猬囷笍
...l_bolts/models/rl/vanilla_policy_gradient_model.py 0.00% <0.00%> (-95.50%) 猬囷笍
pl_bolts/models/rl/common/networks.py 0.00% <0.00%> (-89.25%) 猬囷笍
pl_bolts/models/rl/reinforce_model.py 0.00% <0.00%> (-88.62%) 猬囷笍
pl_bolts/models/rl/per_dqn_model.py 0.00% <0.00%> (-86.67%) 猬囷笍
pl_bolts/models/rl/common/gym_wrappers.py 0.00% <0.00%> (-85.50%) 猬囷笍
... and 64 more

Continue to review full report at Codecov.

Legend - Click here to learn more
螖 = absolute <relative> (impact), 酶 = not affected, ? = missing data
Powered by Codecov. Last update 1bec122...ed4d252. Read the comment docs.

@wjn0
Copy link
Contributor Author

wjn0 commented Mar 4, 2021

pinging @annikabrundyn for review :)

@akihironitta akihironitta added fix fixing issues... model labels Mar 4, 2021
@akihironitta akihironitta added this to In progress in Self Supervised Learning via automation Mar 4, 2021
@annikabrundyn
Copy link
Contributor

looks good to me. @ananyahjha93 mind taking a look too? this makes sense for using either resnet50 or vgg16 as backbone but if other custom encoders are provided do we want to make it specific for named parameters containing 'weight' or 'bias'?

@Borda Borda merged commit 2911253 into Lightning-Universe:master Mar 4, 2021
Self Supervised Learning automation moved this from In progress to Done Mar 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
callback fix fixing issues... model
Development

Successfully merging this pull request may close these issues.

Biases are excluded from BYOL moving average update
5 participants