Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed memory leak from opt return #1528

Merged
merged 19 commits into from
Apr 19, 2020
Merged

fixed memory leak from opt return #1528

merged 19 commits into from
Apr 19, 2020

Conversation

williamFalcon
Copy link
Contributor

@williamFalcon williamFalcon commented Apr 19, 2020

Fixes #1510
and a few other things along the way.

Now uses .detach to track metrics instead of .item() which keeps speed fast but decreases memory.

Also speeds things up:
Before this PR lightning slowed an epoch by 0.89 seconds, now only by 0.65 seconds (<1 sec per epoch). This baseline is compared with the same model trained using only PyTorch

@mergify mergify bot requested a review from a team April 19, 2020 14:14
@pep8speaks
Copy link

pep8speaks commented Apr 19, 2020

Hello @williamFalcon! Thanks for updating this PR.

Line 114:111: E501 line too long (113 > 110 characters)

Comment last updated at 2020-04-19 20:23:02 UTC

@codecov
Copy link

codecov bot commented Apr 19, 2020

Codecov Report

Merging #1528 into master will increase coverage by 0%.
The diff coverage is 92%.

@@          Coverage Diff           @@
##           master   #1528   +/-   ##
======================================
  Coverage      91%     91%           
======================================
  Files          67      68    +1     
  Lines        3782    3790    +8     
======================================
+ Hits         3437    3445    +8     
  Misses        345     345           

@williamFalcon williamFalcon merged commit ae2e14e into master Apr 19, 2020
@Borda Borda added the bug Something isn't working label Apr 20, 2020
@Borda Borda added this to the 0.7.4 milestone Apr 20, 2020
@Borda Borda deleted the detach branch April 21, 2020 20:02
@Borda Borda modified the milestones: 0.7.4, v0.7.x Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory (CPU and GPU) leaks during the 1st epoch
3 participants