Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change min max gpu memory to be on their own plots #1358

Merged
merged 1 commit into from Apr 3, 2020

Conversation

tullie
Copy link
Contributor

@tullie tullie commented Apr 3, 2020

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

The original min_max gpu memory functionality has unexpected behavior where the min and max memory is plotted in the gpu_{gpu_index} plot. This doesn't make sense when the min and max gpu usage switches between GPUs during training as the plot is uninterpretable. It's also a waste of space having a gpu_{gpu_index} plot for each master node gpu when only the gpus that have the max and min memory have data points.

This PR changes the logger to have two plots: "min_gpu_mem", "max_gpu_mem", which contain the min and max gpu memory plots respectively.

@mergify mergify bot requested a review from a team April 3, 2020 13:21
Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@Borda Borda added the feature Is an improvement or enhancement label Apr 3, 2020
@Borda Borda added this to the 0.7.2 milestone Apr 3, 2020
@tullie tullie force-pushed the master branch 2 times, most recently from 0cf9d62 to d841f03 Compare April 3, 2020 14:30
@williamFalcon williamFalcon merged commit b31edf3 into Lightning-AI:master Apr 3, 2020
alexeykarnachev pushed a commit to alexeykarnachev/pytorch-lightning that referenced this pull request Apr 4, 2020
@Borda Borda mentioned this pull request Apr 4, 2020
@Borda Borda modified the milestones: v0.7., v0.7.x Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants