-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
Setting log_gpu_memory='min_max' in Trainer leads to the following bug.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 347, in fit
self.single_gpu_train(model)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/dp_mixin.py", line 79, in single_gpu_train
self.run_pretrain_routine(model)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 467, in run_pretrain_routine
self.train()
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 60, in train
self.run_training_epoch()
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 126, in run_training_epoch
self.log_metrics(batch_step_metrics, grad_norm_dic)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/logging_mixin.py", line 20, in log_metrics
mem_map = memory.get_memory_profile(self.log_gpu_memory)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/root_module/memory.py", line 205, in get_memory_profile
for k, v in memory_map:
ValueError: too many values to unpack (expected 2)
To Reproduce
On current master, execute the following.
trainer = Trainer(
...
log_gpu_memory='min_max',
...
)
trainer.fit(model)
Expected behavior
Log the min/max utilization of gpu memory, as min_max option is documented.
Desktop (please complete the following information):
- OS: Ubuntu 18.04
- Version: Current master
I am working on this issue. Will submit a PR soon.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working