Skip to content

min_max log_gpu_memory option bug #452

@dreamgonfly

Description

@dreamgonfly

Describe the bug
Setting log_gpu_memory='min_max' in Trainer leads to the following bug.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 347, in fit
    self.single_gpu_train(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/dp_mixin.py", line 79, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 467, in run_pretrain_routine
    self.train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 60, in train
    self.run_training_epoch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 126, in run_training_epoch
    self.log_metrics(batch_step_metrics, grad_norm_dic)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/logging_mixin.py", line 20, in log_metrics
    mem_map = memory.get_memory_profile(self.log_gpu_memory)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/root_module/memory.py", line 205, in get_memory_profile
    for k, v in memory_map:
ValueError: too many values to unpack (expected 2)

To Reproduce
On current master, execute the following.

    trainer = Trainer(
        ...
        log_gpu_memory='min_max',
        ...
    )
    trainer.fit(model)

Expected behavior
Log the min/max utilization of gpu memory, as min_max option is documented.

Desktop (please complete the following information):

  • OS: Ubuntu 18.04
  • Version: Current master

I am working on this issue. Will submit a PR soon.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions