New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU Memory not released after running code with "build" command #2184

Closed
malbergo opened this Issue Feb 8, 2018 · 4 comments

Comments

3 participants
@malbergo

malbergo commented Feb 8, 2018

When submitting a feature or enhancement request:

  1. Explain briefly what the enhancement is and why you think it would be useful.
  2. Provide any other necessary or useful information regarding your issue, such as (code) examples or related links.

When submitting a bug report, please follow the following template:


Summary

When I run a simply GPU compatible package program in Sublime Text (like PyTorch or Tensorflow doing one matrix multiplication or something) with the "build" command (ctrl + b), after the program is done running, the memory usage for python is still present in my GPUS.

I am using NVIDIA Titan Xp GPUs. I experience output like this:

 nvidia-smi
Thu Feb  8 12:36:59 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.34                 Driver Version: 387.34                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:0B:00.0 Off |                  N/A |
| 31%   54C    P2    88W / 250W |   3723MiB / 12189MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:41:00.0 Off |                  N/A |
| 39%   64C    P2    81W / 250W |    458MiB / 12189MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN Xp            Off  | 00000000:42:00.0  On |                  N/A |
| 23%   37C    P8    17W / 250W |    508MiB / 12186MiB |     11%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     30427      C   python                                       407MiB |
|    0     44580      C   /usr/bin/python3                             355MiB |
|    0     44618      C   /usr/bin/python3                             355MiB |
|    0     44643      C   /usr/bin/python3                             389MiB |
|    0     44668      C   /usr/bin/python3                             355MiB |
|    0     44691      C   /usr/bin/python3                             389MiB |
|    0     44718      C   /usr/bin/python3                             429MiB |
|    0    122351      C   /usr/NX/bin/nxnode.bin                       204MiB |
|    1     30427      C   python                                       383MiB |
|    2      1223      G   /usr/lib/xorg/Xorg                           214MiB |
|    2      1891      G   compiz                                       218MiB |
+-----------------------------------------------------------------------------+

...

Expected behavior

I would expect the GPU memory to be released, as it is with any other Tensorflow or PyTorch program I run when I do not run it through Sublime's "build" command and output.

When I completely shut down Sublime, I get the behavior I would hope for:

nvidia-smi
Thu Feb  8 12:43:31 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.34                 Driver Version: 387.34                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:0B:00.0 Off |                  N/A |
| 31%   53C    P2    88W / 250W |   1402MiB / 12189MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:41:00.0 Off |                  N/A |
| 39%   64C    P2    80W / 250W |    410MiB / 12189MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN Xp            Off  | 00000000:42:00.0  On |                  N/A |
| 23%   36C    P8    17W / 250W |    460MiB / 12186MiB |      8%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     30427      C   python                                       407MiB |
|    0    122351      C   /usr/NX/bin/nxnode.bin                       204MiB |
|    1     30427      C   python                                       383MiB |
|    2      1223      G   /usr/lib/xorg/Xorg                           203MiB |
|    2      1891      G   compiz                                       228MiB |
+-----------------------------------------------------------------------------+


Actual behavior

Actual behavior is noted in summary.

Steps to reproduce

  1. Make small PyTorch / Tensorflow program. Something as simple as:
import numpy as np
import torch

x = torch.cuda.FloatTensor(1000,50000).normal_()
y = torch.cuda.FloatTensor(20, 50000).random_()
print(x)	

z = torch.mm(x,y.t())

print(z)	
  1. check nvidia GPU usage by running from the command line nvidia-smi

  2. See if these memory values are still there after running build command on program in Sublime.

Environment

  • Operating system and version: Ubuntu 16.04,
  • Sublime Text:
    • Build 3
@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 8, 2018

Member

The build system functionality in Sublime Text executes external programs. Sublime Text reads the stdout and stderr file handles of the external process and displays them in a buffer. If you are seeing side effects from the external programs you are running, you would need to resolve it via the external program, not Sublime Text.

My hunch is that perhaps the python memory usage you are seeing around is because the Python processes are not exiting for some reason? Perhaps check your process list and see if they are still around?

Member

wbond commented Feb 8, 2018

The build system functionality in Sublime Text executes external programs. Sublime Text reads the stdout and stderr file handles of the external process and displays them in a buffer. If you are seeing side effects from the external programs you are running, you would need to resolve it via the external program, not Sublime Text.

My hunch is that perhaps the python memory usage you are seeing around is because the Python processes are not exiting for some reason? Perhaps check your process list and see if they are still around?

@malbergo

This comment has been minimized.

Show comment
Hide comment
@malbergo

malbergo Feb 8, 2018

Thanks for your response. Yeah, that's what I was taking a look at. The processes aren't exiting when I execute them through Sublime, but they exit fine through other mediums (Jupyter notebooks, terminal, Atom) so I thought there might be some compatibility issue with how they are being executed in Sublime, but I'll take a closer look.

malbergo commented Feb 8, 2018

Thanks for your response. Yeah, that's what I was taking a look at. The processes aren't exiting when I execute them through Sublime, but they exit fine through other mediums (Jupyter notebooks, terminal, Atom) so I thought there might be some compatibility issue with how they are being executed in Sublime, but I'll take a closer look.

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Feb 8, 2018

Member

You can take an exact look at the code running the processes in the Default package under Default/exec.py. It's basically a call to subprocess.Popen with some flags and, depending on whether you used shell_cmd, potentially having your shell take care of running the exact args. Since you're known to Python, you might be able to troubleshoot a bit.

Member

FichteFoll commented Feb 8, 2018

You can take an exact look at the code running the processes in the Default package under Default/exec.py. It's basically a call to subprocess.Popen with some flags and, depending on whether you used shell_cmd, potentially having your shell take care of running the exact args. Since you're known to Python, you might be able to troubleshoot a bit.

@malbergo

This comment has been minimized.

Show comment
Hide comment
@malbergo

malbergo Feb 12, 2018

Thanks for this advice. I'll play with that a bit.

Michael

malbergo commented Feb 12, 2018

Thanks for this advice. I'll play with that a bit.

Michael

@wbond wbond added the Support label Mar 1, 2018

@FichteFoll FichteFoll closed this Jun 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment