tensorflow won't release system memroy

[xiaorongfan]

Hi all,

I started to use tensorflow one month ago. I briefly use it to train neural machine translation models.
I find that after several times of training, there were a huge amount of memory not released. I have to reboot my machine after some numbers of training to release those memory.

I use the sample code( tensorflow/models/rnn/translate python translate.py) to train the model.

Environment info

Operating System: Ubuntu 14.04.4 LTS

tensorflow version: r.0.10

CUDA version: libcudnn.so.4.0
If installed from binary pip package, provide:

What have you tried?

Add a maximum step number to terminate training normally.

1. replace

   with tf.Session() as sess:

with the following code:

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

with tf.Graph().as_default(),sess:

1. close the session after training
   sess.close()
   del sess
2. the version of my nvidia driver is 352.79 now.
   I will update it to the latest version next week.

I have no idea what to do now.

Thanks!
Best,
xiaorongfan

[xiaorongfan]

1. Add a maximum step number to terminate training normally.

2. replace

   with tf.Session() as sess:

with the following code:

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

with tf.Graph().as_default(),sess:
3. close the session after training
sess.close()
del sess
4. the version of my nvidia driver is 352.79 now.
I will update it to the latest version next week.

[hyu-leeky]

I had a same issue like you. After upgrading the GPU driver from 352.79 to 367.35 (the newest one), the problem disappeared.

[girving]

@xiaorongfan Can you check if that driver update fixes it for you too?

[xiaorongfan]

kmu-leeky
Thank you for you information. Because the administrator is on vacation now, I can only upgrade GPU driver next week. I will report the result later.

[hyu-leeky]

@xiaorongfan my pleasure
@girving Out of curiosity, do you know what is the root cause for this issue? How does the old version of GPU driver and Tensorflow system memory management conflicts? If you can redirect me to an article to elaborate it, it would be appreciated.

[girving]

@kmu-leeky Unfortunately I do not know the reason, or of such an article.

[vrv]

My guess is that we aggressively use host-pinned GPU memory to make GPU->CPU transfers faster, and there was probably a driver bug that leaks the memory allocated in this fashion. But impossible to know for sure since cuda driver is closed source :)