-
Notifications
You must be signed in to change notification settings - Fork 19.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clearing GPU memory in Keras #12625
Comments
This function looks promising, stolen from fastai forums from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow
# Reset Keras Session
def reset_keras():
sess = get_session()
clear_session()
sess.close()
sess = get_session()
try:
del classifier # this is from global space - change this as you need
except:
pass
print(gc.collect()) # if it's done something you should see a number being outputted
# use the same config as you used to create the session
config = tensorflow.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 1
config.gpu_options.visible_device_list = "0"
set_session(tensorflow.Session(config=config)) |
@nateraw |
I keep getting "CUDA_ERROR_OUT_OF_MEMORY" when running the above function. |
Thanks! Works perfect!! |
running the above reset_keras() function still throws an OOM error on Ubuntu with 16GB of GPU memory.
TensorFlow installed from conda install tensorflow-gpu Same when I try running:
I find it fascinating that the TensorFlow team has not made a very straightforward way to clear GPU memory from a session. So much is broken with TF. Little annoyances like this; a user reasonably expects TF to handle clearing CUDA memory or have memory leaks, yet there appears no explicit way to handle this. Even
After several hours of scouring StackOverflow and the Github issues, and trying the above approaches (none of which worked for some reason), I'm left with the decidedly inelegant approach of restarting the entire kernel. Frustrating. |
Why do they close issues that are not solved like at all? |
Having this issue too. Restarting doesn't always fix it either. Even shutting down Jupyter doesn't necessarily fix it. Using Windows, no idea how to fix this when it pops up. Any time I restart the kernel with TF on it there's a random chance it'll happen after restarting. |
Is this still happening if you use |
Any solution, yet ? |
I have to do the same |
Tensorflow version: 2.1.0 |
It works for me. Google colab with TF 2.3.0
|
Thanks a lot for sharing your code in TF 2.0, but I still cannot fix it up with TF==2.2.0& keras ==2.3.1. And after I run your code, the console shows following messages BUT GPU still almost full which confuses me a lot :
May I ask you have you ever met this issue? |
80% my GPU memory get's full after loading pre-trained Xception model. but after deleting my model , memory doesn't get empty or flush.
I've also used codes like : K.clear_session() , gc.collect() , tf.reset_default_graph() , del model but none of them worked. Gpu properties say's 85% of memory is full.
Nothing flush gpu memory except numba.cuda.close() but won't allow me to use my gpu again. The only way to clear it is restarting kernel and rerun my code.
I'm looking for any script code to add my code allow me to use my code in for loop and clear gpu in every loop.
Part of my code :
image_input = Input(shape=(224, 224, 3))
base_model = Xception(input_tensor=image_input, include_top=False,weights='imagenet')
base_model.compile(loss='categorical_crossentropy',optimizer='adadelta',metrics=['accuracy'])
hist = base_model.fit(X,Y,epochs=2)
System information
The text was updated successfully, but these errors were encountered: