Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix GPU ResourceExhaustedError after many times of Keras model.load() during self-play #31

Merged
merged 2 commits into from Jan 11, 2018

Conversation

@HaoshengZou
Copy link
Contributor

commented Jan 9, 2018

Regarding #30 .
Fix GPU memory leak caused by repeatedly loading Keras models. Fixed by calling keras.backend.clear_session() before each new model loading.

fix GPU memory leak caused by repeatedly loading Keras models, fixed …
…by keras.backend.clear_session() before each new model loading

@HaoshengZou HaoshengZou changed the title #30 GPU ResourceExhaustedError after many times of Keras model.load() during self-play Issue#30 GPU ResourceExhaustedError after many times of Keras model.load() during self-play Jan 9, 2018

@mokemokechicken

This comment has been minimized.

Copy link
Owner

commented Jan 10, 2018

@HaoshengZou

Thank you very much for analysis and very helpful pull request!

But I think that,
K.clear_session() method effects all models(graphs) in the session.
So calling the method in helper methods of load_best_model_weight(model) and reload_newest_next_generation_model_if_changed(model) is a little large impact.
※ And it is better not to change the current behavior.

Whether the K.clear_session() method should be called or not depends on the global context.
So cloud you refine the implementation like this?

def load_best_model_weight(model, clear_session=False):
    ...
    if clear_session:
        K.clear_session()
    return model.load(......)
conditional clear_session, which lets the user decide whether to clea…
…r_session when calling the functions in lib/model_helper.py
@HaoshengZou

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2018

Good point. I've refined the implementation.

@HaoshengZou HaoshengZou changed the title Issue#30 GPU ResourceExhaustedError after many times of Keras model.load() during self-play Fix GPU ResourceExhaustedError after many times of Keras model.load() during self-play Jan 10, 2018

@mokemokechicken mokemokechicken merged commit 2ed9123 into mokemokechicken:master Jan 11, 2018

@mokemokechicken

This comment has been minimized.

Copy link
Owner

commented Jan 11, 2018

@HaoshengZou Thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.