New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not save model using model.save following multi_gpu_model #8446

Closed
SimonWalsh1000 opened this Issue Nov 9, 2017 · 11 comments

Comments

Projects
None yet
8 participants
@SimonWalsh1000
Copy link

SimonWalsh1000 commented Nov 9, 2017

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

  • Check that you are up-to-date with the master branch of Keras. You can update with:
    pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

  • If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

This is a short and simple issue. Following upgrading to Keras 2.0.9 I have been using the multi_gpu_model utility but I can seem to save my models or best weights using model.save.

The error I get is

TypeError: can’t pickle module objects

I suspect there is some problem gaining access to the model object. Is there a work around this issue?

@datumbox

This comment has been minimized.

Copy link
Contributor

datumbox commented Nov 10, 2017

Are you trying to save directly the model which is parallelized? That can be the problem. Could you provide a snippet that shows the steps you are following?

@SimonWalsh1000

This comment has been minimized.

Copy link

SimonWalsh1000 commented Nov 10, 2017

Yes - thats exactly what I am doing - I'm building the model normally, then calling the multi gpu model method function on it and after training, trying to save it with model.save. Does this need to be converted back to a non-parallelized version of the model? If so, how is this achieved?

@fchollet

This comment has been minimized.

Copy link
Collaborator

fchollet commented Nov 10, 2017

For now we recommend saving the original (template) model instead of the parallel model. I.e. call save on the model you passed to multi_gpu_model, not the model returned by it.

Both models share the same weights.

@SimonWalsh1000

This comment has been minimized.

Copy link

SimonWalsh1000 commented Nov 10, 2017

@SimonWalsh1000

This comment has been minimized.

Copy link

SimonWalsh1000 commented Nov 10, 2017

Just to clarify - do you mean call model.save or model.save_weights on the template model at the end of training?

@fchollet

This comment has been minimized.

Copy link
Collaborator

fchollet commented Nov 10, 2017

Yes. Either method should work fine.

@DingkunLiu

This comment has been minimized.

Copy link

DingkunLiu commented Jan 9, 2018

But how can I save the optimizer state by just saving the template model?

@s-zk

This comment has been minimized.

Copy link

s-zk commented Jan 25, 2018

I am on keras 2.1.2 and encountered same problem. I tried follow this answer on StackOverflow, and it works for me. Hope it helps.

@wt-huang

This comment has been minimized.

Copy link
Collaborator

wt-huang commented Nov 13, 2018

Closing as this is resolved

@wt-huang wt-huang closed this Nov 13, 2018

@taalbrecht

This comment has been minimized.

Copy link

taalbrecht commented Nov 28, 2018

Closing as this is resolved

Has this been resolved by a commit and can models and weights be saved as expected when using multi_gpu_model?

@Heisenberg0391

This comment has been minimized.

Copy link

Heisenberg0391 commented Dec 8, 2018

So, as mentioned above, i should train with parallel_model but save the origin model. But what if i want save weights on every epoch as checkpoints using a callback, what should i do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment