Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not save model using model.save following multi_gpu_model #8446

Closed
2 tasks done
SimonWalsh1000 opened this issue Nov 9, 2017 · 12 comments
Closed
2 tasks done

Can not save model using model.save following multi_gpu_model #8446

SimonWalsh1000 opened this issue Nov 9, 2017 · 12 comments

Comments

@SimonWalsh1000
Copy link

@SimonWalsh1000 SimonWalsh1000 commented Nov 9, 2017

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

  • Check that you are up-to-date with the master branch of Keras. You can update with:
    pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

  • If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

This is a short and simple issue. Following upgrading to Keras 2.0.9 I have been using the multi_gpu_model utility but I can seem to save my models or best weights using model.save.

The error I get is

TypeError: can’t pickle module objects

I suspect there is some problem gaining access to the model object. Is there a work around this issue?

@datumbox
Copy link
Contributor

@datumbox datumbox commented Nov 10, 2017

Are you trying to save directly the model which is parallelized? That can be the problem. Could you provide a snippet that shows the steps you are following?

@SimonWalsh1000
Copy link
Author

@SimonWalsh1000 SimonWalsh1000 commented Nov 10, 2017

Yes - thats exactly what I am doing - I'm building the model normally, then calling the multi gpu model method function on it and after training, trying to save it with model.save. Does this need to be converted back to a non-parallelized version of the model? If so, how is this achieved?

@fchollet
Copy link
Member

@fchollet fchollet commented Nov 10, 2017

For now we recommend saving the original (template) model instead of the parallel model. I.e. call save on the model you passed to multi_gpu_model, not the model returned by it.

Both models share the same weights.

@SimonWalsh1000
Copy link
Author

@SimonWalsh1000 SimonWalsh1000 commented Nov 10, 2017

@SimonWalsh1000
Copy link
Author

@SimonWalsh1000 SimonWalsh1000 commented Nov 10, 2017

Just to clarify - do you mean call model.save or model.save_weights on the template model at the end of training?

@fchollet
Copy link
Member

@fchollet fchollet commented Nov 10, 2017

Yes. Either method should work fine.

@DingkunLiu
Copy link

@DingkunLiu DingkunLiu commented Jan 9, 2018

But how can I save the optimizer state by just saving the template model?

@s-zk
Copy link

@s-zk s-zk commented Jan 25, 2018

I am on keras 2.1.2 and encountered same problem. I tried follow this answer on StackOverflow, and it works for me. Hope it helps.

@wt-huang
Copy link

@wt-huang wt-huang commented Nov 13, 2018

Closing as this is resolved

@wt-huang wt-huang closed this Nov 13, 2018
@taalbrecht
Copy link

@taalbrecht taalbrecht commented Nov 28, 2018

Closing as this is resolved

Has this been resolved by a commit and can models and weights be saved as expected when using multi_gpu_model?

@Heisenberg0391
Copy link

@Heisenberg0391 Heisenberg0391 commented Dec 8, 2018

So, as mentioned above, i should train with parallel_model but save the origin model. But what if i want save weights on every epoch as checkpoints using a callback, what should i do?

@jashshah
Copy link

@jashshah jashshah commented Jun 19, 2019

Are there any updates regarding this either on keras or on tf.keras?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants