Keras - Supporting load/save models and weights to Google Storage

[kim-sardine]

I want to save / load my keras model to / from Google storage bucket.
my environment : docker image - tensorflow/tensorflow-latest-py3 (tensorflow 2.1.0, python 3.6.9)

from tensorflow.keras.models import Sequential

model = Sequential([
...
...
])

model.compile(...)
model.fit(...)

model.save('gs://path/to/my/bucket/model.h5')

and I got this error message

Traceback (most recent call last):
  File "main.py", line 118, in <module>
    model.save('gs://MY-BUCKET/model.h5')
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
    signatures, options)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 92, in save_model_to_hdf5
    f = h5py.File(filepath, mode='w')
  File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 179, in make_fid
    fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 108, in h5py.h5f.create
OSError: Unable to create file (unable to open file: name = 'gs://MY-BUCKET/model.h5', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 242)

I found this PR in keras repository, supporting above behavior. but It seems like that it's not implemented in tensorflow.keras.

Do you have plans to support it? or, are there any alternatives in tensorflow?

[jvishnuvardhan]

@kim-sardine Please check this resource which explains clearly how to save the model in the Google Storage. Thanks!

[kim-sardine]

I just wondered if I can save model in Google Storage in one line.
so for now, I have to do this by overriding ModelCheckpoint callback function and making new model saving function.
I'll refer to the link above, thanks.

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No

[kim-sardine]

BTW, It won't be easy because MoodelCheckpoint is using not only save(), but also load_weights, save_weights.

[lminer]

This feature as been enabled in regular keras. Shouldn't be hard to move it here.

[avasbr]

Having a one liner to load from GCS buckets would be highly desirable. It seems like a regression especially given that saver.restore(sess, gs://path_to_checkpoint) worked in the past, and using model.load_weights() works in regular keras, but this behavior is not maintained for tf.keras. Any plans to support this moving forward?

[bgheneti]

This feature would be extremely helpful.

[lgeiger]

@gadagashwini @jvishnuvardhan I think this issue should be reopend, as the stack overflow post you linked only worksaround the fact that
model.save('gs://bucket/model.h5') isn't natively supported, whereas the TF file format model.save('gs://bucket/model') works as expected.
In my opinion, inconsistencies like this severly hurt to usability of TensorFlow especially in combination with GCS so it would be great if this issue could be reopened.

[jvishnuvardhan]

@lgeiger Sure. Reopening this issue considering the comments above. Thanks!

[lgeiger]

@k-w-w This issue has been tagged as 2.1, has there been any progress on it so far?

[hav4ik]

What is the status of this one so far? Would love to jump in.

[sayakpaul]

I have been able to serialize SavedModels to GCS Bucket folders directly from AI Platform Notebooks in the following manner:

tf.saved_model.save(model, export_module_dir)

where export_module_dir is either a GCS Bucket or a SavedModel name inside a GCs Bucket. Of course, you would need to have write access to the bucket to be able to do this. I don't think that is problematic to configure.

One can load back the model similarly like -

model = tf.saved_model.load(export_module_dir)

Here's an example that benefits from this.

This is doable from a Colab Notebook as well provided you have performed the authorization steps. This should look something like the following:

from google.colab import auth
auth.authenticate_user()

I hope this helps.

[lgeiger]

@sayakpaul That is correct, the SavedModel format correctly supports GCS, however when saving a .h5 file GCS is still not supported.

[sayakpaul]

True that. On a personal level, I think serializing your model as a SavedModel is more uniform with respect to the TensorFlow ecosystem.

[aginpatrick]

@sayakpaul Could you tell me how to save model in SavedModel format at each checkpoint?

[gowthamkpr]

@aginpatrick You can store Keras model checkpoints to google storage by creating a custom GCS callback as shown here

[aginpatrick]

It seems it's now possible to save model to Google Storage without the GCS callback. The problem I have now is that the custom signatures that I've created are not saved. My post on SO is here <https://stackoverflow.com/questions/74747394/keras-model-custom-signatures-are-not-saved-when-learning-is-done-on-gcp>.

…

On Thu, Dec 15, 2022 at 5:54 PM gowthamkpr ***@***.***> wrote: @aginpatrick <https://github.com/aginpatrick> You can store Keras model checkpoints to google storage by creating a custom GCS callback as shown here <https://www.kaggle.com/code/subrahmanyajoshi/save-keras-model-checkpoints-to-cloud-storage> — Reply to this email directly, view it on GitHub <#36453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM6JSC3UN7YQAJI6NID57LWNOOQ7ANCNFSM4KPQHCUQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>