classifier.save(save_here, save_format='h5') => RuntimeError: Unable to create link (name already exists)

[alexryan]

I am trying to add a base64 decoding layer to a pretrained model so the model can be deployed to tensorflow serving.

My current understanding is that I need to convert the saved model to hd5 format and then use the keras.estimator.model_to_estimator method do do the surgery like so:
https://stackoverflow.com/a/58003748/2341218

However, this code:

from __future__ import absolute_import, division, print_function, unicode_literals                                                                                                                                  
import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.keras import layers
import numpy as np
import PIL.Image as Image
import sys

classifier_url ="https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/2" #@param {type:"string"}                                                            

IMAGE_SHAPE = (224, 224)

classifier = tf.keras.Sequential([
    hub.KerasLayer(classifier_url, input_shape=IMAGE_SHAPE+(3,))
])

save_here='/Users/alexryan/.keras/models/hd5-tf2/model-and-weights.hd5'
classifier.save(save_here, save_format='h5')

generates this error:

(tf2)  😈   >./save_model.sh 
MODEL_DIR=|/Users/alexryan/.keras/models/hd5-tf2|
/Users/alexryan/.keras/models/hd5-tf2

0 directories, 0 files
2019-10-01 12:11:10.060958: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:132] HadoopFileSystem load error: dlopen(libhdfs.dylib, 6): image not found
2019-10-01 12:11:12.099867: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-01 12:11:12.123701: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fefd12f07d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-10-01 12:11:12.123722: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
  File "save_model.py", line 19, in <module>
    classifier.save(save_here, save_format='h5')
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 986, in save
    signatures, options)
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 631, in save_weights_to_hdf5_group
    param_dset = g.create_dataset(name, val.shape, dtype=val.dtype)
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/h5py/_hl/group.py", line 139, in create_dataset
    self[name] = dset
  File "/Users/alexryan/miniconda3/envs/tf2/lib/python3.7/site-packages/h5py/_hl/group.py", line 373, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)
MODEL_DIR=|/Users/alexryan/.keras/models/hd5-tf2|
/Users/alexryan/.keras/models/hd5-tf2
└── model-and-weights.hd5

0 directories, 1 file
(tf2)  😈   >ls -lF /Users/alexryan/.keras/models/hd5-tf2/
total 16
-rw-r--r-- 1 alexryan staff 14360 Oct  1 12:11 model-and-weights.hd5
(tf2)  😈   >

Here are the package versions I am using:

(tf2)  😈   >pip freeze
absl-py==0.8.0
astor==0.8.0
attrs==19.1.0
certifi==2019.9.11
chardet==3.0.4
cycler==0.10.0
dill==0.3.1.1
future==0.17.1
gast==0.2.2
google-pasta==0.1.7
googleapis-common-protos==1.6.0
grpcio==1.23.0
h5py==2.10.0
idna==2.8
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==3.1.1
numpy==1.17.2
opt-einsum==3.0.1
Pillow==6.1.0
promise==2.2.1
protobuf==3.9.2
psutil==5.6.3
pyparsing==2.4.2
python-dateutil==2.8.0
requests==2.22.0
six==1.12.0
tb-nightly==2.1.0a20191001
tensorboard==2.0.0
tensorflow-datasets==1.2.0
tensorflow-estimator==2.0.0
tensorflow-estimator-2.0-preview==2.0.0
tensorflow-hub==0.6.0
tensorflow-metadata==0.14.0
termcolor==1.1.0
tf-estimator-nightly==1.14.0.dev2019080601
tf-nightly-2.0-preview==2.0.0.dev20191001
tqdm==4.36.1
urllib3==1.25.6
Werkzeug==0.16.0
wrapt==1.11.2
(tf2)  😈   >

[alexryan]

P.S. I was able to save a similar keras model using tf 1.14 like so:

from keras.applications.resnet50 import ResNet50
from keras import optimizers

model = ResNet50(weights='imagenet')
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer=sgd)
save_here='/Users/alexryan/.keras/models/hd5/model-and-weights.hd5'
model.save(save_here)

I am only interested in using an already trained model for predictions on TensorFlow/serving but it still required me to compile the model with a loss function and optimizer in order to save both the weights and the model in the .hd5 file.

[gowthamkpr]

similar to the issue #287 its fixed for those 2 modules but looks like this module is not fixed. Please find the github gist here.

[gowthamkpr]

We have a work around here. Just try saving the model using a tf format instead of .h5 and it works .
classifier.save(save_here, save_format='tf')
Heres my github gist

[arnoegw]

Isn't this an exact duplicate of issue #287?

Module https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/2 used in the repro example was replaced by /3 (and eventually /4) due to that problem. Switching to version /4 or indeed the current https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/4 makes the crash go away.