Model weight not saved when exporting SavedModel #59287

BorisPolonsky · 2023-01-18T03:30:54Z

Click to expand!

Issue Type

Support

Have you reproduced the bug with TF nightly?

No

Source

binary

Tensorflow Version

tf2.8

Custom Code

Yes

OS Platform and Distribution

WSL2+tensorflow/tensorflow:2.8.0-gpu-jupyter image

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

I need to change the signature in SavedModel by loading and re-export it with different configuration. The new exported model lost track of the model weight. Here's what I've tried
- Load the SavedModel (>300MB, with two signatures, exported in TF1.15) with TF2.8
- Select the signature I need
- Export Model with new signature config
- Load Model And Test

The re-exported model can be successfully loaded. But it won't be able to make any prediction due to initialized variable. I checked the exported model and the size is approximately 300kb, which means that most of the model weights were lost.

I looked into this answer that solves the problem on his case. To me the only difference is that he converted variables in the original variable to constants. But for this approach the original .ckpt file is a prerequisite, which is not my case

Standalone code to reproduce the issue

Script for reproducing the issue

# https://stackoverflow.com/questions/57048064/saved-model-prune-in-tf2-0
# https://stackoverflow.com/questions/59504276/resave-tf1-x-saved-model-pb-into-new-tf2-0-saved-model-pb
import tensorflow as tf
import numpy as np
import os

os.environ["CUDA_VISIBLE_DEVICES"] = ""
export_path_origin = "baidu-ske-2019-len128"  # A Model with multiple signatures
assert tf.__version__[0] == '2'
print("test")

imported = tf.saved_model.load(export_dir=export_path_origin)
print(imported.signatures)
f_serving_default = imported.signatures["serving_default"]
f_predict = imported.signatures["predict"]


inputs = {
    "input_ids": tf.constant([[0] * 128]),
    "segment_ids": tf.constant([[0] * 128]),
    "input_mask":  tf.constant([[0] * 128])
}
serving_default_output = f_serving_default(**inputs)
# print(serving_default_output)
predict_output = f_predict(**inputs)
# print(predict_output)

signature_to_keep = f_predict

fetches = {k: v.name for k, v in signature_to_keep.structured_outputs.items()}
feeds = [input_.name for input_ in signature_to_keep.inputs]
new_fn = imported.prune(feeds, fetches=fetches)
new_fn.graph.finalize()
original_output = signature_to_keep(**inputs)
new_outputs = new_fn(**inputs)
print(new_outputs)

# Assert equal
for k in new_outputs:
    t_origin, t_new = original_output[k], new_outputs[k]
    assert t_origin.dtype == t_new.dtype
    if t_origin.dtype.is_floating:
        np.testing.assert_allclose(t_origin.numpy(), t_new.numpy())
    else:
        assert np.all(t_origin.numpy() == t_new.numpy())



class Exportable(tf.Module):
    def __init__(self, fn):
        super(Exportable, self).__init__()
        self.tf1_model = fn

    tf.function(input_signature=[tf.TensorSpec([None, 128], tf.int32), tf.TensorSpec([None, 128], tf.int32), tf.TensorSpec([None, 128], tf.int32)])
    def forward(self, input_ids, segment_ids, input_mask):
        with tf.init_scope():
            out = self.tf1_model(input_ids=input_ids,
                                segment_ids=segment_ids, input_mask=input_mask)
        return out


new_saved_model = Exportable(new_fn)

print(new_saved_model.forward(**inputs))    # Eager Execution Works
new_model_path = './new_saved_model'

tf.saved_model.save(new_saved_model, new_model_path)
imported = tf.saved_model.load(export_dir=new_model_path)
imported.signatures["serving_default"](**inputs) # Error: Uninitialized variables

Here's the metadata of the original model from by tensorflow/serving

{
    "model_spec":{
     "name": "default",
     "signature_name": "",
     "version": "1"
    }
    ,
    "metadata": {"signature_def": {
     "signature_def": {
      "predict": {
       "inputs": {
        "segment_ids": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "segment_ids:0"
        },
        "input_mask": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "input_mask:0"
        },
        "input_ids": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "input_ids:0"
        }
       },
       "outputs": {
        "token_label_predictions": {
         "dtype": "DT_INT64",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "token_label_loss/ArgMax:0"
        },
        "predicate_head_probabilities": {
         "dtype": "DT_FLOAT",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           },
           {
            "size": "49",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "predicate_head_select_loss/Sigmoid:0"
        }
       },
       "method_name": "tensorflow/serving/predict"
      },
      "serving_default": {
       "inputs": {
        "segment_ids": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "segment_ids:0"
        },
        "input_mask": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "input_mask:0"
        },
        "input_ids": {
         "dtype": "DT_INT32",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "input_ids:0"
        }
       },
       "outputs": {
        "predicate_head_probabilities": {
         "dtype": "DT_FLOAT",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           },
           {
            "size": "49",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "predicate_head_select_loss/Sigmoid:0"
        },
        "token_label_logits": {
         "dtype": "DT_FLOAT",
         "tensor_shape": {
          "dim": [
           {
            "size": "-1",
            "name": ""
           },
           {
            "size": "128",
            "name": ""
           },
           {
            "size": "61",
            "name": ""
           }
          ],
          "unknown_rank": false
         },
         "name": "token_label_loss/Reshape_1:0"
        }
       },
       "method_name": "tensorflow/serving/predict"
      }
     }
    }
    }
    }



### Relevant log output

```shell
test
2023-01-18 11:28:10.218951: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-01-18 11:28:10.219019: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (b0c1756a148b): /proc/driver/nvidia/version does not exist
2023-01-18 11:28:10.219329: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(21128, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/beta:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/gamma:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(21128, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/beta:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/gamma:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(21128, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/beta:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/gamma:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(21128, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/beta:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/gamma:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
_SignatureMap({'serving_default': <ConcreteFunction pruned(input_ids, input_mask, segment_ids) at 0x7FFB8402C9A0>, 'predict': <ConcreteFunction pruned(input_ids, input_mask, segment_ids) at 0x7FFB38122730>})
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(21128, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/beta:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'bert/embeddings/LayerNorm/gamma:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
{'predicate_head_probabilities': <tf.Tensor: shape=(1, 128, 128, 49), dtype=float32, numpy=
array([[[[1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        ...,

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711487e-06, 2.0477496e-06, 7.3399161e-07, ...,
          3.9122031e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]]]], dtype=float32)>, 'token_label_predictions': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=
array([[60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60]])>}
{'predicate_head_probabilities': <tf.Tensor: shape=(1, 128, 128, 49), dtype=float32, numpy=
array([[[[1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122105e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        ...,

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711504e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]],

        [[1.7711487e-06, 2.0477496e-06, 7.3399161e-07, ...,
          3.9122031e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         ...,
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07],
         [1.7711487e-06, 2.0477514e-06, 7.3399161e-07, ...,
          3.9122068e-07, 9.6800636e-07, 5.0179818e-07]]]], dtype=float32)>, 'token_label_predictions': <tf.Tensor: shape=(1, 128), dtype=int64, numpy=
array([[60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
        60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60]])>}
2023-01-18 11:28:13.781743: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
Traceback (most recent call last):
  File "/home/polonsky/Documents/volcano_poc/Bravo/test.py", line 69, in <module>
    imported.signatures["serving_default"](**inputs) # Error: Uninitialized variables
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1601, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1610, in _call_impl
    return self._call_with_structured_signature(args, kwargs,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1691, in _call_with_structured_signature
    return self._call_flat(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 133, in _call_flat
    return super(_WrapperFunction, self)._call_flat(args, captured_inputs,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1853, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 499, in call
    outputs = execute.execute(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.FailedPreconditionError: Graph execution error:

Detected at node 'bert/encoder/layer_5/output/dense/bias/read' defined at (most recent call last):
    File "/home/polonsky/Documents/volcano_poc/Bravo/test.py", line 68, in <module>
      imported = tf.saved_model.load(export_dir=new_model_path)
Node: 'bert/encoder/layer_5/output/dense/bias/read'
Attempting to use uninitialized value StatefulPartitionedCall/bert/encoder/layer_5/output/dense/bias
         [[{{node bert/encoder/layer_5/output/dense/bias/read}}]] [Op:__inference_signature_wrapper_11982]

The text was updated successfully, but these errors were encountered:

gaikwadrahul8 · 2023-01-19T09:20:24Z

Hi, @BorisPolonsky

Apologize for the delay and I'm really sorry to inform that we do not support TF 1.x anymore so you'll have to change your TF 1.x code to Tensorflow 2.x by following our Migrate from TensorFlow 1.x to TensorFlow 2 official documentation and you can refer Migrate the SavedModel workflow Thank you!

BorisPolonsky · 2023-01-21T04:58:28Z

Hi, @BorisPolonsky

Apologize for the delay and I'm really sorry to inform that we do not support TF 1.x anymore so you'll have to change your TF 1.x code to Tensorflow 2.x by following our Migrate from TensorFlow 1.x to TensorFlow 2 official documentation and you can refer Migrate the SavedModel workflow Thank you!

I know TF1.X is not longer support hence I'm migrating the existing TF1.X SavedModel two TF2 (for serving purpose only). I've looked through the official document and this post, where a .ckpt file is required (obtained by retraining in TF2 or reuse the TF1 version for re-exporting SavedModel). And in my case I've got merely a SavedModel itself. So is it still possible to convert it to TF2.X SavedModel, given that I've successfully made forward pass work in TF2 as the script above suggests?

gaikwadrahul8 · 2023-01-26T18:06:12Z

Hi, @SuryanarayanaY

Could you please look into this ? Thank you!

sachinprasadhs · 2023-02-02T23:14:10Z

Hi, Instead of using tf.saved_model.save to save the model, could you please try saving using tf.keras.Model.save and to load use tf.keras.models.load_model and let us know if you still face same behavior. Thanks!

google-ml-butler · 2023-02-10T00:07:00Z

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler · 2023-02-17T01:07:01Z

Closing as stale. Please reopen if you'd like to work on this further.

google-ml-butler · 2023-02-17T01:07:04Z

Are you satisfied with the resolution of your issue?
Yes
No

google-ml-butler bot added the type:support Support issues label Jan 18, 2023

google-ml-butler bot assigned gaikwadrahul8 Jan 18, 2023

gaikwadrahul8 added TF 2.8 comp:core issues related to core part of tensorflow labels Jan 18, 2023

gaikwadrahul8 added the stat:awaiting response Status - Awaiting response from author label Jan 19, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Jan 21, 2023

gaikwadrahul8 assigned SuryanarayanaY and unassigned gaikwadrahul8 Jan 26, 2023

SuryanarayanaY assigned sachinprasadhs and unassigned SuryanarayanaY Feb 2, 2023

sachinprasadhs added the stat:awaiting response Status - Awaiting response from author label Feb 2, 2023

google-ml-butler bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Feb 10, 2023

google-ml-butler bot closed this as completed Feb 17, 2023

testhowtest mentioned this issue May 23, 2024

Model weights cannot be saved #68467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model weight not saved when exporting SavedModel #59287

Model weight not saved when exporting SavedModel #59287

BorisPolonsky commented Jan 18, 2023 •

edited

Loading

Issue Type

Have you reproduced the bug with TF nightly?

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

gaikwadrahul8 commented Jan 19, 2023

BorisPolonsky commented Jan 21, 2023 •

edited

Loading

gaikwadrahul8 commented Jan 26, 2023

sachinprasadhs commented Feb 2, 2023 •

edited

Loading

google-ml-butler bot commented Feb 10, 2023

google-ml-butler bot commented Feb 17, 2023

google-ml-butler bot commented Feb 17, 2023

Model weight not saved when exporting SavedModel #59287

Model weight not saved when exporting SavedModel #59287

Comments

BorisPolonsky commented Jan 18, 2023 • edited Loading

Issue Type

Have you reproduced the bug with TF nightly?

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

gaikwadrahul8 commented Jan 19, 2023

BorisPolonsky commented Jan 21, 2023 • edited Loading

gaikwadrahul8 commented Jan 26, 2023

sachinprasadhs commented Feb 2, 2023 • edited Loading

google-ml-butler bot commented Feb 10, 2023

google-ml-butler bot commented Feb 17, 2023

google-ml-butler bot commented Feb 17, 2023

BorisPolonsky commented Jan 18, 2023 •

edited

Loading

BorisPolonsky commented Jan 21, 2023 •

edited

Loading

sachinprasadhs commented Feb 2, 2023 •

edited

Loading