Export of trained model from checkpoint doesn't write savedModel #9495
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- I am reporting the issue to the correct repository. (Model Garden official or research directory)
- I checked to make sure that this issue has not already been filed.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/research/object_detection/exporter_main_v2.py
2. Describe the bug
I was trying to train a custom model as described in the tutorial below. Training worked fine, but generating the final model failed. Myfolder structure looking like this:
The export just prints a lots of matrixes to the terminal:
2020-11-22 19:36:35.519637: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-22 19:36:35.530263: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc3f0c9a520 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-22 19:36:35.530279: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432090 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432273 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432394 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432474 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432657 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1122 19:36:39.432769 4590804480 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
Traceback (most recent call last):
File "exporter_main_v2.py", line 159, in
app.run(main)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "exporter_main_v2.py", line 152, in main
exporter_lib_v2.export_inference_graph(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/object_detection/exporter_lib_v2.py", line 265, in export_inference_graph
status.assert_existing_objects_matched()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 788, in assert_existing_objects_matched
raise AssertionError(
AssertionError: Some Python objects were not bound to checkpointed values, likely due to changes in the Python program: [<tf.Variable 'bn_Conv1/moving_mean:0' shape=(32,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
dtype=float32)>, <tf.Variable 'block_16_expand_BN/gamma:0' shape=(960,) dtype=float32, numpy=
array([1., 1., 1., 1., 1., 1., 1., 1., 1.,...
...
[ 0.00355499, 0.02184128, 0.00864329, ..., -0.02945782,
-0.00161076, -0.01344879]]]], dtype=float32)>, <tf.Variable 'block_14_project_BN/moving_variance:0' shape=(160,) dtype=float32, numpy=
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1.], dtype=float32)>]
3. Steps to reproduce
Follwed this tutorial: https://gilberttanner.com/blog/tensorflow-object-detection-with-tensorflow-2-creating-a-custom-model. Used mobilenet instead of efficientdet.
And tried to export the model with:
python3 exporter_main_v2.py --input_type image_tensor --trained_checkpoint_dir training --output_directory inference_graph --pipeline_config_path training/ssd_mobilenet_v2_quantized_300x300_coco.config
4. Expected behavior
At least a frozen inference graph file in the folder inference_graph
5. Additional context
6. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacBook Pro 16, OS BigSur
- Mobile device name if the issue happens on a mobile device:-
- TensorFlow installed from (source or binary): binary via pip
- TensorFlow version (use command below): 2.3.1
- Python version: 3.8.6
- Bazel version (if compiling from source):-
- GCC/Compiler version (if compiling from source):-
- CUDA/cuDNN version:-
- GPU model and memory:-