Cannot infer shape for FusedBatchNornV3

[ragavendrams]

Hi,

I am trying to convert a frozen graph to ONNX using tf2onnx programmatically and i run into some warnings. (same as the ones mentioned in 745). I am using the 1.5.6 version of tf2onnx which has the changes merged already.

log:

Cannot infer shape for conv1_1/conv1_1/bn/FusedBatchNormV3: conv1_1/conv1_1/bn/FusedBatchNormV3:5
Cannot infer shape for conv1_2/conv1_2/bn/FusedBatchNormV3: conv1_2/conv1_2/bn/FusedBatchNormV3:5
Cannot infer shape for conv2_1/1/conv2_1/1/bn/FusedBatchNormV3: conv2_1/1/conv2_1/1/bn/FusedBatchNormV3:5
Cannot infer shape for conv2_3/bn/FusedBatchNormV3: conv2_3/bn/FusedBatchNormV3:5
Cannot infer shape for conv2_3/1/conv2_3/1/bn/FusedBatchNormV3: conv2_3/1/conv2_3/1/bn/FusedBatchNormV3:5
Cannot infer shape for conv3_1/bn/FusedBatchNormV3: conv3_1/bn/FusedBatchNormV3:5
Cannot infer shape for conv3_1/1/conv3_1/1/bn/FusedBatchNormV3: conv3_1/1/conv3_1/1/bn/FusedBatchNormV3:5
Cannot infer shape for conv3_3/bn/FusedBatchNormV3: conv3_3/bn/FusedBatchNormV3:5
Cannot infer shape for conv3_3/1/conv3_3/1/bn/FusedBatchNormV3: conv3_3/1/conv3_3/1/bn/FusedBatchNormV3:5
Cannot infer shape for conv4_1/bn/FusedBatchNormV3: conv4_1/bn/FusedBatchNormV3:5
Cannot infer shape for conv4_1/1/conv4_1/1/bn/FusedBatchNormV3: conv4_1/1/conv4_1/1/bn/FusedBatchNormV3:5
Cannot infer shape for conv4_3/bn/FusedBatchNormV3: conv4_3/bn/FusedBatchNormV3:5
Cannot infer shape for conv4_3/1/conv4_3/1/bn/FusedBatchNormV3: conv4_3/1/conv4_3/1/bn/FusedBatchNormV3:5
Cannot infer shape for fc1/fc1/bn/FusedBatchNormV3: fc1/fc1/bn/FusedBatchNormV3:5

System config:
tf2onnx : 1.5.6
onnx: 1.7.0
TF 1.15
hardware : Jetson Tx2 - Linux aarch64

pb file link:
https://drive.google.com/file/d/12IMU7wLmsnDwynIuyUtbmhoRTeTyyFqS/view?usp=sharing

Minimal Code:

import os
import sys
import tensorflow as tf
import tf2onnx
import onnx

path = "models/mars.pb"
output_name = ['features']
dims = [3, 128, 64]

# Function to load frozen graph
def load_frozen_pb(path_to_pb):
    with tf.io.gfile.GFile(path_to_pb, "rb") as f:
        graph_def = tf.compat.v1.GraphDef()
        graph_def.ParseFromString(f.read())
    with tf.Graph().as_default() as g:
        tf.import_graph_def(graph_def, name='')
        return g

g = load_frozen_pb(path)
sess = tf.compat.v1.Session(graph=g)
onnx_graph = tf2onnx.tfonnx.process_tf_graph(sess.graph, input_names=["images:0"], output_names=["features:0"], opset=11)
model_proto = onnx_graph.make_model("test")
with open("model.onnx", "wb") as f:
    f.write(model_proto.SerializeToString())

model1 = onnx.load('model.onnx')
onnx.checker.check_model(model1)

[jignparm]

Thanks for the quick way to reproduce. Seems like your code runs fine without warning/errors in a dry-run, using master branch tf2onnx, Anaconda Python 3.7, Onnx= 1.7, and two different TF versions: 1.14 and 2.2.0.

See the converted graph below. There is possibly a Conv2D conversion bug, which shows up randomly in some logs but not others. Since this model has lots of Conv2D operators, it's possible that you see the warnings for that reason.

The conversion should succeed however (despite the warnings). Do you end up with a model at the end?

[guschmue]

For the warnings - I think they are fixed in master
(pip install git+https://github.com/onnx/tensorflow-onnx)

2 changes you might want in your app:
when you pass the graph to tf2onnx, run some tensorflow optimizations on it (unless your frozen graph already has this). We have some function for this:

graph_def = tf2onnx.tf_optimize(input_names, output_names, graph_def)

after you converted the model, run our optimizer on it to get better perf:

onnx_graph = tf2onnx.tfonnx.process_tf_graph(...)
onnx_graph = tf2onnx optimizer.optimize_graph(onnx_graph )
model_proto = onnx_graph.make_model("test")

[ragavendrams]

Thanks a lot for the quick replies.
@jignparm : The model is being generated but i wanted to see if these warnings solved a TensorRT error of mine(unfortunately it didnt). But I was able to get rid of the warnings with the master branch. Thank you.

@guschmue : Thanks for the hints. However i was unable to optimize the graph for some reason. Each of the optimizations fail due to something related to deep copy.

[ragavendrams]

I have a follow-up question. I am trying to understand why tf2onnx adds cast operators even though it is not in the original pb file. The cast operator is not added before the previous BatchNorm layers but only for the last one.(probably cause last layer's input dim is 16384x128 and not NCHW) This causes the Tensorrt .engine generation to fail. Is there a way to prevent the cast ?

(Left: Onnx ; Right: .TF)
Also not sure why there are 2 cast operators back to back doing the same thing.

Thanks a lot for your help.

(I can create a new issue if required. Do let me know.)

[guschmue]

In your case the upper left cast is because reshape wants the shape as int64
https://github.com/onnx/onnx/blob/master/docs/Operators.md#Reshape
where tensorflow would use an int32.
If you run the optimizer - it would remove a lot of the transpose and casts.

[jignparm]

The original error seems to be fixed, and I'm assuming running the optimizer removed the extra cast nodes. Closing for now. Please reopen if needed.