2 errors while building NodeDef 'tf_op_layer_Maximum_2/Maximum_2' using Op<name=Maximum; signature=x:T, y:T -> z:T ...Inconsistent values for attr 'T' DT_INT32 vs. DT_INT64

[neilteng]

System information
I am using colab to reproduce the issue and the ipynb is attached below.

You can collect some of this information using our environment capture
tf.version.GIT_VERSION: v1.12.1-32511-g2cc80a74f2
tf.version.VERSION: 2.3.0-dev20200525

Describe the current behavior
cannot load the saved tf model

Describe the expected behavior
successifully save the model and serve it like this example: https://github.com/tensorflow/transform/blob/master/examples/census_example_v2_test.py

Standalone code to reproduce the issue
https://colab.research.google.com/drive/1h2QIX_QZetIzSuG0J6lNWkHoSa2nnIyS?usp=sharing

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

error is show in the last cell of the colab notebook.

WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer LSTM_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs, op_def)
   1819   try:
-> 1820     c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
   1821   except errors.InvalidArgumentError as e:

InvalidArgumentError: 2 errors while building NodeDef 'tf_op_layer_Maximum_2/Maximum_2' using Op<name=Maximum; signature=x:T, y:T -> z:T; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_UINT8, DT_INT16, DT_INT32, DT_INT64]>:
Inconsistent values for attr 'T' DT_INT32 vs. DT_INT64
Inconsistent values for attr 'T' DT_INT32 vs. DT_INT64

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
15 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs, op_def)
   1821   except errors.InvalidArgumentError as e:
   1822     # Convert to ValueError for backwards compatibility.
-> 1823     raise ValueError(str(e))
   1824 
   1825   return c_op

ValueError: 2 errors while building NodeDef 'tf_op_layer_Maximum_2/Maximum_2' using Op<name=Maximum; signature=x:T, y:T -> z:T; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_UINT8, DT_INT16, DT_INT32, DT_INT64]>:
Inconsistent values for attr 'T' DT_INT32 vs. DT_INT64
Inconsistent values for attr 'T' DT_INT32 vs. DT_INT64

[amahendrakar]

Was able to reproduce the error. Please find the gist of it here. Thanks!

[karmel]

Thanks for the report @neilteng . The example is fairly complex, and it is hard for us to pinpoint the issue. Can you remove the dependencies + get the example down to the simplest possible repro? That will allow us to more easily determine whether this is an issue with TFT, Beam, feature columns, graph mode, etc.

[neilteng]

ok, I will try to reproduce the error with a simpler example.

[gowthamkpr]

@neilteng Were you able to reproduce the error with a simpler example?

[neilteng]

@neilteng Were you able to reproduce the error with a simpler example?

I think I accidentally delete the notebook.. And I cannot recover it.. If we are luck and you guys have a copy, I will work on it again. Otherwise, I can only close this issue for now.

[gowthamkpr]

@neilteng Please share us the notebook. Thanks!

[neilteng]

I think I accidentally delete the notebook.. And I cannot recover it.. If we are luck and you guys have a copy, I will work on it again. Otherwise, I can only close this issue for now.