Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Op type not registered 'FusedBatchNormV3'" exception when trying to use a TensorFlow 2.0 model #4506

Closed
jsgonsette opened this issue Nov 27, 2019 · 1 comment

Comments

@jsgonsette
Copy link

@jsgonsette jsgonsette commented Nov 27, 2019

System information

SDK .NET Core (reflétant tous les global.json) :
Version: 3.0.100
Commit: 04339c3a26

Environnement d'exécution :
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.0.100\

Host (useful for support):
Version: 3.0.0
Commit: 7d57652f33

.NET Core SDKs installed:
3.0.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Anaconda Navigator 1.9.7 with TensorFlow 2.0.0

Visual Studio 2019 Community Edition, version 16.3.10
with

  • Microsoft ML v1.4.0
  • TensorFlow.NET v0.12.0

Issue

  • What did you do?
    Taking inspiration from this ML.Net project,
    I trained a neural network with Tensorflow 2.0, saved it to disk (with python function tf.saved_model.save) and then tried to load it and use it in a custom .Net project in C#.

  • What happened?
    I expected my C# program to be able to load a picture, to push it through the Tensorflow model and to get an output vector corresponding to classification results.

Everything is fine when I load my model with function LoadTensorFlowModel. Calling GetInputSchema or GetModelSchema on the model returns coherent information.

Fitting my pipeline with Fit also works fine. However, as soon as I try to push a picture in the pipeline, I get an exception Op type not registered 'FusedBatchNormV3 (see below).

  • What did you expect?
    I didn't expect to get this strange exception during inference.
    Obviously this exception seems to happen inside the TensorFlow library called by ML.NET. I have no idea if the bug is on the TensorFlow side, or on the ML.NET side, or if missed something.

Below you can find minimal code to reproduce the problem.

Exception

2019-11-27 16:17:24.783604: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: D:\training\model
2019-11-27 16:17:24.820727: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-11-27 16:17:25.045189: I tensorflow/cc/saved_model/loader.cc:182] Restoring SavedModel bundle.
2019-11-27 16:17:25.954912: I tensorflow/cc/saved_model/loader.cc:132] Running initialization op on SavedModel bundle.
2019-11-27 16:17:26.205591: I tensorflow/cc/saved_model/loader.cc:285] SavedModel load for tags { serve }; Status: success. Took 1421948 microseconds.
2019-11-27 16:17:33.391938: W tensorflow/core/kernels/partitioned_function_ops.cc:197] Grappler optimization failed. Error: Op type not registered 'FusedBatchNormV3' in binary running on AUTISTE. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
2019-11-27 16:17:33.409723: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at partitioned_function_ops.cc:118 : Not found: Op type not registered 'FusedBatchNormV3' in binary running on AUTISTE. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
Op type not registered 'FusedBatchNormV3' in binary running on AUTISTE. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
[[{{node StatefulPartitionedCall}}]]
[[{{node StatefulPartitionedCall}}]]
Exception levée : 'System.InvalidOperationException' dans Microsoft.ML.Data.dll
Une exception non gérée du type 'System.InvalidOperationException' s'est produite dans Microsoft.ML.Data.dll

Source code - TensorFlow side

import os
import tensorflow as tf
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras.models import Model

base_model = tf.keras.applications.ResNet50 (include_top=False, weights='imagenet', input_shape=(224, 224, 3))
x = base_model.output
x = GlobalAveragePooling2D (name="top_avgpool2d")(x)
x = Dense (512, activation='relu', name="top_mlp")(x)
predictions = Dense (2, activation='softmax', name="top_gender")(x)
model = Model (inputs=base_model.input, outputs=predictions)

tf.saved_model.save (model, 'D:\\training\\model\\')

Source code - Visual Studio side

See attached file
VS Code.txt

@jsgonsette

This comment has been minimized.

Copy link
Author

@jsgonsette jsgonsette commented Nov 28, 2019

I think I have my answer. I finally understood what version of TensorFlow is supported by ML.net through the TensorFlow.Net package. Obviously, using TensorFlow 2.0 is not a good idea for now (see supported versions below).

image

@jsgonsette jsgonsette closed this Nov 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.