-
Notifications
You must be signed in to change notification settings - Fork 74.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Model converted to TFLite always returns NaN as output. #22803
Comments
I faced the same problem too! After days of investigation, I found the problem is caused by batch norm. I'm not sure if this is a problem for specific tensorflow version? |
Hi @hubert0527 and @sercant, |
Hi @hazirbas, |
@hubert0527 Thank you for pointing at batch normalization: when I had removed it from network my outputs became normal (not nan). But of course the quality of my network fell drammatically without batch normalization. I tried to replace tf.layer.batch_normalization with tf.keras.layers.BatchNormalization and tf.contrib.layers.batch_norm, but no effect. Finally I solved the problem by implementing my own batch normalization like this:
Note that this is not literal implementation of batch norm (here moving average is not used), because only train mode was required for my project. Also note that we cannot use tf.nn.moments to calc mean and dev because it is not supported by tflite (so we need to implement own function for moments). After replacing batch normalization with provided functions I was able to train my network, export it to tflite and use it during inference in tflite correctly. |
Have you tried using |
Yes, I tried tf.layers.batch_normalization. It was my original normalization function which I started my investigation from. And I also was using tensorflow v1.12 when the problem was faced. |
Load balancing... @karimnosseir could you take a look? |
@sercant When freezing a model for inference, the attribute "is_training" of the BN layers should be set as false. In your frozen model, "is_training" is true. That makes the means/variances of BN layers to be all 0s. |
Hi Sercant, As IrvingMeng mentioned, you need to make sure is_training is false. Can you please confirm and share tflite &pb files for the model Thanks |
@IrvingMeng Thank you, that really fixes the problem. In my case I was using training=True during export intentionally because that gives better performance on my task. It worked fine during pb inference, so I was surprised that tflite was not working. But now I see that batch normalization in tflite works correct when using as intended. And for my task I should use custom normalization, for example suggested above. |
@sercant Have you found some solution for this problem? I have the same problem :( |
Thank you @IrvingMeng for pointing out the issue with the batch normalization parameter.
@ricardobnjunior Have you tried the solution suggested by @IrvingMeng ? I, unfortunately, don't work on this issue anymore since I initially opened it in October and got the answer in March. If you can confirm that the suggested solution works, the issue can be closed. |
Hello, I have been playing around with the pix2pix example: If you check out the link you can see setting trainable=False for BatchNormalization does not help. |
@ablenesi I tested my solution with the tf.layers.batch_normalization from tensorflow v1 (following the steps in this page https://www.tensorflow.org/lite/guide/get_started ). However, it won't work with the model built with keras layers. |
@ablenesi Hi, did you find any workaround? |
I find a solution about this question. Just change the attribute fused of tf.keras.layers.bn to false. The output is correct. |
Marking issue as resolved due to inactivity. Feel free to re-open this if it's unresolved or file a new issue |
Hi, I'm not sure if I should be filing a different issue or it's ok to re-open this one. I am facing the same issue with nan values after converting pix2pix' generator to tflite. As @wangrui261 said, I replaced all I am using tensorflow version 2.3.0 and Python 3.7.7 Has anyone been able to solve this or has found a workaround? |
Re-opening unresolved issue. |
I discovered this generator also returns NaN values when running on CPU. As @hubert0527 said, this is due to big values of feature maps after Batch Normalization layers. I found somewhere in stack overflow that GPU zeroes these NaN values, while CPU keeps them, so I added a Lambda layer with a I have the feeling that this would fix the TFLite conversion, but when calling |
@javirk: I have the same issue as you do. Do you find any solution for transferring pix2pix GAN to tf lite model? Thank you so much! |
@javirk One way is to use TF SELECT which will fallback to TF kernel for the missing TF ops |
I still have this problem. |
Finally I solve the problem by quantizing deep model... |
System information
python3 test.py
Describe the problem
I have been trying to convert a frozen graph trained using this repo for using on android with TFLite. Trained model uses MobileNetV2 as frontend and Mobile UNet for Semantic Segmentation as the model. The problem I am facing is: the frozen pb graph segments the image correctly but TFLite converted model returns all
nan
for the output. To try the problem I wrote the following script. The model is converted without any errors or warnings, but the output is not correct. Do you have any idea what might be causing this?Note: converted model is also returning NaNs on android device.
Frozen graph: output_graph.pb
Source code / logs
test.py
output
The text was updated successfully, but these errors were encountered: