-
Notifications
You must be signed in to change notification settings - Fork 74.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TFLite not support Dynamic input size #24607
Comments
Can you provide a link/reference to an existing (trained) model that requires dynamic shape elements? That will help us focus support for this feature. Thanks. |
To add some context, support for conversion with explicitly dynamic dimensions is on our radar. |
hello,I'm trying to deploy Superresolution on android, and I found it seems not support Dynamic input size, |
We're actively working to improve this. Expect a change in the next few days which will improve the ability to dynamically resize at runtime. You'll still need to give it a static shape during conversion, but the graph should be much more robust to dynamic size changes. |
@jdduke Any updates in supporting dynamic size at runtime? |
We've landed several fixes, but they've been rolled back due to downstream clients depending on the current (somewhat broken) behavior in our converter. We're hoping to resolve this properly in Q3, and might expose an experimental flag in the converter to generate graphs that always preserve rank/shape operators. |
@jdduke any update? |
No concrete updates yet, but we are investing this quarter in improving the situation, and will be sure to address this specific case. Thanks for your patience. Some workarounds that folks use are 1) for images, do the resize outside the graph and use a fixed shape input, 2) if you do require actual resizing, but only for a small number of fixed shapes, create a separate interpreter instance for each shape. |
@jdduke tflite support multi-size image input. Is there some sample about how to do that? |
@jdduke Is plan to support multi-size image input on tflite? |
Is there any update on this? This feature would be very useful for supporting seq2seq models in TFLite. |
@jdduke any update? |
We're still actively working on this, and it remains a priority for Q1. |
We added support for unknown dimensions in TensorFlow Lite today (5591208). Can you try converting your model again with tonight's (1/31) When you load the model it should have an additional field You can then call If it does not work, can you provide a detailed error and repro instructions? |
If I have model version tf 1.x, exported as saved_model format. Can I still use this functionality for dynamic input shapes? |
@harsh306 Yes. As long as the 1.X TensorFlow SavedModel takes in dynamic input shapes, then it will work with TFLite after reconverting the model. I recommend using |
Thanks, Yes, I will try that. Currently I am stuck on different issue of |
ok, I am using tf-nightly==2.2.0-dev20200204
Error:
|
Unfortunately, dynamic shapes are currently not supported with quantization. We are hoping to enable dynamic shapes with weight-only quantization support in the near future (which seems to be sufficient for your use case). However, full integer post training quantization support won't be added in the immediate future. |
@gargn For speech recognition applications, as opposed to image, it is natural using dynamic shapes due to varying lengths of the input signal. Would you point to an example that deals with this dynamic aspect and uses quantization and TFLite? I am aware of Command Recognition Android Demo, but the speech input is fixed to 1 second in that example. Thank you. |
@dusanmacho how about if u set fixed size for length and it large enough (let say around 1minutes). Then u can apply masking input on the begining of the code ? |
Thanks! Unfortunately, I am unable to run quantization on a fixed shape input either. See details below. TF inference model:
The following runs correctly, and the TFLite model inference results match those of the original TF model (@gargn thumbs up!!!):
The following gives an error due to dynamic shapes not being supported in TFLite quantization yet:
Setting a fixed input shape gives a different error:
|
@dusanmacho be carefull with rnn, at this time, lstm haven’t fully support yet. Pls replace rnn to dilated convution (eg temporal convolution network). |
@gargn Is stateful LSTM supported with converter.experimental_new_converter? When using
converter.convert() works. However when using
converter.convert() gives the following error:
|
Hi everyone and @gargn, I am getting the same error in a slightly different use case as I am working with images:
Tensorflow nightly version: My code for conversion from SavedModel to tflite:
If I convert slightly differently:
I get the error:
More Information:
Code to convert from h5 model to SavedModel model:
Now, this model performs as expected if I use the SavedModel for inference directly on my ubuntu cloud instance. The issue only arises when I try to convert this model from SavedModel to tflite model for deploying on iOS and Android, as I have shown on the top of my comment. As you can see, my serving function takes randomly sized input images becauses it resizes them to required size while maintaining the aspect ration. Therefore, I need to have [None, None, None, 3] for input dimensions. Can anyone share if there's a way to do this and what I might be missing? Is there really this functionality missing at this point? I thought this would be a very common use case (it does work flawlessly using SavedModel). |
Hey @gargn @karimnosseir @miaout17 @dusanmacho @Saduf2019 does it support dynamic input shape, while using frozen graphs for conversion to tflite in TF 1.x version. |
i think this issue might be related: #40268 |
I am running into a similar issue where input sizes are updating to accomodate for a dynamic batch size, however the output batch dimension appears to remain fixed at 1 despite the batch size.
I outlined all the details in another similar issue here. |
@alfarok updated the other issue. Thanks |
hi @gargn, |
It works, helps a lot!! |
@yuqiu1233 GPU and NNAPU delegates doesn't support handling dynamic batch size currently. |
does GPU and NNAPI delegates support handling dynamic input shape such as [1, None, 100] ?. |
System information
Describe the feature and the current behavior/state.
Try running tflite_convert again, with --input_shapes=1,none,none,3, but these conversion cannot be supported.
For some networks requiring dynamic input, the static input of tflite is too limited, such as Superresolution Neural Network.
The text was updated successfully, but these errors were encountered: