Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize_for_inference.py should remove Dropout operations #5867

Closed
mattrajca opened this issue Nov 25, 2016 · 26 comments
Closed

optimize_for_inference.py should remove Dropout operations #5867

mattrajca opened this issue Nov 25, 2016 · 26 comments
Assignees

Comments

@mattrajca
Copy link

@mattrajca mattrajca commented Nov 25, 2016

When I first tried using an exported MNIST model with TensorFlow on iOS, I got the following error:

Invalid argument: No OpKernel was registered to support Op 'RandomUniform' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

     [[Node: dropout/random_uniform/RandomUniform = RandomUniform[T=DT_INT32, dtype=DT_FLOAT, seed=0, seed2=0](dropout/Shape)]]

Since Dropout operations are no-ops during inference (we pass in a keep probability of 1), it would be nice if they were removed (or turned into no-ops of some kind that can be parsed and ignored by the iOS library).

While I was able to work around this by explicitly exporting a separate graph that does not contain Dropout, it was pretty tedious and it would be nice if the optimize_for_inference.py script did this automatically.

Environment info

Operating System: macOS 10.12

Installed version of CUDA and cuDNN:
None

Source:
This week's tip-of-tree (around d93d526)

@nikashitsa
Copy link

@nikashitsa nikashitsa commented Feb 8, 2017

Yes, it would be nice if the optimize_for_inference.py script did this automatically.
My current manual solution https://dato.ml/drop-dropout-from-frozen-model/

@aselle aselle removed the enhancement label Feb 9, 2017
@cancan101
Copy link
Contributor

@cancan101 cancan101 commented Feb 28, 2017

related / dupe: #6124

@cancan101
Copy link
Contributor

@cancan101 cancan101 commented Feb 28, 2017

There is also this SO (attempt at a) solution: http://stackoverflow.com/questions/40358892/wipe-out-dropout-operations-from-tensorflow-graph
and my attempt at this:

for node in temp_graph_def.node:
    for idx, i in enumerate(node.input):
        input_clean = node_name_from_input(i)
        if input_clean.endswith('/cond/Merge') and input_clean.split('/')[-3].startswith('dropout'):
            identity = node_from_map(input_node_map, i).input[0]
            assert identity.split('/')[-1] == 'Identity'
            parent = node_from_map(input_node_map, node_from_map(input_node_map, identity).input[0])
            pred_id = parent.input[1]
            assert pred_id.split('/')[-1] == 'pred_id'            
            good = parent.input[0]
            node.input[idx] = good
@swarathesh
Copy link

@swarathesh swarathesh commented Mar 8, 2017

i'am getting the following error in android
ERROR: Analysis of target '//tensorflow/contrib/util:convert_graphdef_memmapped_format' failed; build aborted.

@haydenth
Copy link

@haydenth haydenth commented May 19, 2017

Is there anyway this can be accomplished using the transform_graph tool and remove_nodes?

@kmonachopoulos
Copy link

@kmonachopoulos kmonachopoulos commented Aug 22, 2017

Is there a way to get rid of the dropout in a model? I am currently trying to remove it but I haven't found a way yet :

Operation Name : dropout_1/keep_prob
Tensor Stats : (<tf.Tensor 'dropout_1/keep_prob:0' shape=() dtype=float32>,)
Operation Name : dropout_1/Shape
Tensor Stats : (<tf.Tensor 'dropout_1/Shape:0' shape=(4,) dtype=int32>,)
Operation Name : dropout_1/random_uniform/min
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform/min:0' shape=() dtype=float32>,)
Operation Name : dropout_1/random_uniform/max
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform/max:0' shape=() dtype=float32>,)
Operation Name : dropout_1/random_uniform/RandomUniform
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform/RandomUniform:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/random_uniform/sub
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform/sub:0' shape=() dtype=float32>,)
Operation Name : dropout_1/random_uniform/mul
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform/mul:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/random_uniform
Tensor Stats : (<tf.Tensor 'dropout_1/random_uniform:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/add
Tensor Stats : (<tf.Tensor 'dropout_1/add:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/Floor
Tensor Stats : (<tf.Tensor 'dropout_1/Floor:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/Inv
Tensor Stats : (<tf.Tensor 'dropout_1/Inv:0' shape=() dtype=float32>,)
Operation Name : dropout_1/mul
Tensor Stats : (<tf.Tensor 'dropout_1/mul:0' shape=(?, ?, ?, 4096) dtype=float32>,)
Operation Name : dropout_1/mul_1
Tensor Stats : (<tf.Tensor 'dropout_1/mul_1:0' shape=(?, ?, ?, 4096) dtype=float32>,)

Any updates on this ?

@alextp
Copy link
Contributor

@alextp alextp commented Feb 7, 2018

Is using a tensor for keep_prob which is set to 1 during inference not enough?

@ngeorgis
Copy link

@ngeorgis ngeorgis commented Feb 16, 2018

Any updates on this? Just tried optimize_for_inference.py but dropout still there confusing NVIDIA TensorRT and Jetson TX2 inference.

@kmonachopoulos
Copy link

@kmonachopoulos kmonachopoulos commented Feb 17, 2018

There is a way to create a new graph and copy all the nodes you want (except dropout) from the old graph to the new one, fixing the input/output names of the nodes before and after the dropout layer. This way you can create a new graph without the dropout layer.

@AndRossi
Copy link

@AndRossi AndRossi commented Apr 6, 2018

Please let optimize_for_inference remove dropout layers from the graph.
I am trying to convert a MNIST model from TF to TFLite.
It seems to me that this should be the very first thing to try when one is learning to use TFLite, since MNIST is the first tutorial in TensorFlow.

To allow the conversion to TFLite, I need to remove the dropout layer in my MNIST model frozen graph.

If I get it right, the only way to do that (without writing a python script for that) is to use the Graph Transform Tool.
Which can only run with Bazel.
Which requires to download TF from sources.
This feels like a huge overkill.

The funny thing is that in prediction mode the dropout would not even be used.
So I'm spending a day downloading and configuring components in my environment to just remove something that would not even be used.

If I am missing something please tell me - I would be overjoyed to know I got it wrong.

@shrutim90
Copy link

@shrutim90 shrutim90 commented Apr 24, 2018

Run training script -> export_inference_graph -> freeze_graph . This removes all the extra nodes added during training.

I was trying to convert finetuned inception model to uff format required by Jetson, and got issues due to the same RandomUniform node and a few others dropout related nodes. Running export_inference_graph on finetuned model before freezing fixed the issue.

@AndRossi
Copy link

@AndRossi AndRossi commented Apr 26, 2018

Hi @shrutim90 , can you please provide more information on the export_inference_graph script?

@shrutim90
Copy link

@shrutim90 shrutim90 commented Apr 30, 2018

@AndRossi : In this link: https://github.com/tensorflow/models/tree/master/research/slim#Pretrained , they have used export_inference_graph script to export default inception graph. If you'll check the code for that script, you'll see there is an option to specify dataset_name as well.

python export_inference_graph.py
--alsologtostderr
--model_name=inception_v3
--dataset_name=flowers
--output_file=/tmp/inception_v3_inf_graph_flowers.pb

So you can specify any dataset predefined in slim library or add you own data in the slim dataset and then use above script, with required dataset_name. It will create graph with all the extra layers removed and correct number of output nodes.

@DanglingPointer
Copy link

@DanglingPointer DanglingPointer commented May 3, 2018

@shrutim90 It didn't work for me. When I use a tf.slim model with my own dataset and is_training=True I obtain a checkpoint with 97% accuracy. If I then import this checkpoint into the same tf.slim model but with is_training=False I get only around 40% accuracy.

@tofulawrence
Copy link

@tofulawrence tofulawrence commented Aug 16, 2018

You should freeze the training checkpoint into a separate eval/inference graph. You should have separate python code to constructor this eval_graph. Take a look at tf_estimator model_fn to see how it encourages constructing a different graph for training/eval/inference. That graph should be compatible with TF Lite.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py

In general, it's difficult to convert a training graph into an inference graph because batch norm and dropout create different graphs for training or eval.

@tofulawrence
Copy link

@tofulawrence tofulawrence commented Aug 24, 2018

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

@kbalka
Copy link

@kbalka kbalka commented Aug 29, 2018

@tofulawrence I bet you had a look on this #19264, can you comment why it can't be upstreamed ? Does it fail some internal test cases ?

@qiaohaijun
Copy link
Contributor

@qiaohaijun qiaohaijun commented Aug 30, 2018

I got dropout ops convert fail in tensorrt 4.0

2018-08-30 11:11:20.711520: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:10 due to: "Unimplemented: Require 4 dimensional input. Got 3 query_lookup/Dropout/cond/dropout/Shape/Switch" SKIPPING......( 3 nodes)
@petewarden
Copy link
Member

@petewarden petewarden commented Feb 22, 2019

I've just approved the PR related to this one. Sorry for the delay in reviewing.

@nanshihui
Copy link

@nanshihui nanshihui commented Feb 23, 2019

which version can solve this problem。hope comming soon

@mohapatras
Copy link

@mohapatras mohapatras commented Jul 28, 2019

Any update on the solution yet ?

@kmonachopoulos
Copy link

@kmonachopoulos kmonachopoulos commented Jul 28, 2019

I think that the only way to remove dropout for now is to comment out the dropout lines in your python source code and export the model without them ...

@danvargg
Copy link

@danvargg danvargg commented Oct 10, 2019

Any update on this issue yet?

1 similar comment
@zh794390558
Copy link
Contributor

@zh794390558 zh794390558 commented Oct 10, 2019

Any update on this issue yet?

@danvargg
Copy link

@danvargg danvargg commented Oct 10, 2019

There is also this SO (attempt at a) solution: http://stackoverflow.com/questions/40358892/wipe-out-dropout-operations-from-tensorflow-graph
and my attempt at this:

for node in temp_graph_def.node:
    for idx, i in enumerate(node.input):
        input_clean = node_name_from_input(i)
        if input_clean.endswith('/cond/Merge') and input_clean.split('/')[-3].startswith('dropout'):
            identity = node_from_map(input_node_map, i).input[0]
            assert identity.split('/')[-1] == 'Identity'
            parent = node_from_map(input_node_map, node_from_map(input_node_map, identity).input[0])
            pred_id = parent.input[1]
            assert pred_id.split('/')[-1] == 'pred_id'            
            good = parent.input[0]
            node.input[idx] = good

Hi @cancan101, what is node_name_from_input?
I'm trying to implement this manual fix for a conversion Tensorflow > CoreML.

@beniroquai
Copy link

@beniroquai beniroquai commented Apr 27, 2020

One comment to add - it's hard to debug if you don't know what this means:

Here is a list of operators for which you will need custom implementations: RandomUniform.

or

Exception: Placeholder Placeholder should be specied by input_arrays.

which either indicate the use for a tf.placeholder for the dropout probability or the dropout node in the tf.lite conversion process. Outcommenting as suggested helped!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet