Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse toco generated file (.tflite) in python? #16561

Closed
santoshchilkunda opened this issue Jan 30, 2018 · 12 comments
Closed

Parse toco generated file (.tflite) in python? #16561

santoshchilkunda opened this issue Jan 30, 2018 · 12 comments

Comments

@santoshchilkunda
Copy link

@santoshchilkunda santoshchilkunda commented Jan 30, 2018

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.3 LTS
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): 1.4.1
  • Python version: 3.5.4
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:
  • Exact command to reproduce:

Describe the problem

I am using toco to optimize a frozen model (.pb). How do I read the .tflite file in python - something similar to tf.gfile.GFile('frozen.pb', 'rb')?

@santoshchilkunda santoshchilkunda changed the title How to parse toco generated file (.tflite) in python? Parse toco generated file (.tflite) in python? Jan 30, 2018
@reedwm

This comment has been minimized.

Copy link
Member

@reedwm reedwm commented Jan 30, 2018

/CC @bjacob

@santoshchilkunda

This comment has been minimized.

Copy link
Author

@santoshchilkunda santoshchilkunda commented Feb 3, 2018

bump

@tensorflowbutler

This comment has been minimized.

Copy link
Member

@tensorflowbutler tensorflowbutler commented Feb 18, 2018

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

@aselle

This comment has been minimized.

Copy link
Member

@aselle aselle commented Feb 20, 2018

@santoshchilkunda, what exactly are you trying to do once you read it? Do you want to run inference on it. Do you want to analyze it?

@santoshchilkunda

This comment has been minimized.

Copy link
Author

@santoshchilkunda santoshchilkunda commented Feb 20, 2018

@aselle
Thanks for looking into this.

I am trying to read/parse the model, convert it to our internal format, and then run inference on it.

As a workaround (to not being able to parse tflite), I set the output_format while running toco tool to TENSORFLOW_GRAPHDEF (i.e. both input and output formats are TENSORFLOW_GRAPHDEF). And then parse the generated protobuf.

However, I see that protobuf is "less" optimized compared to tflite

Following is the log when output_format is TENSORFLOW_GRAPHDEF:
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 773 operators, 1072 arrays (0 quantized)
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 141 operators, 260 arrays (0 quantized)
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 141 operators, 260 arrays (0 quantized)
I tensorflow/contrib/lite/toco/toco_tooling.cc:273] Estimated count of arithmetic ops: 3.01265 billion (note that a multiply-add is counted as 2 ops).

Following is the log when output_format is set to TFLITE:
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 773 operators, 1072 arrays (0 quantized)
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 84 operators, 203 arrays (0 quantized)
I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 84 operators, 203 arrays (0 quantized)
I tensorflow/contrib/lite/toco/allocate_transient_arrays.cc:313] Total transient array allocated size: 4014080 bytes, theoretical optimal value: 4014080 bytes.
I tensorflow/contrib/lite/toco/toco_tooling.cc:273] Estimated count of arithmetic ops: 3.01265 billion (note that a multiply-add is counted as 2 ops).

The number of operators after graph transformation is different (141 v/s 84) while the estimated count of arithmetic ops is same (3.01265 billion).
From a performance perspective, I am not completely sure if this means both the output formats are same or not.

@aselle

This comment has been minimized.

Copy link
Member

@aselle aselle commented Feb 20, 2018

well you can use flatc to generate a python api that can read the tflite format.
https://google.github.io/flatbuffers/flatbuffers_guide_use_python.html
Does that answer your question?

@santoshchilkunda

This comment has been minimized.

Copy link
Author

@santoshchilkunda santoshchilkunda commented Feb 20, 2018

I will try it and update. Thanks!

Regarding the second part of my question, is there a performance difference between the two output formats?

@bjacob

This comment has been minimized.

Copy link
Contributor

@bjacob bjacob commented Feb 21, 2018

@santoshchilkunda, it is an inherent characteristic of the TensorFlow Lite flatbuffer format that it allows to represent the same neural network in fewer nodes than are needed in the TensorFlow GraphDef format, as you found from this logging. To find out more about what the difference is in your graph, use --dump_graphviz as explained there,
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md#using---dump_graphviz

@santoshchilkunda

This comment has been minimized.

Copy link
Author

@santoshchilkunda santoshchilkunda commented Mar 8, 2018

dump_graphviz worked, thanks!
Still working on flatc method that was suggested

@andrehentz andrehentz added the comp:lite label Mar 8, 2018
@aselle

This comment has been minimized.

Copy link
Member

@aselle aselle commented Mar 9, 2018

Another approach is to generate json from the flatbuffer using flatc. This is used by the tflite visualizer:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/tools/visualize.py
Please close the issue if it is resolve. Thanks!

@santoshchilkunda

This comment has been minimized.

Copy link
Author

@santoshchilkunda santoshchilkunda commented Mar 12, 2018

Thanks for all the suggestions.
dump_graphviz served my purpose.
Will explore flat buffer method soon.

@cumberb1tch

This comment has been minimized.

Copy link

@cumberb1tch cumberb1tch commented Mar 25, 2019

well you can use flatc to generate a python api that can read the tflite format.
https://google.github.io/flatbuffers/flatbuffers_guide_use_python.html
Does that answer your question?

@aselle could you please make it a little bit clear for me

I am using pyhon generated code to read and get node attributes from the model graph.

I generated it this way (output is tflite/ folder with autogenerated *.py files):
flatc -python tensorflow/tensorflow/lite/schema/schema.fbs

Than I read the model:

    from tflite.Model import Model
    def read_tflite_model(file):
        buf = open(file, "rb").read()
        buf = bytearray(buf)
        model = Model.GetRootAsModel(buf, 0)
        return model

Getting model parameters:

    def print_model_info(model):
        version = model.Version()
        print("Model version:", version)
        description = model.Description().decode('utf-8')
        print("Description:", description)
        subgraph_len = model.SubgraphsLength()
        print("Subgraph length:", subgraph_len)

Than I realized that graph nodes could be interpreted as Tensor object or Operator object.

Tensor object stores quantization params, shape, tensor type (I don't understand the meaning of it yet)

Operator object has BuiltinOptions and CustomOptions methods that seems to me should give me access to node parameters (such as paddings, dilations and all layer specific info)

I tried to iterate over them since there are Inputs and Outputs methods. But failed. I don't understand w

This is my scratch:

def print_nodes_info(model):
    # what does this 0 mean? should it always be zero?
    subgraph = model.Subgraphs(0)
    operators_len = subgraph.OperatorsLength()
    print('Operators length:', operators_len)

    from collections import deque
    nodes = deque(subgraph.InputsAsNumpy())

    STEP_N = 0
    MAX_STEPS = operators_len
    print("Nodes info:")
    while len(nodes) != 0 and STEP_N <= MAX_STEPS:
        print("MAX_STEPS={} STEP_N={}".format(MAX_STEPS, STEP_N))
        print("-" * 60)

        node_id = nodes.pop()
        print("Node id:", node_id)

        tensor = subgraph.Tensors(node_id)
        print("Node name:", tensor.Name().decode('utf-8'))
        print("Node shape:", tensor.ShapeAsNumpy())

        # which type is it? what does it mean?
        type_of_tensor = tensor.Type()
        print("Tensor type:", type_of_tensor)

        quantization = tensor.Quantization()
        min = quantization.MinAsNumpy()
        max = quantization.MaxAsNumpy()
        scale = quantization.ScaleAsNumpy()
        zero_point = quantization.ZeroPointAsNumpy()
        print("Quantization: ({}, {}), s={}, z={}".format(min, max, scale, zero_point))

        # I do not understand it again. what is j, that I set to 0 here?
        operator = subgraph.Operators(0)
        for i in operator.OutputsAsNumpy():
            nodes.appendleft(i)

        STEP_N += 1

    print("-"*60)

Please help me to get access the node attributes.
Thank you in advance for your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.