tensorflowjs-converter / Mask_RCNN : Unsupported Ops in the model before optimization Unique, DenseToDenseSetOperation

[alexis-gruet-deel]

TensorFlow.js version

tensorflowjs 0.6.7
Dependency versions:
keras 2.2.2
tensorflow 1.12.0

Browser version

N/A

Describe the problem or feature request

Can't convert a freezed mask_rcnn_coco.pb to tensorflowjs using tensorflowjs-converter

Code to reproduce the bug / link to feature request

I first freezed the keras model with :

from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
from keras import backend as K

K.set_image_data_format('channels_last')
sess = K.get_session()
output_node_names = [node.op.name for node in model.keras_model.outputs]

print(output_node_names)

constant_graph = graph_util.convert_variables_to_constants(
            sess,
            sess.graph.as_graph_def(),
            output_node_names)

graph_io.write_graph(
    constant_graph, 
    '/tmp', 
    'mask_rcnn_coco.pb', 
    as_text=False)

The print output_node_names gives :

['mrcnn_detection/Reshape_1', 
'mrcnn_class/Reshape_1', 
'mrcnn_bbox/Reshape', 
'mrcnn_mask/Reshape_1', 
'ROI/packed_2', 
'rpn_class/concat', 
'rpn_bbox/concat']

I then run the tensorflowjs-converter with :

tensorflowjs_converter --input_format=tf_frozen_model --saved_model_tags=serve --output_node_names='mrcnn_detection/Reshape_1,mrcnn_class/Reshape_1,mrcnn_bbox/Reshape,mrcnn_mask/Reshape_1,ROI/packed_2,rpn_class/concat,rpn_bbox/concat' '/tmp/frozen_mask_rcnn_coco.pb'   '/tmp/output/fronzen_mask_rcnn_coco_web.pb'

tensorflowjs-converter failed with :

Using TensorFlow backend.
Traceback (most recent call last):
  File "/usr/local/bin/tensorflowjs_converter", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/converters/converter.py", line 333, in main
    strip_debug_ops=FLAGS.strip_debug_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/converters/tf_saved_model_conversion.py", line 330, in convert_tf_frozen_model
    skip_op_check=skip_op_check, strip_debug_ops=strip_debug_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/converters/tf_saved_model_conversion.py", line 117, in optimize_graph
    ', '.join(unsupported))
ValueError: Unsupported Ops in the model before optimization
Unique, DenseToDenseSetOperation

Any idea ?

[alexis-gruet-deel]

Hello @pyu10055, any update on this ?

[pyu10055]

@MI-LA01 We are missing these two ops.
Unique is variable output length op, which would be an async op in tfjs. We currently don't have support for set operations, I will look into prioritize these ops.

[Johannes0Horn]

@MI-LA01 May i ask which Implementation of MaskRCNN you tried to convert?

[alexis-gruet-deel]

Hello @Johannes0Horn,
The one from Matterport using originally keras

[Johannes0Horn]

@MI-LA01 Dont you have to rewrite all Customlayers and Lamdalayers in Javascript to get it Working on tfjs, since they cant be converted? I was also trying to convert the exact Same Model to tfjs. I could convert it successfully but wasnt able to load it in tfjs because tfjs didnt know how to handle the custom and Lambda layers.

Currently, i try to rewrite all Customlayers in Javascript but i found Out several Operations which are used in these layers, arent Supported yet by tfjs.

[Johannes0Horn]

Afaik, the following Operations are missing in tfjs, to use the MaskRCNN in tfjs:
tf.boolean_mask
tf.constant
tf.sets.set_intersection
tf.map_fn
tf.cond
tf.stopGradient
tf.Assert
tf.random_shuffle

and the dtype int64

[chenqing]

@pyu10055 any update?

[Johannes0Horn]

I created a google colab to test the the conversion from MaskRCNN saved as .h5 file to tfjs using tensorflowjs_converter 1.0.1.
Since the tensorflow_converter executes without any error/warning i assume no more operations are missing to convert MaskRCNN to a tfjs Model.

[duhaime]

@Johannes0Horn were you able to load that model in tfjs? I can use tensorflow_converter to save a keras model to disk, but calling tf.loadModel (now called tf.loadLayersModel) throws an error because of non-supported Lambda layers inside the keras model. If you got non-supported layers running in tfjs, I'd love to hear about it!

[caisq]

Lambda layers are not supported in tf.js because they are wedded to the Python language and is hence not portable. To use them in TensorFlow.js, you can re-implemented them in JavaScript as custom layer types. See here for an example we have for how to write and register a custom layer in TF.js:
https://github.com/tensorflow/tfjs-examples/tree/master/custom-layer