Serving problem: "Op type not registered 'PyFunc'"

[vaklyuenkov]

Hello!

I've successfully trained model and want to use TensorFlow Serving components to export a trained TensorFlow model and use the standard tensorflow_model_server to serve it. So, using this code I can export model to serving:

 net = resnetv1(batch_size=1, num_layers=101)
 net.create_architecture( "TEST", 2,
                          tag='default', anchor_scales=[8, 16, 32])

variables_to_restore = slim.get_variables_to_restore()
 init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, variables_to_restore)

 saver = tf.train.Saver(sharded=True)
 model_exporter = exporter.Exporter(saver)

 with tf.Session() as sess:
       # for i in range(run_size):    
            init_fn(sess)
            print('Exporting trained model to', export_path)
            model_exporter.init(
                sess.graph.as_graph_def(),
                named_graph_signatures={
                'inputs': exporter.generic_signature({'image': net._image, 'size': net._im_info }),
                'outputs': exporter.generic_signature({ 'scores' : net._predictions['cls_prob'],'bbox_pred':net._predictions['bbox_pred'] ,'rois':net._predictions['rois']})})

            model_exporter.export(export_path, tf.constant(export_version), sess)
            print('model ', export_version, ' exported')

But when I try to serve model, serving show me an error: "Op type not registered 'PyFunc'".

It is bacause:

anchors, anchor_length = tf.py_func(generate_anchors_pre,
                                          [height, width,
                                           self._feat_stride, self._anchor_scales, self._anchor_ratios],
                                          [tf.float32, tf.int32], name="generate_anchors"

 rois, rpn_scores = tf.py_func(proposal_layer,
                                    [rpn_cls_prob, rpn_bbox_pred, self._im_info, self._mode,
                                     self._feat_stride, self._anchors, self._num_anchors],
                                    [tf.float32, tf.float32])

How can rewrite this functions using tensorflow and python to use serving?

Gratefull for any thougths!

[endernewton]

i haven't tried serving for this code.. i have pyfunc wrapped in the code so probably this is causing the problem. closing as of now

[markusnagel]

Hi @vaklyuenkov,

I have this model running in TF serving. As you already pointed out the issue is the use of tf.py_func in some of the layers. TF serving is written in C++ and therefore does not support custom python layers. As far as I'm aware there is one way how to solve it, and that's by replacing all python layers with equivalent tensorflow operations or layers.

In my own fork I made a basic implementations of all layers required for inference. It also contains an example script (in tools/export_tf_serving.py) which should work out of the box if you have the demo running. I have all layer implementations (without the example) in a separate branch and plan to soon make a PR so it can be merged into the main repository. I hope that helps you.

[endernewton]

Wow this is amazing! Please test it out (in terms of speed, accuracy) before the PR. I am more than happy to help out!

…

-Xinlei

On Tue, Jun 6, 2017 at 9:06 AM, markusnagel ***@***.***> wrote: Hi @vaklyuenkov <https://github.com/vaklyuenkov>, I have this model running in TF serving. As you already pointed out the issue is the use of tf.py_func in some of the layers. TF serving is written in C++ and therefore does not support custom python layers. As far as I'm aware there is one way how to solve it, and that's by replacing all python layers with equivalent tensorflow operations or layers. In my own fork <https://github.com/markusnagel/tf-faster-rcnn/tree/export_tensorflow_serving> I made a basic implementations of all layers required for inference. It also contains an example script (in tools/export_tf_serving.py) which should work out of the box if you have the demo running. I have all layer implementations (without the example) in a separate branch <https://github.com/markusnagel/tf-faster-rcnn/tree/tensorflow_proposal_layer> and plan to soon make a PR so it can be merged into the main repository. I hope that helps you. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#113 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACTwEQhU0yVMemu1vZkxu5uYnZJRYwtiks5sBXjrgaJpZM4Nrd7X> .

[vaklyuenkov]

@markusnagel thank you very much. Again sorry for spam.
It's awesome!

[vaklyuenkov]

@markusnagel, would you implement this layers also for training?

[rishabhmalhotra]

Is there any parallel implementation/workaround this since I am interested in knowing whether I can deploy this using TF_Serving (and the tensorflow official docs say that it's not possible to serialize with py_func) ?