Exporting contrib.learn models for tensorflow_serving

[fventer]

Is it possible that a tensorflow serving tutorial based on one of the contrib.learn estimators can be added? It would be really nice if the tutorial shows how to export the deep-and-wide estimator from this tutorial: https://www.tensorflow.org/versions/r0.11/tutorials/wide_and_deep/index.html

If this is not possible in the short term, I would appreciate some help:

I have trouble deriving a deep-and-wide estimator for tensorflow_serving based on the provided tensorflow serving examples.

My estimator takes several input features and predicts 1 or 0. I have a mixture of numeric and categorical (sparse tensor) input features. However the mnist and inception tutorials only show how to export a model for tensor flow serving that takes a single input 'x' (an image) and produces an output 'y'. How do I create a named_graph_signatures structure from a dictionary of possibly multiple types of input values?

[fventer]

Is there no help for this one? What I am struggling with is to transform feature columns as given to the tf.contrib.learn.DNNLinearCombinedClassifier init function when creating the estimator and provided features as returned by the training input_fn into the parameters required by the estimator.export function and to do the analogous task inside the serving client.

I tried to extract this data by running the ops returned by layers.input_from_feature_columns just like composable_model.build_model creates the inputs for the network. But I run into dependencies that I have not figured out yet.

Concretely, in my efforts to figure this out, I temporarily stored the list of tensors returned by layers.input_from_feature_columns inside composable_model.build_model in a global var (feature_column_ops.stored_tensors) to run them later. this This list looks like this:

print(feature_column_ops.stored_tensors)

[<tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/CommandLineRule_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/FileName_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/FileNameHasTmp_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/FilePathLikeTemp_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/IsWhiteListed_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/ParentFileName_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/R1_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/R2_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/R3_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>, <tf.Tensor 'dnn/input_from_feature_columns/input_from_feature_columns/R4_embedding/embedding_lookup/Reshape_1:0' shape=(?, 8) dtype=float32>]

But when I try to run these ops:

with tf.Session() as sess: result = sess.run(feature_column_ops.stored_tensors) print(result)

I get the following error:

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value dnn/input_from_feature_columns/CommandLineRule_embedding/weights/part_0
[[Node: dnn/input_from_feature_columns/CommandLineRule_embedding/weights/part_0/read = IdentityT=DT_FLOAT, _class=["loc:@dnn/input_from_feature_columns/CommandLineRule_embedding/weights/part_0"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

[abhitopia]

@fventer - +1, I would like to know the answer as well.

[hailiang-wang]

Well, I also want to use TensorFlow Serving, by reading more codes in tf.contrib.learn.monitors.ValidationMonitor. The ValidationMonitor actually calls _estimator.export which does export jobs.

  def every_n_step_end(self, step, outputs):
    super(ExportMonitor, self).every_n_step_end(step, outputs)
    try:
      self._last_export_dir = self._estimator.export(
          self.export_dir,
          exports_to_keep=self.exports_to_keep,
          signature_fn=self.signature_fn,
          input_fn=self._input_fn,
          default_batch_size=self._default_batch_size,
          input_feature_key=self._input_feature_key,
          use_deprecated_input_fn=self._use_deprecated_input_fn)

[gsvijayraajaa]

+1

[dale-cooper]

+1 as well. Need more documentation on the interplay between canned estimators and tensorflow serving. Thanks!

[fventer]

Hi all I could get a canned estimator to export, using the new estimator export_savedmodel method, but I am still figuring out what exactly the keys of the signatures should be in the client. tf serving running the exported model gives me an error: (code=StatusCode.NOT_FOUND, details="FeedInputs: unable to find feed output inputs") in the rpc response. I am focusing on other work now, but as soon as I get time, I will come back to this problem and let you know when I got it figured out. However, https://github.com/nubbel has done this successfully using a ruby client. If someone can port this his ruby code to python, please post your solution! If we can get an additional tutorial in the tf serving documentation based on a canned estimator like the deep+wide one, life would be so much easier! Regards Fritz Venter fritz.venter@gmail.com cell: +15122937896

…

On Fri, Mar 17, 2017 at 7:57 AM, Christian Schneider < ***@***.***> wrote: +1 as well. Need more documentation on the interplay between canned estimators and tensorflow serving. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#228 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwRcvyDpLgL-3dS_8F8lTxmbr4TcTEcks5rmoNEgaJpZM4KlpAt> .

[MtDersvan]

Any feedback on this from @tensorflow_serving team? This is important, no one fully can explain how to use estimators .export() or .export_savedmodel() what functionality is deprecated and what is not, what is the proper way to use input_fn()/serving_input_fn()/InputFnOps or signature_fn.
Please, provide some insight, usecases for using Estimators (custom or canned) with tf serving.
Thanks.

[sukritiramesh]

.export() uses SessionBundle (deprecated export format); .export_savedmodel() is the recommended way to save a SavedModel.

For SavedModel related documentation, please see: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md

Also, for documentation related to setting up SignatureDefs for TensorFlow Serving, please see: https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/signature_defs.md

@davidsoergel, can you add some details too? Thanks!

[MtDersvan]

@sukritiramesh Thanks a lot. I actually figured out how to serve tf.learn models (LinearRegressor, etc.). If you guys are busy, I can easily contribute a tutorial on how to serve these types of models, e.g. maybe as an extension to tensorflow tf.learn tutorials (Linear or Wide&Deep).

[noahgamboa]

Yes please!! Been looking for an answer to this for a while

[BrianPhiri]

I have been trying to figure this out for a week now, @MtDersvan do you mind sharing how to accomplished this?

[MtDersvan]

Yes, I'll write a little tutorial this week.

[lirchi]

@MtDersvan, I'm waiting for your tutorial as well :)

[MtDersvan]

Ok, guys.
Here is a simple tutorial on how to export and serve a wide & deep tf.learn model.
Exporting and Serving a TensorFlow Wide & Deep Model
It works for me, so try it out and tell me if it also works for you.

@sukritiramesh if this is good and proper enough I can send a PR (at least the serving part), if not I will be glad to hear any feedback on how to improve it.

Known issues:
I found out that if I build from a docker image, it gives me AbortionError(code=StatusCode.NOT_FOUND, details="FeedInputs: unable to find feed output images"). But when I just build from scratch it works like a charm. I'll file an issue separately.

Also, if you are using the latest release, you will probably encounter this tensorflow/tensorflow#9436 and this #421 issues. They are fairly simple to resolve.

Big cheers to @fventer, @dale-cooper and @nubbel for their insights.

[sukritiramesh]

Thanks for putting together this tutorial, @MtDersvan! Looping in a few folks who can help review it: @wolffg @dr4b @davidsoergel.

[lirchi]

Hi @MtDersvan!
I tried following your tutorial and there seems to be an issue with the line:
servable_model_path = m.export_savedmodel(servable_model_dir, export_input_fn)
because export_input_fn is undefined. I tried replacing it with serving_input_fn which actually generates a saved_model.pb file. However, when I try to load it (via C++ using ReadBinaryProto), I get an error Data loss: Can't parse saved_model.pb as binary proto.
Any chance that you know how to resolve that?
Thanks, -Liron

[MtDersvan]

@lirchi Yes, it was a typo, you need to use serving_input_fn instead of export_input_fn . Fixed that.
As for ReadBinaryProto, I never used it, so I don't know the cause of your error. It might be an exporting issue (using new SavedModel instead of old GraphDef).
It definitely works with the latest model_servers:tensorflow_model_server.
I think someone from the team or stackoverflow can give a better answer.

[fventer]

Thanks MtDersvan This is long overdue!

On Thu, May 11, 2017 at 9:39 PM MtDersvan ***@***.***> wrote: @lirchi <https://github.com/lirchi> Yes, it was a typo, you need to use serving_input_fn instead of export_input_fn. Fixed that. As for ReadBinaryProto, I never used it, so I don't know the cause of your error. It might be an exporting issue (using new SavedModel instead of old GraphDef). It definitely works with the latest model_servers:tensorflow_model_server. I think someone from the team or stackoverflow can give a better answer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#228 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwRctm9KXNGf37rE6teuL03Uy16e3joks5r48ZvgaJpZM4KlpAt> .

-- Regards Fritz Venter fritz.venter@gmail.com cell: +15122937896

[lirchi]

@MtDersvan -- BTW, it is possible to load the model from C++ using LoadSavedModel. (see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md#c for more details.)

[semicolo]

It looks like your tutorial doesn't work anymore with tf 1.3 @MtDersvan
I'm getting:
TypeError: feature_columns should only contain instances of _FeatureColumn. Given column is _IndicatorColumn(categorical_column=_VocabularyListCategoricalColumn(key='workclass', vocabulary_list=('Self-emp-not-inc', 'Private', 'State-gov', 'Federal-gov', 'Local-gov', '?', 'Self-emp-inc', 'Without-pay', 'Never-worked'), dtype=tf.string, default_value=-1, num_oov_buckets=0))
It looks like the wide_n_deep_tutorial has been altered, there's no wide_columns variable anymore.
Trying to figure it out...

[MtDersvan]

@semicolo I see, I'll take time during next several days to update it to 1.3.
Probably a contrib change. I also think the official docs now have way more info on how to do this the proper way since 1.0.

[semicolo]

@MtDersvan I think there's a mixup between tensorflow/python/feature_column/feature_column.py and tensorflow/contrib/layers/python/layers/feature_column.py.
calling create_feature_spec_for_parsing(feature_columns) generates a TypeError because the columns don't have the correct type (the columns defined at the beginning of the script use the classes defined in the first file but create_feature_spec_for_parsing expects the classes defined in the second file)
I converted categorical_column_with_vocabulary_list into sparse_column_with_keys, categorical_column_with_hash_bucket into sparse_column_with_hash_bucket, numeric_column into real_valued_column...
Training a DNNClassifier still works after that and the feature_spec is created correctly.
Lastly, I changed serving_input_fn for serving_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)

[MtDersvan]

@semicolo I pushed the update.
I think tf.feature_column.*, tf.estimator.* are more stable APIs, and because it's part of core TensorFlow now, better maintained (at least until TF r2.0) then tf.contrib.*, so I advise you to use them.
I added a new jupyter notebook (wide_and_deep_export_r1.3.ipynb) to support the latest (r1.3), as of today, wide_n_deep_tutorial.py tutorial with tf.feature_column.* and tf.estimator.* APIs.
Haven't checked fully the wide_and_deep_client.py, so if there are any problems - please tell me.

[MtDersvan]

Also, I think an official extended tutorial is slowly getting it's way into the docs - Exporting a Trained Model for Serving.

[jakiechris]

on android phone Xiaomi 5s , android 7.1 , ReadBinaryProto crashes here too:
if (!proto->ParseFromCodedStream(&coded_stream)) {
TF_RETURN_IF_ERROR(stream->status());
return errors::DataLoss("Can't parse ", fname, " as binary proto");
}

it's wired that other android phone don't crashes here.

@fventer : if u solved this issue , pls tell me how , thks~

[jakiechris]

@fventer there was a misunderstanding here, i took a broken pb file to ReadBinaryProto so it crashed.

sorry for disturbing~

[gautamvasudevan]

Closing due to staleness. If this is still an issue, please file a new updated issue with current steps to reproduce the bug. If this is a question, please ask it on:

https://stackoverflow.com/questions/tagged/tensorflow-serving

Thanks!

[Byest]

@MtDersvan Thanks so much for your work and the tutorial. It's super appreciated. =) 👍👍👍💯