Clarity issues when creating keras raw serving signature with multipe inputs hosted on ai platform prediction service

[yrianderreumaux]

Thanks in advance for any guidance on this issue and I apologize if I am missing something in the docs. However, despite attempting related solutions (e.g., #1108, #1885, #1906 etc.) I have failed to successfully create two export signature defs one that would allow of raw text predictions via ai platform's prediction service and one that would allow for the examplegen component example output (example_gen.outputs['examples']) to be used for model evaluation in the evaluator component.

For reference, I am following the taxi example closely, with the main difference being that I am pulling my own data via BigQuery with BiqQueryExampleGen.

dependencies

1. tfx[kfp]==0.30.0
2. python 3.7

My latest attempt follows this solution which integrates a separate export signature for raw data via a MyModule(tf.Module) class. Similar to the author @jason-brian-anderson, we avoided the use of tf.reshape because we use the _fill_in_missing operation in our preprocessing_fn which expects and parses sparsetensors. Below is the code embedded within the scope of the run_fn.

class MyModule(tf.Module):
      def __init__(self, model, tf_transform_output):
          self.model = model
          self.tf_transform_output = tf_transform_output
          self.model.tft_layer = self.tf_transform_output.transform_features_layer()

      @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string, name='examples')])
      def serve_tf_examples_fn(self, serialized_tf_examples):
          feature_spec = self.tf_transform_output.raw_feature_spec()
          feature_spec.pop(features.LABEL_KEY) 
          parsed_features = tf.io.parse_example(serialized_tf_examples, feature_spec)
          transformed_features = self.model.tft_layer(parsed_features)
          return self.model(transformed_features)

      @tf.function(input_signature=[tf.TensorSpec(shape=(None), dtype=tf.string, name='raw_data')])
      def tf_serving_raw_input_fn(self, raw_data):
          raw_data_sp_tensor = tf.sparse.SparseTensor(
              indices=[[0, 0]],
              values=raw_data,
              dense_shape=(1, 1)
          )
          parsed_features = {'raw_data': raw_data_sp_tensor, }
          transformed_features = self.model.tft_layer(parsed_features)
          return self.model(transformed_features)

  module = MyModule(model, tf_transform_output)

  signatures = {"serving_default": module.serve_tf_examples_fn,
                "serving_raw_input": module.tf_serving_raw_input_fn,

                }
  tf.saved_model.save(module,
                      export_dir=fn_args.serving_model_dir,
                      signatures=signatures,
                      options=None,
                      )

The keras model expects 3 inputs: 1 DENSE_FLOAT_FEATURE_KEY and 2 VOCAB_FEATURE_KEYS.

The error I am currently experiencing is "can only concatenate str (not "SparseTensor") to str" which is occurring at parsed_features = {'raw_data': raw_data_sp_tensor, }

I also attempted to manually create the feature spec re:(https://github.com/tensorflow/tfx/pull/1906/files) but there were naming convention issues and I was unable to expose schema2tensorspec to ensure similar expected input names/data types.

Any and all help is welcome. I am posting this issue as a docs issue because there does not seem to be any consensus or of official documentation regarding creating a raw input signature def with multiple inputs.

[yrianderreumaux]

hi @arghyaganguly,

any updates?

Thanks,
Yrian

[ConverJens]

Hi @yrianderreumaux,

I think you might have missed a thing or two:
First: the Evaluator usually works perfectly well with raw data as this is how the basic serving signature is created (see https://github.com/tensorflow/tfx/blob/master/tfx/examples/bert/cola/bert_cola_pipeline.py#L144)
Second: your first serving signature is operating on raw data so just passing raw examples to the Evaluator will likely work fine.

Could you explain further what your are trying to achieve with your second serving fn?

Think of it this way: the serving fn defines the "data interface" (input) to the model, the Evaluator uses whatever interface is specified. Your first serving fn is defined to take "raw data", i.e. output from ExampleGen.

To explain what is happening in your serving fn:

feature_spec = self.tf_transform_output.raw_feature_spec() # get raw data proto spec
feature_spec.pop(features.LABEL_KEY) # remove label since it isn't present for serving
parsed_features = tf.io.parse_example(serialized_tf_examples, feature_spec) # add operation to parse byte string ("raw data") to tf Example
transformed_features = self.model.tft_layer(parsed_features) # pass tf Example through tft preprocessing layer
return self.model(transformed_features) # connect to model input. model will receive preprocessed features

In my pipeline, I use the opposite approach to feed the preprocessed data (materialized by Transform) to Evaluator and this requires an additional serving fn which doesn't apply the tft layer.

Let me know if you still have issues or questions!

[yrianderreumaux]

Hi @ConverJens,

I really appreciate you taking the time to respond to my question as well as clarifying the different features of the serving fn. I am clearly new to TFX and I realize that I may have added confusion (both to myself and to others) when I used the term "raw data" and so I will attempt to clarify what our goal is with the two serving functions.

In our pipeline, ExampleGen pulls in data from BigQuery using BigQueryExampleGen. Thus "raw data" refers to JSON data. The ExampleGen component splits this data for training and evaluation, which it emits as tf.example records. The tf.example records are then fed into TFT, which preprocesses our data (e.g., computing a vocabulary) before it is fed into the model.

The motivation for having two separate serving functions is that we would like to have one that is used in the evaluator component that expects tf.example records, and a second that is used when the model is served that expects JSON data. More specifically, we would like to define this second signature in a way that would allow a tf transform coupled model to be hosted on ai platform's prediction service that expects JSON data as well as serializes and transforms this incoming data consistent with the model's expected inputs. That way, we would not have to create a full serialized tf.example on the client side prior to submission.

In sum, I am struggling to create a second signature that takes json data with multiple different data types coming from BigQuery into a digestible format to be sent to our prediction api (not training).

[ConverJens]

@yrianderreumaux Thanks for the clarification, now I understand the issue!

As far as I can tell, your first serving fn can be used for inputing tf Examples, hence it should work with the evaluator.

As for serving json strings directly: depending on how you are hosting your model, this might not be an issue. If your are using tf serving, it supplies a http port for REST calls in json format and the parsing to tf Example is done server side: https://www.tensorflow.org/tfx/serving/api_rest#start_modelserver_with_the_rest_api_endpoint. In this case it shoud be sufficient with the first serving fn only.

If you are hosting in some other way (kfserving, a custom service etc) you will need to change your second serving fn to handle that mapping from json string to tf example. Perhaps something similar to (in your def tf_serving_raw_input_fn(self, raw_data) fn):

# Disclaimer: this code is completely untested and most likely has some issue.
from tfx.components.example_gen import utils as example_gen_utils
...
json_dict = json.loads(json_string) # json to dict
parsed_features = example_gen_utils.dict_to_example(json_dict) # TFX contains a utility fn for converting a dict to (unserialized) tf Example
transformed_features = self.model.tft_layer(parsed_features)

But apart from that I would raise some concerns as to NOT using tf Examples for serving:

1. protobuf is faster than http and it is also smaller in size
2. when using a model with a tft layer the data needs to in tf Example format. Serialization needs to happen, being on the client or server side. But note that this is generally a cheap operation and not usually something that needs to be avoided.
3. json is also not type safe in anyway compared to protobuf. You would rather have your serving failing on the client side on single data points than mess up an entire batch server side due to one bad data point.

But note that the same util method I used in the code example above can be used client side to easily build tf Examples from your json.

[yrianderreumaux]

Fantastic, that was an exceeding helpful response @ConverJens and now I have a clear picture about what I need to do and what my options are. I also had not appreciated the cons of not using tf examples for serving and will weigh that in my decision. Once again, I can't thank you enough for being kind enough to engage and share your knowledge.

Best,
Yrian

[ConverJens]

Happy to help @yrianderreumaux and best of luck!

[zion-b]

@yrianderreumaux, can you please share how did you eventually handle this?

I'm using a TFX pipeline (w/ transformer) endpoint on vertexAI - the default signature from the tutorial code is asking for a single input examples and I'm not sure what to pass on to the client.predict(endpoint, instances).
Tried something in the lines of: instances = [{"examples": Dict/List[Dict]}] but nothing seems to work.
Using the pb directly or through the saved_model works great, just having trouble with the format for the VertexAI REST API serving.

[pindinagesh]

@yrianderreumaux

Can you please confirm if your issue is resolved so that we can close this issue? Thanks!

[hartofdaniel]

@zion-b Did you manage to solve this issue? I have been fighting with it over the last month with no success :(

[zion-b]

Yes (with help), not sure if this is the intended solution because it's a bit cumbersome - so would very much appreciate feedback.

This is what I use:

You need to create a tf serialized_example, use serialize_example from here.

Once you have the proper example tensor you need to encode it as base64, so assuming you're using serialize_example and now you have an example_tensor

b64_example = base64.b64encode(example_tensor).decode("utf-8")
instances = [{'b64': b64_example}]

and this instances could be used in :

response = client.predict(endpoint=endpoint, instances=instances)

[hartofdaniel]

@zion-b Thank you for pointing me in the right direction. I have tried many similair solutions including that one and I seem to keep getting exception when posting to the Vertex AI endpoint:

InvalidArgument: 400 {
    "error": "Could not parse example input, value: 'CuQBCg8KBmdlbmRlchIFCgMKAUYKIwoac3RheV9pbl9jdXJyZW50X2NpdHlfeWVhcnMSBRoDCgECChMKCm9jY3VwYXRpb24SBRoDCgEKChsKEnByb2R1Y3RfY2F0ZWdvcnlfMhIFGgMKAQIKFwoObWFyaXRhbF9zdGF0dXMSBRoDCgEACg8KA2FnZRIICgYKBDAtMTcKFgoNY2l0eV9jYXRlZ29yeRIFCgMKAUEKGwoKcHJvZHVjdF9pZBINCgsKCVAwMDA2OTA0MgobChJwcm9kdWN0X2NhdGVnb3J5XzESBRoDCgEF'\n\t [[{{function_node __inference_serve_tf_examples_fn_3188}}{{node ParseExample/ParseExampleV2}}]]"
}

All methods seem to return the same error unfortunately. Any ideas on what I should do or how I can debug it? It does not seem to have any relating logs in the endpoint. I know the serialization is working as I am able to parse it back so it may be an issue with the base64?

The issue might be the serving function

[zion-b]

I've used the same serving function from the example tutorial, can you share a snippet of the request you're sending and the parsing to instances?

[entrpn]

@zion-b take a look at this notebook on deploying TFHub models to Vertex AI endpoints and call it from apis and curl. Might have what you are looking for https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb

[zion-b]

Thanks @entrpn!
I think that example is the image version of the solution I described.
I was surprised that there's no tabular data example for like the Taxi Keras tutorial , but with VertexAI deployment..

[singhniraj08]

@zion-b, Please refer to this example for tabular data example on training and serving on Vertex AI. Hope it helps.

[singhniraj08]

@yrianderreumaux,

Please let us know if this answer helps your use case. Thank you!

[singhniraj08]

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!