Problems replicating web demo using TensorFlow Serving on Docker

[kjczarne]

Hi,
I was trying to run the toy CelebA model using TF Serving and I couldn't connect to the model.

Here is the command:

 docker run -p 8500:8500 --mount type=bind,source='/Users/user/Downloads/wit_testing',target=/models/wit_testing -e MODEL_NAME=wit_testing -it tensorflow/serving

I ran TensorBoard on localhost:6006 and then I configured the WIT tool as follows:

I serialized the model to a ProtoBuf as instructed, serialized data to a TFRecords file but:

1. The photos on the right do not display in the same way they are displayed in the Web Demo, there are just dots representing datapoints.

2. Whenever I try to run an inference call to the model, I get bad request error (500). The error I am getting:

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Expects arg[0] to be float but string is provided"
        debug_error_string = "{"created":"@1560888632.009402000","description":"Error received from peer ipv6:[::1]:8500","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Expects arg[0] to be float but string is provided","grpc_status":3}"

Is there some convention in naming and paths here that I am missing? Or am I doing something else completely wrong?

[jameswex]

In order for images in TF examples to show up as images in the What-If Tool, you need to encode the image as a bytes string and put it in a feature in the example named "image/encoded". Check out https://colab.research.google.com/github/PAIR-code/what-if-tool/blob/master/WIT_Smile_Detector.ipynb#scrollTo=4H2nX-2dEgsR for an example of packaging images into TF examples in a way that the What-If Tool will properly display them.

When you click on a dot in WIT, for that example, what does the image feature display as in the left side panel?

Then if you encode your image in this way, you need your TF model to decode the encoded image string into a tensor to then perform learning on. The tf.io.decode_image op should work for this (https://www.tensorflow.org/api_docs/python/tf/io/decode_image).

The inference error may have to do with a mismatch between the format that each feature takes in your dataset (which you can view in WIT by clicking on the dots and looking at their values on the left side), and the format of the features that the model is expecting. When the model parses the tf examples for inference, how does it extract the feature values?

Feel free to share code, saved models, and example files if you want.

[kjczarne]

Thank you very much! I was able to display the photos. As for the second problem, here are the functions I use to serialize the data:

def load_byte_img(im_bytes, IMAGE_H, IMAGE_W):
        buf = BytesIO(im_bytes)
        im = np.array(Image.open(buf).resize((IMAGE_H, IMAGE_W)), dtype=np.float64) / 255.
        return np.expand_dims(im, axis=0)

def image_bytestring(path_to_image):
    with open(path_to_image, 'rb') as f:
        return f.read()


def image_example(image_string, label, extra_features=None):
    """
    taken from TensorFlow TFRecords tutorial
    :param image_string: image bytestring
    :param label: image label (integer)
    :return: tf.Example
    """
    image_shape = tf.image.decode_jpeg(image_string).shape
    feature = {
        'image/height': _int64_feature(image_shape[0]),
        'image/width': _int64_feature(image_shape[1]),
        'image/depth': _int64_feature(image_shape[2]),
        'image/label': _int64_feature(label),
        'image/encoded': _bytes_feature(image_string),
    }
    if extra_features is None:
        pass
    else:
        for k, v in extra_features.items():
            feature[k] = v
    return tf.train.Example(features=tf.train.Features(feature=feature))


def labels_to_integers(label_list):
    """
    converts a list of string labels to integer list
    :param label_list: list of labels (str)
    :return: list
    """
    mapping = {v:k for k, v in enumerate(list(set(label_list)))}
    return [mapping[i] for i in label_list]


def write_tfrecord(df, 
                   image_path_series='x_col', 
                   label_series_name='y_col', 
                   cols_to_ignore=[],  
                   outfile='images.tfrecord',
                   verbose=True):
    """
    writes TFRecord files
    :param df: pd.DataFrame with at least 2 columns containing paths to images and labels
    :param image_path_series: string naming the column with paths, default is 'x_col'
    :param label_series_name: string naming the column with labels, default is 'y_col'
    :param cols_to_ignore: list of columns from the DataFrame to not treat as features and ignore them
    :param outfile: path to output TFRecord file
    :param verbose: verbose mode, True by default
    :return: None
    """
    # truncate df, put label separately, remove non-feature columns
    cols = list(df.columns)
    
    for i in cols_to_ignore:
        cols.remove(i)
    
    sub_df = df.loc[:, cols]
    examples = []
    sub_df[label_series_name] = labels_to_integers(df[label_series_name])
    for i in sub_df.iterrows():
        series = i[1]  # underlying pd.Series
        im_bytes = image_bytestring(series[image_path_series])
        label = series[label_series_name]
        extra_feature_names = list(series.keys())
        extra_feature_names.remove(label_series_name)
        extra_feature_names.remove(image_path_series)
        series = series.loc[extra_feature_names]
        # ignore image path and label for extra features
        extra_features = dict()
        for k, ft in series.items():
            if type(ft) == int:
                val = _int64_feature(ft)
            elif type(ft) == object:
                val = _bytes_feature(ft)
            elif type(ft) == float:
                val = _float_feature(ft)
            else:
                raise ValueError('Unsupported data type!')
            extra_features[k] = val
        im_example = image_example(im_bytes, label, extra_features)
        examples.append(im_example)
        if verbose:
            print(f'Generated {len(examples)} examples')
    
    def write(outfile):
        with tf.io.TFRecordWriter(outfile) as writer:
            for example in examples:
                writer.write(example.SerializeToString())
    
    write(outfile)


def _parse_image_function(example_proto, additional_features=None):
    """
    taken from TensorFlow TFRecords tutorial
    :param example_proto: tf.Example
    :return: parsed binary data, human readable
    """
    image_feature_desc = {
            'image/height': tf.io.FixedLenFeature([], tf.int64),
            'image/width': tf.io.FixedLenFeature([], tf.int64),
            'image/depth': tf.io.FixedLenFeature([], tf.int64),
            'image/label': tf.io.FixedLenFeature([], tf.int64),
            'image/encoded': tf.io.FixedLenFeature([], tf.string),
    }
    if additional_features is None:
        pass
    else:
        for k, v in additional_features.items():
            image_feature_desc[k] = v
    # Parse the input tf.Example proto using the dictionary above.
    return tf.io.parse_single_example(example_proto, image_feature_desc)


def parse_image_dataset(raw_image_dataset):
    """
    transforms raw binary data in a tf.Dataset to human readable form
    :param raw_image_dataset: tf.Dataset with raw binary data
    :return: human readable tf.Dataset instance
    """
    return raw_image_dataset.map(_parse_image_function)


def serialize_model(model_file, model_version='1', weights_file=None):
    """
    saves model as a protobuf compatible with TensorFlow Serving
    :param model_file: path to JSON or HDF5 file with the model
    :param model_version: string specifying model version name
    :param weights_file: path to optional HDF5 file with weights
    :return: None
    """
    if weights_file is None:
        model = tf.keras.models.load_model(model_file)
    else:
        with open(model_file, 'r') as f:
            model = tf.keras.models.model_from_json(f.read())
        model.load_weights(weights_file)
    tf.saved_model.save(model, os.getcwd() + '/' + model_version)

Then I run:

write_tfrecord(df, 'Smiling', 'image_id', ['Unnamed: 0'])
# column 'Smiling' as labels, column 'image_id' points to files, column 'Unnamed: 0' is ignored

The model is exactly the one used in CelebA demo but here are all the files:
https://drive.google.com/open?id=1qCEoWWYSpugExtTivNoklJz_nw0qyATX

I have a feeling that I should probably use a custom predict function for the Predict API but have no clue on how to define it when using TensorFlow Serving on Docker. Do I overload original predict function or is there a better way to do this?

[jameswex]

I don't have any experience with taking a keras model and pushing it to TF Serving for querying using the TFServing Classification API with serialized TF Examples. So it's possible that the way the model is saved and then served, it doesn't know how to correctly parse the serialized tf.Example protos being provided to it.

For models pushed to TF Serving, we've always trained TF estimators and served them with input receiver functions such as documented here: https://www.tensorflow.org/guide/saved_model#using_savedmodel_with_estimators

[florence27]

@kjczarne did you manage to run an inference call to your Keras model? I am currently stuck with the same error that you were getting & I can't figure out a way to make it work.

[kjczarne]

@florence27 It's been a while ago but I remember we ended up not using TensorBoard for that. 😔

[jameswex]

Since you are using a keras model, you may be better off running WIT in notebook mode (in colab or jupyter notebook), instead of in TensorBoard and using your own custom prediction function in the notebook that calls directly into the model, as opposed to querying a model served by TF serving.

See https://pair-code.github.io/what-if-tool/learn/tutorials/notebooks/ for notebook model details.