TFServing Compatibility issue with the TF2.0 model using feature columns

[brightcoder01]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  MacOS Mojave 10.14.6
• TensorFlow Serving installed from (source or binary):
  Use docker image: docker pull tensorflow/serving
• TensorFlow Serving version:
  1.14.0

Describe the problem

We use the same model definition, run it using tensorflow 1.14.0 and tensorflow 2.0.0 and then get two models. Host these two models with the same tensorflow/serving:latest docker image and send the same request. The expectation is that the response from these two models should be the same. Actually, the model from tf 1.14 is fine but the model from tf 2.0.0 returns error.

Exact Steps to Reproduce

1. docker pull tensorflow/serving
2. Download adult.data from https://archive.ics.uci.edu/ml/machine-learning-databases/adult/
3. Run the census_keras.py as follows using tf 1.14.0 and then get the SavedModel. Run docker run -t --rm -p 8501:8501 -v "$(pwd)/saved_models/census_keras/:/models/census_keras" -e MODEL_NAME=census_keras tensorflow/serving & to host this model using TF-Serving
4. curl -d '{"instances": [{"age": 16, "education": "Masters"}]}' -X POST http://localhost:8501/v1/models/census_keras:predict .We can get the correct response.
5. Run docker kill xxx to kill the launched image
6. Run the census_keras.py using tf 2.0.0 and the same docker command to host the model from tf2.0
7. Run the same curl command to test the service, and then returns the error message: { "error": "indices = 1 is not in [0, 1)\n\t [[{{node StatefulPartitionedCall/StatefulPartitionedCall/sequential/dense_features/education_embedding/education_embedding_weights/GatherV2}}]]" }

Source code / logs

census_data.py

import tensorflow as tf 
import pandas as pd 
from sklearn.model_selection import train_test_split

__COLUMN_NAMES = ['age', 'workclass','fnlwgt', 'education', 'education_num', 'marital_status',
                'occupation', 'relationship', 'race', 'gender', 'capital_gain',
                'capital_loss', 'hours_per_week', 'native_country', 'income_bracket']

__SELECT_COLUMN_NAMES = ['age', 'education', 'income_bracket']

def get_train_test_pandas_data():
    census = pd.read_csv("./data/adult.data", header=None, skipinitialspace=True)
    census.columns = __COLUMN_NAMES

    census['income_bracket'] = census['income_bracket'].apply(lambda label: 0 if label==' <=50K' else 1)
    census = census[__SELECT_COLUMN_NAMES]

    y_labels = census.pop('income_bracket')
    x_data = census

    x_train, x_test, y_train, y_test = train_test_split(x_data, y_labels, test_size=0.3)

    return x_train, x_test, y_train, y_test

def get_feature_columns():
    age = tf.feature_column.numeric_column("age", dtype=tf.int64)
    education = tf.feature_column.embedding_column(
        tf.feature_column.categorical_column_with_hash_bucket("education", hash_bucket_size=1000),
        dimension=100)

    feat_cols = [age, education]

    return feat_cols

census_keras.py

import tensorflow as tf 
import census_data
import pathlib
import time

if (tf.__version__ < '2.0'):
    tf.enable_eager_execution()

x_train, _, y_train, _ = census_data.get_train_test_pandas_data()

dataset = tf.data.Dataset.from_tensor_slices((dict(x_train), y_train))

dataset = dataset.shuffle(len(x_train)).batch(4)

feat_cols = census_data.get_feature_columns()

model = tf.keras.Sequential([
    tf.keras.layers.DenseFeatures(feature_columns=feat_cols),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
    ])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

model.fit(dataset, epochs=1)

__SAVED_MODEL_DIR = './saved_models/census_keras/{}'.format(int(time.time()))
pathlib.Path(__SAVED_MODEL_DIR).mkdir(parents=True, exist_ok=True)

tf.saved_model.save(model, export_dir=__SAVED_MODEL_DIR)

[brightcoder01]

Update the request from:
curl -d '{"instances": [{"age": 16, "education": "Masters"}]}' -X POST http://localhost:8501/v1/models/census_keras:predict
to
curl -d '{"instances": [{"age": [16], "education": ["Masters"]}]}' -X POST http://localhost:8501/v1/models/census_keras:predict
the models from both version works. Close this issue now.

The input shape of the models are different between tf1.4 and tf2.0.
TF1.4:
inputs['education'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: serving_default_education:0

TF2.0:
inputs['education'] tensor_info:
dtype: DT_STRING
shape: (-1, 1)
name: serving_default_education:0

[chengnignzhang]

hey @brightcoder01 good finding, can I check with you what version of tfs you are using for both tf models? I saw from above msg, you are using tfs 1.14 to support both tf1.14 and tf 2.0 model.
Can you help double confirm?
Because we are also using tensorflow serving 1.14 , but we cannot serve tf2.0 properly.

[brightcoder01]

hey @brightcoder01 good finding, can I check with you what version of tfs you are using for both tf models? I saw from above msg, you are using tfs 1.14 to support both tf1.14 and tf 2.0 model.
Can you help double confirm?
Because we are also using tensorflow serving 1.14 , but we cannot serve tf2.0 properly.

Yes, we are using tf serving 1.14 to serve the models from tf1.14 and tf2.0.

[workingloong]

tfs 1.14 can not serve tf 2.0 SavedModel with tf.feature_column.categorical_column_with_vocabulary_list.

[workingloong]

The error can be reproduced by followed codes

import time
import numpy as np
import tensorflow as tf 

def get_feature_columns():
    age = tf.feature_column.numeric_column("age", dtype=tf.int64)
#     education = tf.feature_column.categorical_column_with_hash_bucket(
#         'education', hash_bucket_size=4)
    
    education = tf.feature_column.categorical_column_with_vocabulary_list(
      'education', ['Bachelors', 'Master', 'Some-college'])
    education_one_hot = tf.feature_column.indicator_column(education)
    
    return [age, education_one_hot]

def get_input_layer():
    input_layers = {}
    input_layers['age'] = tf.keras.layers.Input(name='age', shape=(1,), dtype=tf.int64)
    input_layers['education'] = tf.keras.layers.Input(name='education', shape=(1,), dtype=tf.string)
    return input_layers

def custom_model(feature_columns):
    input_layers = get_input_layer()
    dense_feature = tf.keras.layers.DenseFeatures(feature_columns=feat_cols)(input_layers)
    dense = tf.keras.layers.Dense(10, activation='relu')(dense_feature)
    dense = tf.keras.layers.Dense(1, activation='sigmoid')(dense)
    return tf.keras.models.Model(inputs=input_layers, outputs=dense)

feat_cols = get_feature_columns()
model = custom_model(feat_cols)  
output = model.call({'age':tf.constant([[10],[16]]),
                     'education':tf.constant([['Bachelors'],['Master']])})
print(output)

export_dir = './saved_models/feature_columns/{}'.format(int(time.time()))
tf.saved_model.save(model, export_dir=export_dir)

[allenlavoie]

@rohan100jain could you look into this? Looks like a change to lookup table initialization has broken Session-based loading, possibly tensorflow/tensorflow@64586f1. Presumably related to loading multiple tables. If so we should add an integration test with multiple tables in tensorflow/python/saved_model/load_v1_in_v2_test.py

[allenlavoie]

Please assign @rohan100jain; apparently I have no access.

[brightcoder01]

Please assign @rohan100jain; apparently I have no access.

@rmothukuru Could you please assign this bug to @rohan100jain as @allenlavoie suggested. It seems that I don't have access either.

[gowthamkpr]

@rohan100jain Can you please take a look at this issue. Thanks!

[brightcoder01]

Hi, @gowthamkpr @rohan100jain, could you please share some updated information about this issue? It seems that it will block the TFServing usage with tf.feature_column.categorical_column_with_vocabulary_list.
Thanks. :)

[rohan100jain]

For the latest reproduction, the one with the table initialization issue - could you specify what version of TF you were using? I ran with the latest version (2.0.0) and seems to run fine.

[workingloong]

For the latest reproduction, the one with the table initialization issue - could you specify what version of TF you were using? I ran with the latest version (2.0.0) and seems to run fine.

Yes, the case runs fine with the latest version tf-serving 2.0.0 which is released on 22 Oct. I used tf-serving 1.14.0 because version 2.0.0 was not released previously. Thanks for your validation.

[rohan100jain]

So then i'm closing the issue now. Please reopen or file another issue if more problems appear with TF2.0

[xixiddd]

@rohan100jain Could we expect that SavedModel exported by TF 1.x can be serving normally using TF-Serving 2.x

[owenliang]

'{ "error": "indices = 1 is not in [0, 1)\\n\\t [[{{node sequential/dense_features_6/Ticket_embedding/Ticket_embedding_weights/GatherV2}}]]" }'

tensorflow==2.1.0
tensorflow-estimator==2.1.0

 tensorflow_model_server --version
TensorFlow ModelServer: 2.1.0-rc1+dev.sha.d83512c
TensorFlow Library: 2.1.0

Have the same problem, but with tensorflow 2.1.0, anyone same?