Sagemaker picks up old random model archive instead of the one pointed to.

[whatdhack]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Tensorflow Serving
• Framework Version: 1.14
• Python Version:3.6
• CPU or GPU:CPU
• Python SDK Version:
• Are you using a custom image:No

Describe the problem

Use the following code to create a Sagemake endpoint. Although I point the archive to a custom bucket, Sagemeake seems to point to a different bucket in the model.

import os
import sagemaker
from sagemaker import get_execution_role

sagemaker_session = sagemaker.Session()
role = get_execution_role()

from  sagemaker.tensorflow.model import TensorFlowModel
from  sagemaker.tensorflow.serving import Model as TensorFlowServingModel
saved_model_file = "s3://xxxxx/models/uncompiled/mymodel/mymodel.tgz"
model = TensorFlowServingModel(role=role,
                               model_data= saved_model_file,
                               framework_version = "1.14.0",
                               name = "mymodel-06",
                               env = {'SAGEMAKER_MODEL_SERVER_TIMEOUT' : '3600' },
                               entry_point ="inference.py"
                              )

predictor = model.deploy(initial_instance_count=1,
                                         instance_type='ml.c5.4xlarge',
                                      endpoint_name="mymodel-ep-06",
                                      )

After the above is completed (which takes a long time!!), checking the model in console ( Amazon Sagemaker > Models > mymodel06 ) shows that model data is pointing to somewhere else. Secondly, molde.tar.gz content seems to be an older buggy version. Am I missing something in the above ?

Model data location
s3://sagemaker-us-west-2-???????????/mymodel-06/model.tar.gz

The mymodel.tgz archive is as follows.

mymodel/375000/
mymodel/375000/saved_model.pb
mymodel/375000/variables/
mymodel/375000/variables/variables.index
mymodel/375000/variables/variables.data-00000-of-00001
code/inference.py
code/requirements.txt

I downloaded the model.tar.gz that the Sagemaker model is pointing to. It has an older version of inference.py and the requirements.txt is missing.

mymodel/375000/
mymodel/375000/saved_model.pb
mymodel/375000/variables/
mymodel/375000/variables/variables.index
mymodel/375000/variables/variables.data-00000-of-00001
code/inference.py
code/

Minimal repro / logs

See above

• Exact command to reproduce:

[nadiaya]

I do not think TensorFlowServingModel is a sagemaker-python-sdk class.
Could you share this class as well to help us reproduce the issue?

[whatdhack]

from sagemaker.tensorflow.serving import Model as TensorFlowServingModel

[chuyang-deng]

Hi @whatdhack, sorry for the late response.

Can you confirm if you are using the latest version of SageMaker Python SDK? Our sagemaker.tensorflow.serving does not have TensorFlowServingModel, instead, it has Model class (https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/serving.py#L122)

The usage of the Model can be found in our example notebook:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_serving_container/tensorflow_serving_container.ipynb

from sagemaker.tensorflow.serving import Model

Alos, when you check your model data in SageMaker console. You should be able to find your model from SageMaker->Endpoints->[your endpoint with the deployed model]->Endpoint configuration settings(and then under Production variants section), to avoid accidentally looking at a wrong model.

[laurenyu]

I believe there's a limitation with the repack_model function where it uses Session's default bucket (sagemaker-{region}-{account number}). It sounds like there's another bug on top of that with the files not being correctly copied into the repacked model. I'll try to see if I can push a fix soon.

In the meantime, can you try using the sagemaker.model.FrameworkModel class? you will need to pass predictor_cls=sagemaker.tensorflow.serving.Predictor in the constructor, but I think most other things should be the same.

[laurenyu]

@whatdhack I might've misspoke. Can you try including the following parameters when creating model = TensorFlowServingModel:

dependencies=["path/to/requirements.txt"],
code_location="s3://xxxxx/models/uncompiled/mymodel/mymodel.tgz",

[laurenyu]

closing due to lack of activity

[ckang244]

@laurenyu I'm dealing with this exact same issue. Would it be possible to reopen this? SDK version is 2.5.0.

I'm trying to deploy after updating some script issues inside model.tar.gz and reuploading to S3. In the console the Endpoint displays to a Sagemaker-generated bucket with an old version of the model (even if the timestamp in the S3 path is updated to the current time). It doesn't show the path of the model I pass in model_data as shown below. This happens in both Sagemaker Studio notebook as well as locally using Python.

sagemaker_session = sagemaker.Session()
role = 'arn:aws:iam::12345676890:role/service-role/AmazonSageMaker-ExecutionRole-12345676890'

serving_model = MXNetModel(model_data='s3://my-bucket/model/model.tar.gz',
                           role=role,
                           entry_point='inference.py',
                           py_version='py3',
                           framework_version='1.6.0')
predictor = serving_model.deploy(initial_instance_count=1,
                                 instance_type='ml.c5.xlarge')

[laurenyu]

@ckang244 the TF and MXNet classes can be a little different at times, so can you open a new issue for this?

MXNetModel repacks your model.tar.gz, so that's why you're seeing a different S3 path. You can specify which S3 path to upload the repacked model to with code_location (docs).