Attach and deploy doesn't create the endpoint

[aninoy]

Describe the bug
When trying to attach and deploy a previously completed training job, the deploy function fails healthcheck and the logs show errors like:
FileNotFoundError: [Errno 2] No such file or directory: 'nginx': 'nginx'

To reproduce
I'm using the keras script mode example for creating my model, train, attach and deploy scripts.

Code for starting a training job:

from sagemaker.tensorflow import TensorFlow

role = sagemaker.get_execution_role()
estimator = TensorFlow(base_job_name='training-job',
                       entry_point='model.py',
                       source_dir=source_dir,
                       output_path=output_path,
                       role=role,
                       framework_version='1.15.0',
                       py_version='py3',
                       hyperparameters=hyperparameters,
                       train_instance_count=1,
                       train_instance_type=train_instance_type,
                       metric_definitions=keras_metric_definition)

train_channel = sagemaker.session.s3_input(train_input_path)
valid_channel = sagemaker.session.s3_input(validation_input_path)
test_channel = sagemaker.session.s3_input(test_input_path)

data_channels = {
    'train': train_channel, 
    'val': valid_channel,
    'test': test_channel
}
estimator.fit(inputs=data_channels)

Code for attaching and deploying:

import logging
import argparse
import sagemaker as sm

session = sm.session.Session(default_bucket=bucket_name)
role = sm.get_execution_role(sagemaker_session=session)

estimator = sm.estimator.Estimator.attach(args.model_name)
logging.info("Deploying model %s", args.model_name)
predictor = estimator.deploy(initial_instance_count=args.instance_count, 
                                 instance_type=args.instance_type,
                                 endpoint_name=args.endpoint_name,
                                 update_endpoint=False
                                )

Expected behavior
The endpoint should be deployed and ready to be used via rest API or sagemaker python sdk's predict method.

Screenshots or logs
Add any other context about the problem here.

System information

• SageMaker Python SDK version: 1.51.4
• Framework name: Tensorflow (custom keras model)
• Framework version: 1.15.0
• Python version: 3.6 (script mode)
• CPU or GPU: GPU
• Custom Docker image (Y/N): N
• Model Container Image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-training:1.15.0-gpu-py3

[nadiaya]

Could you post what is the args.model_name value is in your case?

Usually to attach the estimator you are expected to use training-job-name because we are getting training job details from the SageMaker backend based on it:

        job_details = sagemaker_session.sagemaker_client.describe_training_job(
            TrainingJobName=training_job_name
        )

and then python-sdk parses the job details returned from SageMaker and based on that decides what image to use for training and for inference https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/estimator.py#L481-L532

[aninoy]

Yes, the model_name is the same as the training job name that successfully created model artifacts. Something along the lines of sagemaker-grader-2020-03-29-09-10-37-784
The training job image name is specified as 763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-training:1.15.0-gpu-py3. The container image name under 'Models' for that model name is also noted as 763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-training:1.15.0-gpu-py3.

[aninoy]

@nadiaya any more insight here?

[chuyang-deng]

Hi @aninoy, sorry for the delayed response. Looks like the estimator is still trying to use the training image (763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-training:1.15.0-gpu-py3) to deploy the model.

Starting from TensorFlow version 1.13.0, the our TensorFlow containers have split the training and serving, with different docker image URIs. However, with the attach() method, the training image will be extracted to init_params and then passed into constructor: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py#L597

A workaround for your case is that you can update your estimator.image to "763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-inference:1.15.0-gpu-py3"

[aninoy]

Hi @ChuyangDeng thanks so much for your response. I'll try that workaround next.

[cfregly]

I had the same issue with create_model_from_job and I can confirm that passing the image is a successful workaround.

primary_container_image is used in my case as follows:

sess.create_model_from_job(name=model_b_name,
                           training_job_name=training_job_name,
                           role=role,
                           primary_container_image='763104351884.dkr.ecr.{}.amazonaws.com/tensorflow-inference:2.1.0-cpu-py36-ubuntu18.04'.format(region))

[nadiaya]

I just realized that you were calling attach from the base Estimator classsm.estimator.Estimator.attach(args.model_name) which would just use the same exact image that is used for training as it is supposed to.

But since we have different images for TensorFlow training and hosting, in order for the correct image to be picked you need to use TensorFlow estimator class that has special logic to choose the right image.

Here is an example on how to correctly attach using TensorFlow estimator:

from sagemaker.tensorflow import TensorFlow

estimator = TensorFlow(entry_point='mnist.py',
                             role=role,
                             train_instance_count=1,
                             train_instance_type='ml.p2.xlarge',
                             framework_version='1.15.0',
                             py_version='py3')

training_job_name = "test-tf-sm-async-ticket-issues-1379"
estimator.fit(inputs=inputs, wait=False, job_name=training_job_name)

# and now reattach using TensorFlow class
estimator_attached = TensorFlow.attach(training_job_name=training_job_name)

predictor = estimator_attached.deploy(....)

Other option would specify the image directly as mentioned above.

I am removing the bug label since everything works as expected.

[aninoy]

@nadiaya Thanks for looking into this again. I probably picked up this sample for re-attaching for a deploy from the wrong non-tensorflow example. I didn't realize the estimator base class wouldn't have that extra logic for image selection in case of tensorflow images. Thanks for clarifying that again.