SageMaker exhibits unpredictable behavior when passing entry point to Model. #3772
Description
Describe the bug
SageMaker exhibits unpredictable behavior when passing entry point to Model.
To reproduce
We have an estimator that drops a single pickle file loaded into model_data. This is now an s3 location. inference.py specifies a inference protocol (model_fn, predict_fn,...).
model = sagemaker.model.Model(
'123456789.dkr.ecr.us-east-2.amazonaws.com/xxx/sagemaker-container:latest',
role=role,
sagemaker_session=sagemaker_session,
entry_point='inference.py',
model_data = estimator.model_data,
predictor_cls=sagemaker.predictor.Predictor,
code_location='s3://some-bucket/tmp/sm-dev/code',
)
The expected behavior here is that we use a local file inference.py as an entry point. It is loaded to s3://some-bucket/tmp/sm-dev/code at deploy time. However, what happens is that sagemaker correctly loads the code to this location but silently loses the inference.py script and throws a NotImplementedError because it cannot find model_fn.
On the other hand this WILL work if we do not pass code_location and use default code location. What I believe is happening is that we are skipping the repack step because we passed a code_location which was parsed to key_prefix (not sure the reasoning for this logic:
https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/model.py#L544-L546
Then SageMaker looks inside the model_data instead of SAGEMAKER_SUBMIT_DIRECTORY for code for some reason...
** Additional Context **
Here is the container def heading in. It looks correct.
{
'Image': '123456789.dkr.ecr.us-east-2.amazonaws.com/xxx/sagemaker-container:latest',
'Environment': {
'SAGEMAKER_PROGRAM': 'inference.py',
'SAGEMAKER_SUBMIT_DIRECTORY': 's3://some-bucket/tmp/sm-dev/code/sagemaker-container-2023-04-11-14-48-47-013/sourcedir.tar.gz',
'SAGEMAKER_CONTAINER_LOG_LEVEL': '20',
'SAGEMAKER_REGION': 'us-east-2'
},
'ModelDataUrl': 's3://uip-datalake-bucket-prod/tmp/sm-dev/output/sagemaker-preprocessing-container-2023-04-11-14-29-56-942/output/model.tar.gz'}