Missing dependencies in SKLearn based endpoint/Batch Transform job

[lihip]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): SKLearn
• Framework Version: sagemaker 1.36.1
• Python Version: 3.6
• CPU or GPU: CPU
• Python SDK Version: sagemaker 1.36.1
• Are you using a custom image: yes, using entry point script

Describe the problem

I created an SKLearn estimator with dependencies parameter defined to include a python library on the instance (that is not installed on the container by default). Training using the estimator worked (dependency was installed), but when trying to create an endpoint for that model (using the deploy() method), it was failing due to import error of that library. Same happens with Batch Transform jobs.
I would expect that any dependency that was defined for the estimator, will also be available for the endpoint container. In any case, I couldn't find another way to import it to the endpoint as well.

Minimal repro / logs

CloudWatch error:
sagemaker_containers._errors.ImportModuleError: No module named 'xgboost'

[knakad]

Hi there @lihip !
Do you mind linking the script you're using where you're running into this issue? It would help me reproduce it and offer solutions.

Thanks!

[lihip]

Hi @knakad
The entry point script is attached.
The code of training and prediction:

sklearn = SKLearn(
    entry_point=script_path,
    train_instance_type="ml.m5.large",
    role=role,
    enable_network_isolation=True,
    output_path = output_location,
    code_location=source_location,
    hyperparameters={'min_child_weight': 2, 'max_depth':8},
    dependencies=['/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/xgboost'])

sklearn.fit({'train': train_input, 'validation': validation_input})
predictor = sklearn.deploy(initial_instance_count=1, instance_type="ml.m5.large")

xgboost_model.zip

[knakad]

Hmm..

I wonder if the xgboost folder isn't at the correct directory level.

Would you mind following these steps and letting me know what you see?

• Go into "Amazon SageMaker > Training Jobs > 'job_name' > Input data configuration: code > URI"
• Take the S3 URI, go to the S3 console and download the *.tar.gz file.
• Run the following in a terminal: tar -tvf file.tar and link me the contents.

[lihip]

The content is attached.
We can see both python script and the xgboost folder in the same level.

tar_content.txt

[lihip]

Could it be a python version issue?
On the one hand, the notebook kernel on which I installed the xgboost package (and the one I'm using as dependency) is python 3.6. On the other hand, SageMaker's container which downloads the package has python 3.5.
Also, when I packaged my sources by myself into sourcedir.tar.gz file (containing the entry point script and xgboost folder), I got the [Edit:] same error.

[ChoiByungWook]

Hello @lihip,

Thank you for bringing this up to our attention and apologies for the late response.

It looks like the issue is indeed the difference in the Python version.

We are looking into updating the container from installing 3.5 to 3.6, however we need to ensure we don't introduce any bugs and repercussions.
https://github.com/aws/sagemaker-scikit-learn-container/blob/master/docker/0.20.0/base/Dockerfile.cpu#L15

Thank you for your patience!

[aws-patlin]

I would like to add that it should be possible to install a dependency into the endpoint when creating the SKLearnModel that will be deployed (https://sagemaker.readthedocs.io/en/v1.36.1/sagemaker.sklearn.html#sagemaker.sklearn.model.SKLearnModel). The model will pass **kwargs to the FrameworkModel (https://sagemaker.readthedocs.io/en/v1.36.1/model.html#sagemaker.model.FrameworkModel) which takes a parameter dependencies.

This is definitely not an ideal customer experience, as it intuitively makes sense that an endpoint created from an estimator should use the same image and dependencies as the estimator. Can the python sdk team look into a solution for this?

[lihip]

Hi @ChoiByungWook
Is there any progress with supporting python 3.6 in the container?