Skip to content

ModelBuilder support for XGBoost container is impaired due to boto3 dependencies issues #4288

@gwang111

Description

@gwang111

Describe the bug
XGB DLC fails to load the model with boto3>=1.29 so our workaround was to pin boto3 in ModelBuilder as such:

 ModelBuilder(
    model_path=XGB_RESOURCE_DIR,
    inference_spec=xgb_inference_spec,
    schema_builder=SchemaBuilder(xgb_test_sets.x_test, xgb_test_sets.y_test),
    dependencies={
        boto3<1.29.0
    },
)

Because SageMaker 2.199 does not support that boto3 version, it is downcast to 2.197. The issue here is that 2.197 does not have ModelBuilder serve modules

To reproduce
A clear, step-by-step set of instructions to reproduce the bug.

Expected behavior
XGB container should load and host the model successfully

Screenshots or logs

ModelBuilder: DEBUG:     Successfully installed PyYAML-6.0.1 attrs-23.1.0 boto3-1.28.85 botocore-1.31.85 cloudpickle-2.2.1 contextlib2-21.6.0 dill-0.3.7 google-pasta-0.2.0 importlib-resources-6.1.1 inference-1.0.0 jsonschema-4.20.0 jsonschema-specifications-2023.11.1 multiprocess-0.70.15 pathos-0.3.1 pkgutil-resolve-name-1.3.10 platformdirs-4.0.0 pox-0.3.3 ppft-1.7.6.7 referencing-0.31.1 rpds-py-0.13.2 s3transfer-0.7.0 sagemaker-2.197.0 schema-0.7.5 smdebug_rulesconfig-1.0.1 tblib-1.7.0
ModelBuilder: DEBUG:     WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ModelBuilder: DEBUG:     sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
ModelBuilder: DEBUG:     sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml
ModelBuilder: DEBUG:     Traceback (most recent call last):
ModelBuilder: DEBUG:       File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_modules.py", line 258, in import_module
ModelBuilder: DEBUG:         module = importlib.import_module(name)
ModelBuilder: DEBUG:       File "/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
ModelBuilder: DEBUG:         return _bootstrap._gcd_import(name[level:], package, level)
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap>", line 991, in _find_and_load
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap_external>", line 843, in exec_module
ModelBuilder: DEBUG:       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ModelBuilder: DEBUG:       File "/miniconda3/lib/python3.8/site-packages/inference.py", line 10, in <module>
ModelBuilder: DEBUG:         from sagemaker.serve.validations.check_integrity import perform_integrity_check
ModelBuilder: DEBUG:     ModuleNotFoundError: No module named 'sagemaker.serve'

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.199
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): XGBoost
  • Framework version: 1.7-1
  • Python version: 3.8
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): N

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions