-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Closed
Description
Describe the bug
XGB DLC fails to load the model with boto3>=1.29 so our workaround was to pin boto3 in ModelBuilder as such:
ModelBuilder(
model_path=XGB_RESOURCE_DIR,
inference_spec=xgb_inference_spec,
schema_builder=SchemaBuilder(xgb_test_sets.x_test, xgb_test_sets.y_test),
dependencies={
boto3<1.29.0
},
)
Because SageMaker 2.199 does not support that boto3 version, it is downcast to 2.197. The issue here is that 2.197 does not have ModelBuilder serve modules
To reproduce
A clear, step-by-step set of instructions to reproduce the bug.
Expected behavior
XGB container should load and host the model successfully
Screenshots or logs
ModelBuilder: DEBUG: Successfully installed PyYAML-6.0.1 attrs-23.1.0 boto3-1.28.85 botocore-1.31.85 cloudpickle-2.2.1 contextlib2-21.6.0 dill-0.3.7 google-pasta-0.2.0 importlib-resources-6.1.1 inference-1.0.0 jsonschema-4.20.0 jsonschema-specifications-2023.11.1 multiprocess-0.70.15 pathos-0.3.1 pkgutil-resolve-name-1.3.10 platformdirs-4.0.0 pox-0.3.3 ppft-1.7.6.7 referencing-0.31.1 rpds-py-0.13.2 s3transfer-0.7.0 sagemaker-2.197.0 schema-0.7.5 smdebug_rulesconfig-1.0.1 tblib-1.7.0
ModelBuilder: DEBUG: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ModelBuilder: DEBUG: sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
ModelBuilder: DEBUG: sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml
ModelBuilder: DEBUG: Traceback (most recent call last):
ModelBuilder: DEBUG: File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_modules.py", line 258, in import_module
ModelBuilder: DEBUG: module = importlib.import_module(name)
ModelBuilder: DEBUG: File "/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
ModelBuilder: DEBUG: return _bootstrap._gcd_import(name[level:], package, level)
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap_external>", line 843, in exec_module
ModelBuilder: DEBUG: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ModelBuilder: DEBUG: File "/miniconda3/lib/python3.8/site-packages/inference.py", line 10, in <module>
ModelBuilder: DEBUG: from sagemaker.serve.validations.check_integrity import perform_integrity_check
ModelBuilder: DEBUG: ModuleNotFoundError: No module named 'sagemaker.serve'
System information
A description of your system. Please provide:
- SageMaker Python SDK version: 2.199
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): XGBoost
- Framework version: 1.7-1
- Python version: 3.8
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
Add any other context about the problem here.
acere