Numpy version issues when importing load_detector from alibi_detect.saving

[cfolan17]

I am currently having issues importing the load detector due to a NumPy being None when importing. The class below is what is used to load in the detector and provide predictions on a dataset. The code is being run on a AWS Sagemaker SKLearn Container framework version 1.0-1 for Batch Transformation.
The issue does not arise every time the container is run (About 25% of the time)

import sys
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
import os 
import numpy
import traceback
class DriftDetector:
    def __init__(self):
        self.is_init = False
    
    def load_detector(self, model_dir): 
        if self.is_init:
            return self.detector
        else:
            print(f"NUMPY VERSION....{numpy.__version__}")
            try:
                from alibi_detect.saving import load_detector
                self.detector = load_detector(model_dir)
                self.is_init = True
                print("MMD Drift Model Loaded ......")
                return self.detector
            except Exception as e:
                print(e)
                print(f"CHECKING NUMPY VERSION....{numpy.__version__}")
                traceback.print_tb(e.__traceback__)
    def detect(self, input_data, **_others):
        print(f"MMD Drift Model detecting ...... {_others}")
        import pandas as pd
        _detector = self.load_detector(_others['model_dir'])
        import contextlib
        with contextlib.suppress(ValueError):
            _result = _detector.predict(input_data)
            return pd.DataFrame([_result['data']])

The requirements.txt file for the container includes:
alibi-detect[tensorflow]==0.10.4
markupsafe==2.0.1
werkzeug==2.0.3
importlib-metadata==5.0.0
smart-open==5.2.1

The output of the logs are:

grkz9wuob7-algo-1-ywaz1 | MMD Drift Model detecting ...... {'model_dir': '/opt/ml/model'}
grkz9wuob7-algo-1-ywaz1 | NUMPY VERSION....1.22.4
grkz9wuob7-algo-1-ywaz1 | 2022-11-30 19:27:44.111746: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
grkz9wuob7-algo-1-ywaz1 | 2022-11-30 19:27:44.111781: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
grkz9wuob7-algo-1-ywaz1 | Unable to compare versions for numpy>=1.17: need=1.17 found=None. This is unusual. Consider reinstalling numpy.
grkz9wuob7-algo-1-ywaz1 | CHECKING NUMPY VERSION....1.22.4
grkz9wuob7-algo-1-ywaz1 |   File "/opt/ml/code/utils/driftdetecting.py", line 19, in load_detector
grkz9wuob7-algo-1-ywaz1 |     from alibi_detect.saving import load_detector
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/__init__.py", line 1, in <module>
grkz9wuob7-algo-1-ywaz1 |     from . import ad, cd, models, od, utils, saving
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/ad/__init__.py", line 3, in <module>
grkz9wuob7-algo-1-ywaz1 |     AdversarialAE = import_optional('alibi_detect.ad.adversarialae', names=['AdversarialAE'])
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/utils/missing_optional_dependency.py", line 101, in import_optional
grkz9wuob7-algo-1-ywaz1 |     module = import_module(module_name)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
grkz9wuob7-algo-1-ywaz1 |     return _bootstrap._gcd_import(name[level:], package, level)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/ad/adversarialae.py", line 9, in <module>
grkz9wuob7-algo-1-ywaz1 |     from alibi_detect.models.tensorflow.autoencoder import AE
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/models/tensorflow/__init__.py", line 8, in <module>
grkz9wuob7-algo-1-ywaz1 |     TransformerEmbedding = import_optional(
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/utils/missing_optional_dependency.py", line 101, in import_optional
grkz9wuob7-algo-1-ywaz1 |     module = import_module(module_name)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
grkz9wuob7-algo-1-ywaz1 |     return _bootstrap._gcd_import(name[level:], package, level)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/alibi_detect/models/tensorflow/embedding.py", line 3, in <module>
grkz9wuob7-algo-1-ywaz1 |     from transformers import TFAutoModel, AutoConfig
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/transformers/__init__.py", line 30, in <module>
grkz9wuob7-algo-1-ywaz1 |     from . import dependency_versions_check
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/transformers/dependency_versions_check.py", line 41, in <module>
grkz9wuob7-algo-1-ywaz1 |     require_version_core(deps[pkg])
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/transformers/utils/versions.py", line 123, in require_version_core
grkz9wuob7-algo-1-ywaz1 |     return require_version(requirement, hint)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/transformers/utils/versions.py", line 117, in require_version
grkz9wuob7-algo-1-ywaz1 |     _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
grkz9wuob7-algo-1-ywaz1 |   File "/miniconda3/lib/python3.8/site-packages/transformers/utils/versions.py", line 45, in _compare_versions
grkz9wuob7-algo-1-ywaz1 |     raise ValueError(

As you can see by checking the NumPy version, version 1.22.4 is installed but is not being picked up when loading in the detector.

[ascillitoe]

Hi @cfolan17, from the traceback it looks like the error is surfacing from the transformers package that is a dependency of ours.

It would be helpful to clarify if this is issue is purely related to transformers, or if it is something to do with alibi-detect (beyond us having transformers as a dependency of course!). It would be interesting to see if you get similar errors if you try to import transformers by itself in your container (without alibi-detect installed).

It would also be helpful to understand if this is actually to do with attempting to load a serialised detector (perhaps due to differences between the environment used to save the detector and the one you load it in), or is this purely an installation issue i.e. does the error occur when you simply try import alibi-detect in the container?

As for the error itself, I cannot explain why it is intermittent, but it looks similar to this issue? i.e. the already installed numpy package (looks like numpy 1.19.2 from this?) is not properly uninstalled when your new requirements are installed. Are you able to try using a lighter-weight container that doesn't have scikit-learn and numpy pre-installed? (they are core dependencies of alibi-detect anyway...)

[cfolan17]

For the first clarification, I have attempted to run the script both and without installing transformers in the requirements.txt. For other workflows, I have been able to install and use the transformers in the same base container.

For the second clarification, the issue only arises during the import of alibi-detect. Once the package is imported, the script is able to load the detector and make predictions off of it.

I am using the current container because it is a prebuilt container provided by AWS SageMaker which has allowed me to easily customize and adapt different scripts for deployment. I could potentially create a custom container and upload it to SageMaker but I was hoping to get this to work without that option.

[ascillitoe]

Hi @cfolan17, thanks for giving the additional info. Unfortunately, I've not been able to reproduce the same error in version 1.0-1 of the sklearn image. Would you be able to share your requirements.txt file please so I can investigate further? Also, just to confirm, are you keeping the Dockerfile's the same and only editing the top-level requirements.txt?