AttributeError: module 'numpy' has no attribute '_no_nep50_warning'

[IFenton]

The code in how-to-use-scivision.ipynb currently crashes, both in binder and on a local version at the load_pretrained_model function. Binder gives the following AttributeError (in the local version, the kernel just crashes).

Full error

2023-01-10 15:09:23.308397: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-10 15:09:23.854424: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-01-10 15:09:23.854467: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-01-10 15:09:25.093303: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-01-10 15:09:25.093587: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-01-10 15:09:25.093610: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [4], line 2
      1 # Note: when loading this model repo for the first time you'll need to add the allow_install=True flag
----> 2 resnet_model = load_pretrained_model(model_repo, model_selection='resnet18', allow_install=True)
      3 densenet_model = load_pretrained_model(model_repo, model_selection='densenet121')

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision/koala.py:57, in koala.<locals>.wrapped_fn(*args, **kwargs)
     55 # now call the function, but log any exceptions
     56 try:
---> 57     output = fn(*args, **kwargs)
     58 except Exception as e:
     59     logger.error(e)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision/io/reader.py:176, in load_pretrained_model(path, branch, allow_install, model_selection, load_multiple, *args, **kwargs)
    173     # try to install the package if necessary
    174     install_package(config, allow_install=allow_install, branch=branch)
--> 176     loaded_models.append(PretrainedModel(config))
    177 if load_multiple:
    178     return loaded_models

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision/io/wrapper.py:14, in PretrainedModel.__init__(self, config)
     12 def __init__(self, config: dict):
     13     self._config = config
---> 14     self._plumbing = AutoPlumber(config)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision/io/autoplumber.py:33, in AutoPlumber.__init__(self, config)
     30 def __init__(self, config: dict):
     31 
     32     # import the module and get the model object
---> 33     self._module = importlib.import_module(config["import"])
     34     model = getattr(self._module, config["model"])
     36     # we could instantiate the model using the args here
     37     # args = config["args"]

File /srv/conda/envs/notebook/lib/python3.9/importlib/__init__.py:127, in import_module(name, package)
    125             break
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1030, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1007, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:986, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:680, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:850, in exec_module(self, module)

File <frozen importlib._bootstrap>:228, in _call_with_frames_removed(f, *args, **kwds)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision_classifier/__init__.py:1
----> 1 from .model import vgg16
      2 from .model import vgg19
      3 from .model import resnet18

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scivision_classifier/model.py:4
      2 from classification_models.tfkeras import Classifiers
      3 from skimage.filters import gaussian
----> 4 from skimage.transform import resize
      5 from tensorflow.keras.applications.imagenet_utils import decode_predictions
      8 def tidy_predict(self, image: np.ndarray) -> str:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/skimage/transform/__init__.py:4
      1 from .hough_transform import (hough_line, hough_line_peaks,
      2                               probabilistic_hough_line, hough_circle,
      3                               hough_circle_peaks, hough_ellipse)
----> 4 from .radon_transform import (radon, iradon, iradon_sart,
      5                               order_angles_golden_ratio)
      6 from .finite_radon_transform import frt2, ifrt2
      7 from .integral import integral_image, integrate

File /srv/conda/envs/notebook/lib/python3.9/site-packages/skimage/transform/radon_transform.py:3
      1 import numpy as np
----> 3 from scipy.interpolate import interp1d
      4 from scipy.constants import golden_ratio
      5 from scipy.fft import fft, ifft, fftfreq, fftshift

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/interpolate/__init__.py:167
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    165 (should not be used in new code).
    166 """
--> 167 from ._interpolate import *
    168 from ._fitpack_py import *
    170 # New interface to fitpack library:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/interpolate/_interpolate.py:12
      9 from scipy.special import comb
     10 from scipy._lib._util import prod
---> 12 from . import _fitpack_py
     13 from . import dfitpack
     14 from . import _fitpack

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/interpolate/_fitpack_py.py:10
      8 from ._fitpack_impl import bisplrep, bisplev, dblint
      9 from . import _fitpack_impl as _impl
---> 10 from ._bsplines import BSpline
     13 def splprep(x, w=None, u=None, ub=None, ue=None, k=3, task=0, s=None, t=None,
     14             full_output=0, nest=None, per=0, quiet=1):
     15     """
     16     Find the B-spline representation of an N-D curve.
     17 
   (...)
    151 
    152     """

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/interpolate/_bsplines.py:8
      4 from numpy.core.multiarray import normalize_axis_index
      5 from scipy.linalg import (get_lapack_funcs, LinAlgError,
      6                           cholesky_banded, cho_solve_banded,
      7                           solve, solve_banded)
----> 8 from scipy.optimize import minimize_scalar
      9 from . import _bspl
     10 from . import _fitpack_impl

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/optimize/__init__.py:405
      1 """
      2 =====================================================
      3 Optimization and root finding (:mod:`scipy.optimize`)
   (...)
    401 
    402 """
    404 from ._optimize import *
--> 405 from ._minimize import *
    406 from ._root import *
    407 from ._root_scalar import *

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/optimize/_minimize.py:26
     24 from ._trustregion_krylov import _minimize_trust_krylov
     25 from ._trustregion_exact import _minimize_trustregion_exact
---> 26 from ._trustregion_constr import _minimize_trustregion_constr
     28 # constrained minimization
     29 from ._lbfgsb_py import _minimize_lbfgsb

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/optimize/_trustregion_constr/__init__.py:4
      1 """This module contains the equality constrained SQP solver."""
----> 4 from .minimize_trustregion_constr import _minimize_trustregion_constr
      6 __all__ = ['_minimize_trustregion_constr']

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py:5
      3 from scipy.sparse.linalg import LinearOperator
      4 from .._differentiable_functions import VectorFunction
----> 5 from .._constraints import (
      6     NonlinearConstraint, LinearConstraint, PreparedConstraint, strict_bounds)
      7 from .._hessian_update_strategy import BFGS
      8 from .._optimize import OptimizeResult

File /srv/conda/envs/notebook/lib/python3.9/site-packages/scipy/optimize/_constraints.py:8
      6 from ._optimize import OptimizeWarning
      7 from warnings import warn, catch_warnings, simplefilter
----> 8 from numpy.testing import suppress_warnings
      9 from scipy.sparse import issparse
     12 def _arr_to_scalar(x):
     13     # If x is a numpy array, return x.item().  This will
     14     # fail if the array has more than one element.

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/testing/__init__.py:11
      8 from unittest import TestCase
     10 from . import _private
---> 11 from ._private.utils import *
     12 from ._private.utils import (_assert_valid_refcount, _gen_alignment_data)
     13 from ._private import extbuild, decorators as dec

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/testing/_private/utils.py:480
    476         pprint.pprint(desired, msg)
    477         raise AssertionError(msg.getvalue())
--> 480 @np._no_nep50_warning()
    481 def assert_almost_equal(actual,desired,decimal=7,err_msg='',verbose=True):
    482     """
    483     Raises an AssertionError if two items are not equal up to desired
    484     precision.
   (...)
    548 
    549     """
    550     __tracebackhide__ = True  # Hide traceback for py.test

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/__init__.py:311, in __getattr__(attr)
    300 """
    301 Quick sanity checks for common bugs caused by environment.
    302 There are some cases e.g. with wrong BLAS ABI that cause wrong
   (...)
    308 
    309 """
    310 try:
--> 311     x = ones(2, dtype=float32)
    312     if not abs(x.dot(x) - float32(2.0)) < 1e-5:
    313         raise AssertionError()

AttributeError: module 'numpy' has no attribute '_no_nep50_warning'

[IFenton]

object-detection.ipynb failed at the same function load_pretrained_model with a crashed kernel

[IFenton]

segmentation.ipynb failed at the same function load_pretrained_model

Full error:

Failed to build detectron2 pycocotools
  error: subprocess-exited-with-error
  
  × Building wheel for pycocotools (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-39
      creating build/lib.macosx-10.9-x86_64-cpython-39/pycocotools
      copying pycocotools/coco.py -> build/lib.macosx-10.9-x86_64-cpython-39/pycocotools
      copying pycocotools/mask.py -> build/lib.macosx-10.9-x86_64-cpython-39/pycocotools
      copying pycocotools/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-39/pycocotools
      copying pycocotools/cocoeval.py -> build/lib.macosx-10.9-x86_64-cpython-39/pycocotools
      running build_ext
      cythoning pycocotools/_mask.pyx to pycocotools/_mask.c
      /private/var/folders/xr/xjkpw26d2gqgfzz06tkmfr5c0000gr/T/pip-build-env-_3o8dt2x/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /private/var/folders/xr/xjkpw26d2gqgfzz06tkmfr5c0000gr/T/pip-install-i6eh5fy2/pycocotools_ae1495eeb047421f9fc0328b64d379a6/pycocotools/_mask.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      building 'pycocotools._mask' extension
      creating build/temp.macosx-10.9-x86_64-cpython-39
      creating build/temp.macosx-10.9-x86_64-cpython-39/common
      creating build/temp.macosx-10.9-x86_64-cpython-39/pycocotools
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/ifenton/opt/anaconda3/envs/SciVisionBasic/include -fPIC -O2 -isystem /Users/ifenton/opt/anaconda3/envs/SciVisionBasic/include -I/private/var/folders/xr/xjkpw26d2gqgfzz06tkmfr5c0000gr/T/pip-build-env-_3o8dt2x/overlay/lib/python3.9/site-packages/numpy/core/include -I./common -I/Users/ifenton/opt/anaconda3/envs/SciVisionBasic/include/python3.9 -c ./common/maskApi.c -o build/temp.macosx-10.9-x86_64-cpython-39/./common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
      xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycocotools
ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[3], line 2
      1 # Note: when loading this model repo for the first time you'll need to add the allow_install=True flag
----> 2 facebook_model = load_pretrained_model(model_repo, model_selection='facebook_detr_resnet_50_panoptic', allow_install=True)
      3 nvidia_model = load_pretrained_model(model_repo, model_selection='nvidia_segformer_b5_finetuned_ade_640_640')
      4 microsoft_model = load_pretrained_model(model_repo, model_selection='microsoft_beit_large_finetuned_ade_640_640')

File ~/opt/anaconda3/envs/SciVisionBasic/lib/python3.9/site-packages/scivision/koala.py:57, in koala.<locals>.wrapped_fn(*args, **kwargs)
     55 # now call the function, but log any exceptions
     56 try:
---> 57     output = fn(*args, **kwargs)
     58 except Exception as e:
     59     logger.error(e)

File ~/opt/anaconda3/envs/SciVisionBasic/lib/python3.9/site-packages/scivision/io/reader.py:174, in load_pretrained_model(path, branch, allow_install, model_selection, load_multiple, *args, **kwargs)
    171     assert "X" in config["prediction_fn"]["args"].keys()
    173     # try to install the package if necessary
--> 174     install_package(config, allow_install=allow_install, branch=branch)
    176     loaded_models.append(PretrainedModel(config))
    177 if load_multiple:

File ~/opt/anaconda3/envs/SciVisionBasic/lib/python3.9/site-packages/scivision/io/installer.py:51, in install_package(config, allow_install, branch)
     46 exists = _package_exists(config)
     48 if allow_install == "force" or (allow_install and not exists):
     49     # if a package is not already installed, there is little harm
     50     # to passing the extra arguments, so these cases are combined
---> 51     _install(package, pip_install_args=["--force-reinstall", "--no-cache-dir"])
     52 elif not exists:
     53     raise Exception(
     54         "Package does not exist. Try installing it with: \n"
     55         f"`!pip install -e {package}`"
     56     )

File ~/opt/anaconda3/envs/SciVisionBasic/lib/python3.9/site-packages/scivision/io/installer.py:34, in _install(package, pip_install_args)
     31 if pip_install_args is None:
     32     pip_install_args = []
---> 34 subprocess.check_call(
     35     [sys.executable, "-m", "pip", "install", *pip_install_args, package]
     36 )

File ~/opt/anaconda3/envs/SciVisionBasic/lib/python3.9/subprocess.py:373, in check_call(*popenargs, **kwargs)
    371     if cmd is None:
    372         cmd = popenargs[0]
--> 373     raise CalledProcessError(retcode, cmd)
    374 return 0

CalledProcessError: Command '['/Users/ifenton/opt/anaconda3/envs/SciVisionBasic/bin/python', '-m', 'pip', 'install', '--force-reinstall', '--no-cache-dir', 'git+https://github.com/alan-turing-institute/scivision_huggingface_segmentation@main#egg=huggingface_segmentation']' returned non-zero exit status 1.

[IFenton]

This error relates to the models installing the latest versions of several dependencies that no longer function as expected. I can build an environment.yml that works, however at the moment it is necessary to install the models with no dependencies, as the requirement files of those models are updating to the latest versions. Additionally the object-detection notebook only seems to run if the development version of scivision is used. For details see below.

All three notebooks will run, but only with a certain amount of manual intervention:

• how-to-use-scivision.ipynb. This has two models
  • scivision-classifier. This needs to be pip installed in the environment at the moment, otherwise force install will update the packages.
  • scivision-huggingface. This needs to be installed without dependencies as the requirements file has lines such as transformers[torch] which automatically update packages to the latest version.
• object-detection.ipynb:
  • The model scivision-huggingface-objectdetection has to be installed with no dependencies, similar to scivision-huggingface.
  • At the moment I can only seem to get this to run if scivision is installed in development mode, otherwise the fonts file is not downloaded.
• segmentation.ipynb
  • The scivision-huggingface-segmentation model needs to be installed with no dependencies, as above.

[IFenton]

This has now been resolved with #7