'cupy.core.core.Indexer' has no attribute '__reduce_cython__'

[leimao]

How to reproduce the problem

To download the Spacy language model in Dockerfile, an error occurs:

Step 5/7 : RUN python -m spacy download en
 ---> Running in d872921ac8e2
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 174, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.5/runpy.py", line 133, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.5/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.5/dist-packages/spacy/__init__.py", line 10, in <module>
    from thinc.neural.util import prefer_gpu, require_gpu
  File "/usr/local/lib/python3.5/dist-packages/thinc/neural/__init__.py", line 4, in <module>
    from ._classes.model import Model  # noqa: F401
  File "/usr/local/lib/python3.5/dist-packages/thinc/neural/_classes/model.py", line 11, in <module>
    from ..train import Trainer
  File "/usr/local/lib/python3.5/dist-packages/thinc/neural/train.py", line 7, in <module>
    from .optimizers import Adam, linear_decay
  File "optimizers.pyx", line 13, in init thinc.neural.optimizers
  File "ops.pyx", line 40, in init thinc.neural.ops
  File "/usr/local/lib/python3.5/dist-packages/cupy/__init__.py", line 11, in <module>
    from cupy import core  # NOQA
  File "/usr/local/lib/python3.5/dist-packages/cupy/core/__init__.py", line 1, in <module>
    from cupy.core import core  # NOQA
  File "cupy/core/carray.pxi", line 50, in init cupy.core.core
AttributeError: type object 'cupy.core.core.Indexer' has no attribute '__reduce_cython__'

However, without downloading en in the Dockerfile, the image could be created successfully. Then I could download the en without problem.

Your Environment

• Platform: Linux-5.0.0-27-generic-x86_64-with-Ubuntu-16.04-xenial
• Python version: 3.5.2
• spaCy version: 2.1.8

[ines]

This sounds very similar to this problem, which was caused by the wrong version of cupy: https://stackoverflow.com/questions/53953623/attributeerror-type-object-cupy-core-core-broadcast-has-no-attribute-reduc

Could you run nvcc --version, double-check your cuda version and make sure you've installed the GPU dependencies for the correct version? See here for details: https://spacy.io/usage#gpu

[leimao]

It is weird. Because it was in the container. As I described, I could not install it via Dockerfile. But after the container was built, I could.

[ines]

I think the reason is that when you download the model while you're building the container, it will execute spaCy in the Dockerfile (and that seems to fail because of a cuda version mismatch). If you're not downloading a model, spaCy itself won't be executed during setup. I don't know how you've configured your Dockerfile, but it seems to use your GPU, so you might want to install spaCy via spacy[cuda100] (matching your cuda version).

[leimao]

Thanks @ines . I am attaching my Dockerfile.

FROM nvcr.io/nvidia/tensorflow:19.08-py3
# FROM python:3.6

LABEL maintainer="Lei Mao <dukeleimao@gmail.com>"

# RUN pip install -U spacy && python -m spacy download en
RUN pip install -U spacy[cuda100] && python -m spacy download en

# Working directory
WORKDIR /workspace

I have the following variables:
Workstation: Workstation 1 with TITAN V, Workstation 2 with V100
Base Image: nvcr.io/nvidia/tensorflow:19.08-py3 (CUDA 10.1), python:3.6 (no CUDA)
Spacy: spacy, spacy[cuda100]

Currently, the following configurations have been tested and worked:
Workstation 1 with TITAN V + nvcr.io/nvidia/tensorflow:19.08-py3+ (spacy[cuda100] + download)
Workstation 1 with TITAN V + python:3.6 + (spacy + download)
Workstation 2 with V100 + nvcr.io/nvidia/tensorflow:19.08-py3 + (spacy + download)
Workstation 1 with TITAN V + nvcr.io/nvidia/tensorflow:19.08-py3 -> build -> (spacy + download)
The following configurations have been tested and did not work:
Workstation 1 with TITAN V + nvcr.io/nvidia/tensorflow:19.08-py3 + (spacy + download)

I am also surprised that spacy[cuda100] would work in a CUDA 10.1 container. In addition, I still don't quite understand why I could install spacy and download after the container was built.

[leimao]

It turns out that if I changed the default Docker runtime to NVIDIA runtime, RUN pip install -U spacy && python -m spacy download en could be executed during Docker build successfully. Some of my machines were using NVIDIA runtime, some were not, which caused the discrepancy.
To change the default runtime:
https://docs.nvidia.com/dgx/nvidia-container-runtime-upgrade/index.html#using-nv-container-runtime

[leimao]

Personally I think you can try to remove the dependency to CUDA for spacy download to solve the problem without having to change the default runtime. Why would you ever need CUDA to download a file?

[ines]

Glad it's working now! Thanks for sharing the details and your Dockerfile.

And there's not really a "CUDA dependency". spaCy's machine learning library Thinc uses cupy if you're running it on GPU. So when you import spaCy, it will fail if cupy isn't installed correctly. Typically, the spaCy installation you'd be downloading the models with is the same installation you'd be running the models with later on... and if that's broken, it might as well show you the error as early as possible.

[leimao]

Glad it's working now! Thanks for sharing the details and your Dockerfile.

And there's not really a "CUDA dependency". spaCy's machine learning library Thinc uses cupy if you're running it on GPU. So when you import spaCy, it will fail if cupy isn't installed correctly. Typically, the spaCy installation you'd be downloading the models with is the same installation you'd be running the models with later on... and if that's broken, it might as well show you the error as early as possible.

Thanks @ines for some of the details. So the nvidia container nvcr.io/nvidia/tensorflow:19.08-py3 does come with cupy, what you just said about the breaking dependencies between the existing cupy and the spacy to be installed makes sense to some extent (spacy[cuda100] might just match the existing cupy in the container). But how do we explain after changing the runtime to nvidia-runtime in Docker, which provides access to GPU drivers, the same Dockerfile could be built without problem.

[jeromemassot]

Hi all,
I have the same problem with my configuration running Win10.
I have install the latest version of Rasa.
I have remove all previous cuda and install a fresh Cuda 10.0
I have install space[cuda100]
I have check my laptop configuration in order to use always the NVidia GPU
I have check cuda version (10.0)
I have check nvdia-smi
...
My pip freeze is :
cupy-cuda100==7.0.0b4
...
rasa==1.3.9
rasa-sdk==1.3.3
rasa-x==0.21.4
...
spacy==2.2.1
...
tensorboard==1.14.0
tensorflow==1.14.0
...
tensorflow-gpu==1.14.0

So I do no see what to do....

It is bad as I am doing the coding test to see if I have a chance to work at Rasa... :-(
And if I cannot install correctly on Win10.... I am a shame for applying for the job :-(

Thanks

Best regards

Jerome

[leimao]

Hi all,
I have the same problem with my configuration running Win10.
I have install the latest version of Rasa.
I have remove all previous cuda and install a fresh Cuda 10.0
I have install space[cuda100]
I have check my laptop configuration in order to use always the NVidia GPU
I have check cuda version (10.0)
I have check nvdia-smi
...
My pip freeze is :
cupy-cuda100==7.0.0b4
...
rasa==1.3.9
rasa-sdk==1.3.3
rasa-x==0.21.4
...
spacy==2.2.1
...
tensorboard==1.14.0
tensorflow==1.14.0
...
tensorflow-gpu==1.14.0

So I do no see what to do....

It is bad as I am doing the coding test to see if I have a chance to work at Rasa... :-(
And if I cannot install correctly on Win10.... I am a shame for applying for the job :-(

Thanks

Best regards

Jerome

Try the Docker container and the tricks I provided above.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.