Install from source: ModuleNotFoundError: No module named 'spacy.pipeline.pipe'

[s3bk]

How to reproduce the problem

Follow the instructions in the Readme:

git clone https://github.com/explosion/spaCy
cd spaCy

python -m venv .env
source .env/bin/activate

# make sure you are using the latest pip
python -m pip install -U pip setuptools wheel

pip install .

Then running python:

Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sebk/src/spaCy/spacy/__init__.py", line 13, in <module>
    from . import pipeline  # noqa: F401
  File "/home/sebk/src/spaCy/spacy/pipeline/__init__.py", line 1, in <module>
    from .attributeruler import AttributeRuler
  File "/home/sebk/src/spaCy/spacy/pipeline/attributeruler.py", line 6, in <module>
    from .pipe import Pipe
ModuleNotFoundError: No module named 'spacy.pipeline.pipe'

Environment

• Operating System: Ubuntu 18
• Python Version Used: 3.6.9
• spaCy Version Used: HEAD (3.0.1) (91a3cab)
• Environment Information: -

[honnibal]

Ah there's a subtle trap here. If those are the exact directions we have in the readme we should change them. Usually I run python setup.py build_ext --inplace instead of pip install .

What's happening to you is that the library is getting compiled and installed into your environment, but then when you go to import spacy, priority is given to your local directory. The directory doesn't have the Cython modules compiled, and you therefore hit an error.

The pip install -e . works around this by compiling the artifacts in-place, and then symlinked to your environment. But then when you import, you'll be importing from your local path. I find this a bit unideal. Running python setup.py build_ext --inplace will just compile the modules, and then import spacy will rely on your local directory and PYTHONPATH.

[adrianeboyd]

It is hard to know what to recommend here. For a one-time non-editable install, I do think this is the best option for a clean install where you don't have manage build dependencies or anything, but you do run into this import problem.

[adrianeboyd]

I don't like recommending PYTHONPATH because it's going to cause problems across venvs, right?

[adrianeboyd]

Here's what I would recommend for editable installs with pip: https://spacy.io/usage#source-developers

pip install -r requirements.txt
pip install --no-build-isolation --editable .

[s3bk]

Ah there's a subtle trap here. If those are the exact directions we have in the readme we should change them. Usually I run python setup.py build_ext --inplace instead of pip install .

@honnibal Thank you for the quick solution!
I think your explanation is quite useful and could almost be used as-is in the Readme.
python setup.py build_ext --inplace worked fine.

[adrianeboyd]

The fact that there's no way to specify extras with python setup.py is the other reason for this choice.

[adrianeboyd]

Sorry for the rambling. I spent ages debating about this, and there's no pretty solution. I guess for the README we could use:

pip install -r requirements.txt
pip install --no-build-isolation .[extra]

The pip install command is then +/- --editable depending on how you'd like the install to be completed.

[s3bk]

One trap I fell into after this was that the downloader does not like non-packaged builds.
So building and installing into the user/system paths is probably better.

[adrianeboyd]

What error did you see?

[adrianeboyd]

Ah, you can probably just always substitute python -m spacy command in that case. We try to use that in the docs because it's more likely to work across the board.

[s3bk]

I got this message: https://github.com/explosion/spaCy/blob/master/spacy/cli/download.py#L43

[adrianeboyd]

That's just a warning, so it doesn't mean that the download will fail, but it can cause confusion because some models have extra dependencies that you'd have to install by hand otherwise (Chinese, Japanese, Russian, transformers).

(The problem is that the model also has a compatible version of spacy in its dependencies, and if you leave the dependencies enabled, it can then clobber your source install.)