`ImportError: torchtext C++ Extension is not found`

[msaroufim]

📚 Documentation

Hi I was interested in installing torchtext nightly so I can try out the latest and greatest datapipes @parmeet

TL;DR: It'd be really nice to either start having nightly builds for torchtext or streamline the installation process with some more scripts

I noticed torchtext isn't mentioned on pytorch.org

And if I run the below on Ubuntu

(text) ubuntu@ip-172-31-16-198:~/torchtext$ pip3 install --pre torch torchvision torchaudio torchtext --extra-index-url https://download.pytorch.org/whl/nightly/cpu^C
(text) ubuntu@ip-172-31-16-198:~/torchtext$ python
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchtext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/torchtext/torchtext/__init__.py", line 4, in <module>
    from torchtext import _extension  # noqa: F401
  File "/home/ubuntu/torchtext/torchtext/_extension.py", line 64, in <module>
    _init_extension()
  File "/home/ubuntu/torchtext/torchtext/_extension.py", line 56, in _init_extension
    raise ImportError("torchtext C++ Extension is not found.")
ImportError: torchtext C++ Extension is not found.

I couldn't find a script that uploads nightly binaries https://github.com/pytorch/text/tree/main/.github/workflows

So I figured I'd try installing things from source, I'm using Ubuntu on an AWS machine and after running python setup.py clean install it failed because Ninja was not found, I installed it with sudo apt install ninja-build (maybe a small install_dependencies.py script would be helpful similar to https://github.com/pytorch/serve/blob/master/ts_scripts/install_dependencies.py)

And this worked

python setup.py install
Using /home/ubuntu/anaconda3/envs/text/lib/python3.8/site-packages
Finished processing dependencies for torchtext==0.13.0a0+70fc104`

But gave me the same ImportError: torchtext C++ Extension is not found. when I tried to import torchtext in a python shell and from the error I'm not sure how to proceed

I did not build pytorch core from scratch, am I supposed to?

[mthrok]

Regarding the import error, are you in the directory where torchtext is cloned? (is ~/torchtext where git cloned torchtext?) If so the ImportError is because the source code is shadowing the torchtext installed in the anaconda env. Can you see what happens if you move out of the directory of ~/torchtext?

[msaroufim]

Wow that worked lol

(text) ubuntu@ip-172-31-16-198:~$ python
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchtext
>>> 

[parmeet]

I think for the reasons @mthrok mentioned, you could try python setup.py develop instead. If I am not mistaken, this would create symlink with the working directory. Hence you do not have to re-install as well when modifying python source files.