pip install failed on Linux

[zhangguanheng66]

We tried to install sentencepiece on Mac and Linux. It works fine for Mac but fail on Linux with the following information:

11:08:12 AM:   Building wheel for sentencepiece (setup.py): started
11:08:15 AM:   Building wheel for sentencepiece (setup.py): finished with status 'error'
11:08:15 AM:   Complete output from command /opt/buildhome/python3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-dr7no4n2/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-x_fydfed --python-tag cp36:
11:08:15 AM:   running bdist_wheel
11:08:15 AM:   running build
11:08:15 AM:   running build_py
11:08:15 AM:   creating build
11:08:15 AM:   creating build/lib.linux-x86_64-3.6
11:08:15 AM:   creating build/lib.linux-x86_64-3.6/sentencepiece
11:08:15 AM:   copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:15 AM:   copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:15 AM:   copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:15 AM:   running build_ext
11:08:15 AM:   Package sentencepiece was not found in the pkg-config search path.
11:08:15 AM:   Perhaps you should add the directory containing `sentencepiece.pc'
11:08:15 AM:   to the PKG_CONFIG_PATH environment variable
11:08:15 AM:   No package 'sentencepiece' found
11:08:15 AM:   Cloning into 'sentencepiece'...
11:08:15 AM:   Note: checking out '8336bbd0c1cfba02a879afe625bf1ddaf7cd93c5'.
11:08:15 AM:   You are in 'detached HEAD' state. You can look around, make experimental
11:08:15 AM:   changes and commit them, and you can discard any commits you make in this
11:08:15 AM:   state without impacting any branches by performing another checkout.
11:08:15 AM:   If you want to create a new branch to retain commits you create, you may
11:08:15 AM:   do so (now or later) by using -b with the checkout command again. Example:
11:08:15 AM:     git checkout -b <new-branch-name>
11:08:15 AM:   CMake Error at CMakeLists.txt:15 (cmake_minimum_required):
11:08:15 AM:     CMake 3.1 or higher is required.  You are running version 2.8.12.2
11:08:15 AM:   -- Configuring incomplete, errors occurred!
11:08:15 AM:   make: *** No targets specified and no makefile found.  Stop.
11:08:15 AM:   make: *** No rule to make target `install'.  Stop.
11:08:15 AM:   Package sentencepiece was not found in the pkg-config search path.
11:08:15 AM:   Perhaps you should add the directory containing `sentencepiece.pc'
11:08:15 AM:   to the PKG_CONFIG_PATH environment variable
11:08:15 AM:   No package 'sentencepiece' found
11:08:15 AM:   Failed to find sentencepiece pkg-config
11:08:15 AM:   ----------------------------------------
11:08:15 AM:   Failed building wheel for sentencepiece
11:08:15 AM:   Running setup.py clean for sentencepiece
11:08:16 AM:   Building wheel for PyYAML (setup.py): started
11:08:22 AM:   Building wheel for PyYAML (setup.py): finished with status 'done'
11:08:22 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
11:08:22 AM:   Building wheel for gpustat (setup.py): started
11:08:23 AM:   Building wheel for gpustat (setup.py): finished with status 'done'
11:08:23 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/48/b4/d5/fb5b7f1d040f2ff20687e3bad6867d63155dbde5a7c10f4293
11:08:23 AM:   Building wheel for olefile (setup.py): started
11:08:23 AM:   Building wheel for olefile (setup.py): finished with status 'done'
11:08:23 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/4b/f4/11/bc4166107c27f07fd7bba707ffcb439619197638a1ac986df3
11:08:23 AM:   Building wheel for nvidia-ml-py3 (setup.py): started
11:08:23 AM:   Building wheel for nvidia-ml-py3 (setup.py): finished with status 'done'
11:08:23 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/e4/1d/06/640c93f5270d67d0247f30be91f232700d19023f9e66d735c7
11:08:23 AM:   Building wheel for psutil (setup.py): started
11:08:25 AM:   Building wheel for psutil (setup.py): finished with status 'done'
11:08:25 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/42/32/da/8b12fd6b138c733efd03cfde6c6c8191a32842f9e82aa45fbf
11:08:25 AM:   Building wheel for pyrsistent (setup.py): started
11:08:26 AM:   Building wheel for pyrsistent (setup.py): finished with status 'done'
11:08:26 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/f0/b8/de/b593ad311be4eb458499d100db081e453576032272398b7ddc
11:08:26 AM:   Building wheel for idna-ssl (setup.py): started
11:08:26 AM:   Building wheel for idna-ssl (setup.py): finished with status 'done'
11:08:26 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/d3/00/b3/32d613e19e08a739751dd6bf998cfed277728f8b2127ad4eb7
11:08:26 AM:   Building wheel for contextvars (setup.py): started
11:08:27 AM:   Building wheel for contextvars (setup.py): finished with status 'done'
11:08:27 AM:   Stored in directory: /opt/buildhome/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de
11:08:27 AM: Successfully built sphinx-gallery bs4 wget future PyYAML gpustat olefile nvidia-ml-py3 psutil pyrsistent idna-ssl contextvars
11:08:27 AM: Failed to build sentencepiece
11:08:27 AM: matplotlib 3.3.2 has requirement pillow>=6.2.0, but you'll have pillow 4.1.1 which is incompatible.
11:08:27 AM: botocore 1.12.25 has requirement urllib3<1.24,>=1.20, but you'll have urllib3 1.25.11 which is incompatible.
11:08:27 AM: scikit-image 0.17.2 has requirement pillow!=7.1.0,!=7.1.1,>=4.3.0, but you'll have pillow 4.1.1 which is incompatible.
11:08:27 AM: Installing collected packages: snowballstemmer, pyparsing, six, packaging, sphinxcontrib-serializinghtml, sphinxcontrib-websupport, Pygments, docutils, MarkupSafe, Jinja2, pytz, babel, idna, urllib3, chardet, certifi, requests, alabaster, imagesize, sphinx, kiwisolver, cycler, olefile, pillow, numpy, python-dateutil, matplotlib, sphinx-gallery, tqdm, future, torch, torchvision, sentencepiece, torchtext, torchaudio, PyHamcrest, soupsieve, beautifulsoup4, bs4, jmespath, botocore, PyYAML, pyasn1, rsa, s3transfer, colorama, awscli, click, itsdangerous, Werkzeug, flask, plac, wasabi, cymem, murmurhash, preshed, srsly, blis, zipp, importlib-metadata, catalogue, thinc, spacy, idna-ssl, typing-extensions, multidict, async-timeout, yarl, attrs, aiohttp, aiohttp-cors, py-spy, redis, nvidia-ml-py3, psutil, blessings, gpustat, protobuf, googleapis-common-protos, cachetools, pyasn1-modules, google-auth, google-api-core, immutables, contextvars, opencensus-context, opencensus, grpcio, hiredis, aioredis, filelock, prometheus-client, colorful, pyrsistent, jsonschema, msgpack, google, pandas, dataclasses, tensorboardX, tabulate, ray, pytorch-sphinx-theme, decorator, ipython-genutils, traitlets, pickleshare, wcwidth, prompt-toolkit, parso, jedi, backcall, ptyprocess, pexpect, ipython, PyWavelets, scipy, tifffile, networkx, imageio, scikit-image, wget
11:08:38 AM:   Running setup.py install for sentencepiece: started
11:08:38 AM:     Running setup.py install for sentencepiece: finished with status 'error'
11:08:38 AM:     Complete output from command /opt/buildhome/python3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-dr7no4n2/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-2d01vowq/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.6/include/site/python3.6/sentencepiece:
11:08:38 AM:     running install
11:08:38 AM:     running build
11:08:38 AM:     running build_py
11:08:38 AM:     creating build
11:08:38 AM:     creating build/lib.linux-x86_64-3.6
11:08:38 AM:     creating build/lib.linux-x86_64-3.6/sentencepiece
11:08:38 AM:     copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:38 AM:     copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:38 AM:     copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
11:08:38 AM:     running build_ext
11:08:38 AM:     Package sentencepiece was not found in the pkg-config search path.
11:08:38 AM:     Perhaps you should add the directory containing `sentencepiece.pc'
11:08:38 AM:     to the PKG_CONFIG_PATH environment variable
11:08:38 AM:     No package 'sentencepiece' found
11:08:38 AM:     mkdir: cannot create directory ‘bundled’: File exists
11:08:38 AM:     fatal: destination path 'sentencepiece' already exists and is not an empty directory.
11:08:38 AM:     fatal: destination path 'sentencepiece' already exists and is not an empty directory.
11:08:38 AM:     mkdir: cannot create directory ‘build’: File exists
11:08:38 AM:     CMake Error at CMakeLists.txt:15 (cmake_minimum_required):
11:08:38 AM:       CMake 3.1 or higher is required.  You are running version 2.8.12.2
11:08:38 AM:     -- Configuring incomplete, errors occurred!
11:08:38 AM:     make: *** No targets specified and no makefile found.  Stop.
11:08:38 AM:     make: *** No rule to make target `install'.  Stop.
11:08:38 AM:     Package sentencepiece was not found in the pkg-config search path.
11:08:38 AM:     Perhaps you should add the directory containing `sentencepiece.pc'
11:08:38 AM:     to the PKG_CONFIG_PATH environment variable
11:08:38 AM:     No package 'sentencepiece' found
11:08:38 AM:     Failed to find sentencepiece pkg-config
11:08:38 AM:     ----------------------------------------
11:08:40 AM: Command "/opt/buildhome/python3.6/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-dr7no4n2/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-2d01vowq/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.6/include/site/python3.6/sentencepiece" failed with error code 1 in /tmp/pip-install-dr7no4n2/sentencepiece/
11:08:40 AM: You are using pip version 19.0.3, however version 20.2.4 is available.
11:08:40 AM: You should consider upgrading via the 'pip install --upgrade pip' command.
11:08:40 AM: Error installing pip dependencies
11:08:40 AM: Error running command: Build script returned non-zero exit code: 1
11:08:40 AM: Failing build: Failed to build site
11:08:40 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
11:08:41 AM: Finished processing build request in 2m37.714434404s

Any help is appreciated.

[nickmarton]

Experiencing the exact same issue on Databricks when trying to install spacy-transformers

[taku910]

Could you upgrade your pip command? Sentencepiece uses manylinux2014 which requires pip >= 19.3

[M-Anwar]

Yes this seems to be an issue with the pip version. Upgrading the pip version resolved my issues with installing this library and the transformers library. For Databricks users, you may need to upgrade to the ML 7.0+ runtime.

[nickmarton]

For what it's worth, it seems that upgrading pip on Databricks in a runtime <7.0 works as well, thanks guys

[taku910]

let me close this bug, as this is an issue of pip version.

[JuvenalDuarte]

For some reason the latest version of the package doesn't seems do be working on my Ubuntu 20 LTS. It crashes during the installation with the same error reported above.
I've solved this problem by explicitly setting the version as below:

sudo pip3 install sentencepiece==0.1.92

The module is a dependency to sentence_transformers lib, which worked fine with the older version.

[manojthangaraj92]

any idea for the sagemaker users? the sentencepeice build is failing for me too

[taylorkchan]

any idea for the sagemaker users? the sentencepeice build is failing for me too

Same happening to me on sagemaker, did you have resolution?

[manojthangaraj92]

Hi Taylor,
Yes, I had it resolved. The problem with me is when I instantiated sagemaker, i had entered the framework_version as 0.8.0 which is way basic. I changed that with 1.8.0 which is the latest, py_version as py36. Once I had them, it was working fine.

[taylorkchan]

Hi Taylor,
Yes, I had it resolved. The problem with me is when I instantiated sagemaker, i had entered the framework_version as 0.8.0 which is way basic. I changed that with 1.8.0 which is the latest, py_version as py36. Once I had them, it was working fine.

Thanks, i just resolved by changing framework as well.

[walesdata]

Hi. I had this same error message, but it turns out I didn't have cmake installed. sudo apt install cmake