Pip install sentencepiece failure

[saareliad]

Hi,
pip install sentencepiece fails,
This is the log I get:

pip install sentencepiece
7.4.0
Collecting sentencepiece
Using cached https://files.pythonhosted.org/packages/fd/45/6d0eb609d5cd81df094aab71a867b2ab6b315ffd592e78fb94a625c4d6aa/sentencepiece-0.1.3.tar.gz
ERROR: Complete output from command python setup.py egg_info:
ERROR: /bin/sh: 1: pkg-config: not found
Failed to find sentencepiece pkgconfig
----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-463tj_x8/sentencepiece/

[aatkinson]

Same here, only happens on Python 3.7.4 for me, works on Python 3.6. Observed on Ubuntu 16.04.4

Collecting sentencepiece
  Downloading https://files.pythonhosted.org/packages/fd/45/6d0eb609d5cd81df094aab71a867b2ab6b315ffd592e78fb94a625c4d6aa/sentencepiece-0.1.3.tar.gz (498kB)
     |████████████████████████████████| 501kB 41.1MB/s
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Package sentencepiece was not found in the pkg-config search path.
    Perhaps you should add the directory containing `sentencepiece.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'sentencepiece' found
    Failed to find sentencepiece pkgconfig
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-cqmezxh3/sentencepiece/

[aatkinson]

Seems like a patch-level issue with Python 3.7.4. Installation succeeds with Python 3.7.3.
Possibly related: tensorflow/tensorflow#31523

[taku910]

Seems the valid binary whl package is not downloaded and the fallback src package is fetched instead.

What happens if you install sentencepiece-0.1.83-cp37-cp37m-manylinux1_x86_64.whl directly?
You can find the package at https://pypi.org/project/sentencepiece/#files

[aatkinson]

Installing the wheel directly works for me on Python 3.7.4, thanks.
Any idea why it's falling back to the source package?

[taku910]

Not sure. Probably some network or download trouble. Let me close this bug now. If this issue still persist, please feel free to reopen.

[tgsmith61591]

This solution doesn't work on 64-bit Ubuntu (18.04).

$ wget https://files.pythonhosted.org/packages/f2/e2/813dff3d72df2f49554204e7e5f73a3dc0f0eb1e3958a4cad3ef3fb278b7/sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl .
$ pip install sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl
ERROR: sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl is not a supported wheel on this platform.

[xuchong]

I find a solution works for me.
Just:
pip install --upgrade pip
Env:
python: 3.7.2
Release: 8.11
Codename: jessie

[savethebeesandseeds]

Here was how I solved (partially).
So I'm on Raspberry Pi 4 (armv7l GNU/Linux).

Installing Transformers, unable to install sentencepiece due to:
"Failed to find sentencepiece pkgconfig"

I went deep into the cause,
-------- First install from source:
$ wget https://github.com/google/sentencepiece
$ cd sentencepiece/python/
-------- Then try to install package (so folders are created):
$ sudo python3 setup.py install
-------- This will rise the error for me
ERROR: Package sentencepiece was not found in the pkg-config search path.
Perhaps you should add the directory containing `sentencepiece.pc'
to the PKG_CONFIG_PATH environment variable
No package 'sentencepiece' found
Failed to find sentencepiece pkgconfig"
-------- excecute in terminal the individual set of commands
$ cmake -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ .. -DSPM_ENABLE_SHARED=OFF -DCMAKE_INSTALL_PREFIX=../..
$ make -stdlib=libstdc++ -j $(nproc)
$ make install
--------- once sentencepiece was installed
$ python3
import sentencepiece
------- witch then rise a different error:
Traceback (most recent call last):
File "", line 1, in
File "/home/work/non.conflict/venv/lib/python3.7/site-packages/sentencepiece-0.1.94-py3.7-linux-armv7l.egg/sentencepiece/init.py", line 13, in
from . import _sentencepiece
ImportError: /home/work/non.conflict/venv/lib/python3.7/site-packages/sentencepiece-0.1.94-py3.7-linux-armv7l.egg/sentencepiece/_sentencepiece.cpython-37m-arm-linux-gnueabihf.so: undefined symbol: __atomic_fetch_add_8
--------- gives this error, but at now i'm able to import transformers
import transformers

[DavidChengweiLiu]

此解决方案在64位Ubuntu（18.04）上不起作用。

$ wget https://files.pythonhosted.org/packages/f2/e2/813dff3d72df2f49554204e7e5f73a3dc0f0eb1e3958a4cad3ef3fb278b7/sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl 。
$ pip安装句子-0.1.91-cp37-cp37m-manylinux1_x86_64.whl
错误：此平台不支持句子piece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl。

you can try upgrade pip.

[Akshaysharma29]

The below link worked for me
PaddlePaddle/PaddleHub#813 (comment)
Thanks

[kakaxisisan]

I find a solution works for me.
Just:
pip install --upgrade pip
Env:
python: 3.7.2
Release: 8.11
Codename: jessie

THX very much！works for me！！！

[gokulmanohar]

I find a solution works for me.
Just:
pip install --upgrade pip
Env:
python: 3.7.2
Release: 8.11
Codename: jessie

Thank you so much. Really helped me. I've been trying to instal cdqa for so long.

[goodwiins]

pip3 install --upgrade pip it works for me

[GingerNg]

I come with this problem !

hardware: Apple Macbook M1
Python 3.9.4 (default, Apr 4 2021, 17:42:23)
git clone https://github.com/google/sentencepiece.git
cd python
python3 setup.py install

logs as blew:
/opt/homebrew/lib/python3.9/site-packages/setuptools/dist.py:642: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running install
running bdist_egg
running egg_info
writing src/sentencepiece.egg-info/PKG-INFO
writing dependency_links to src/sentencepiece.egg-info/dependency_links.txt
writing top-level names to src/sentencepiece.egg-info/top_level.txt
reading manifest file 'src/sentencepiece.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/sentencepiece.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11-arm64/egg
running install_lib
running build_py
running build_ext
Package sentencepiece was not found in the pkg-config search path.
Perhaps you should add the directory containing sentencepiece.pc' to the PKG_CONFIG_PATH environment variable No package 'sentencepiece' found mkdir: bundled: File exists fatal: destination path 'sentencepiece' already exists and is not an empty directory. fatal: destination path 'sentencepiece' already exists and is not an empty directory. mkdir: build: File exists ./build_bundled.sh: line 15: cmake: command not found ./build_bundled.sh: line 16: nproc: command not found make: *** No targets specified and no makefile found. Stop. make: *** No rule to make target install'. Stop.
Package sentencepiece was not found in the pkg-config search path.
Perhaps you should add the directory containing `sentencepiece.pc'
to the PKG_CONFIG_PATH environment variable
No package 'sentencepiece' found
Failed to find sentencepiece pkg-config

[knsamsdesk]

Co-sign User:GingerNg above!

[zhijinghe]

pip3 install --upgrade pip it works for me

it also works for me, thanks!

[mahyard]

I don't know which package helped me out. but after installing these packages my problem solved:

sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

It comes from the Build and install SentencePiece command line tools from C++ source section of the README.md

[ni9elf]

For Apple M1 / ARM:
Run these on default terminal (not sure if all packages are required):

brew install cmake
brew install gperftools
brew install pkg-config

Then run this on rosetta emulated terminal (maybe the default terminal will work too):
pip3 install sentencepiece

[millnerryan]

Just brew install cmake fixed the issue for me, thanks @ni9elf. Ran on default terminal window too

[gelodefaultbrain]

The pip install worked for a while but I got this

  Building wheel for sentencepiece (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [42 lines of output]
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Cloning into 'sentencepiece'...
      Note: switching to '0e6dfbf86e2fa6d86a3d9a8a08a628da71c073e0'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:
      
        git switch -c <new-branch-name>
      
      Or undo this operation with:
      
        git switch -
      
      Turn off this advice by setting config variable advice.detachedHead to false
      
      ./build_bundled.sh: 15: cmake: not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target 'install'.  Stop.
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkg-config
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
Installing collected packages: tabulate, sqlitedict, sentencepiece, py4j, pptree, overrides, mpld3, janome, torch, segtok, pillow, networkx, more-itertools, lxml, langdetect, kiwisolver, joblib, importlib-metadata, future, ftfy, fonttools, deprecated, cycler, conllu, cloudpickle, wikipedia-api, scikit-learn, matplotlib, konoha, hyperopt, gensim, gdown, bpemb, flair
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      mkdir: cannot create directory ‘bundled’: File exists
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      mkdir: cannot create directory ‘build’: File exists
      ./build_bundled.sh: 15: cmake: not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target 'install'.  Stop.
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkg-config
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

[tamera-lanham]

One more data point for M1 users: I used the default terminal to run

brew install cmake
brew install pkg-config

And that did it for me.

[gelodefaultbrain]

Just want to add up probably might help. I switched to python version 3.9 and worked! I was actually at 3.10 when I did this and got those error.

[jamescalam]

One more data point for M1 users: I used the default terminal to run

brew install cmake
brew install pkg-config

And that did it for me.

Also on M1 - this worked for me

[parambhatia2004]

Make sure that setuptools is the upgraded version

[seekingtau]

Still broken on Python 3.10.4 - any clue what to do? Intel Mac; MacOS 11.4.

[eastonsuo]

conda install sentencepiece works for me

[sinisarudan]

For Apple M1 / ARM: Run these on default terminal (not sure if all packages are required):

brew install cmake
brew install gperftools
brew install pkg-config

Then run this on rosetta emulated terminal (maybe the default terminal will work too): pip3 install sentencepiece

It worked solving the M2 issues too! Thanks, man!

[marcelfoerster]

Using conda (Linux) creating an environment with flair>=0.11 resulted in sentencepiece installation error (version 0.1.95), setting python=3.9 (instead of >=) solved this problem and all conflicts.

[baqwas]

Installing sentencepiece manually did not work for me under:
Python 3.9.3
pip 20.3.4
RPi4 (Bullseye) aarch64 kernel 5.15.56-v8+ #1575 SMP PREEMPT

Did install sentencepiece successfully via pip after the first error but the error persisted. Would welcome workaround suggestions. I'm unqualified to do this on my own, naturally. Thanks.

Kind regards.

P.S.

python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [5 lines of output]
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkgconfig
      [end of output]

[mprinc]

This is how I've fixed the problem on this particular server: Python 3.11.0, Ubuntu 20.04 (focal):

sudo apt-get install pkg-config
sudo apt-get install cmake

So basically I provided all necessary building tools to build sentencepiece from scratch, referring to @taku910 clarification: Seems the valid binary whl package is not downloaded and the fallback src package is fetched instead.

Personally, I didn't want to depend on a boundary case for a specific scenario, but to keep generic installation

[baqwas]

@mprinc,

A very big THANK YOU!

Kind regards.

[findAkash]

#After installation cmake and pkg-config , I got this error

Package sentencepiece was not found in the pkg-config search path.
Perhaps you should add the directory containing `sentencepiece.pc'
to the PKG_CONFIG_PATH environment variable
No package 'sentencepiece' found
Failed to find sentencepiece pkgconfig

[boydxh]

My goal was to install FlairNLP, which relies on sentencepiece to run. After poking around at this problem for a while, this is what worked for me. I have a MacOS, I am using zsh, and I have Python 3.10.5. After activating my virtual environment, I ran the following command:

pip3 install --only-binary sentencepiece flair

I think this worked because the installation kept getting stuck on the step where it had to build a wheel for the sentencepiece dependency:

Building wheels for collected packages: sentencepiece
Building wheel for sentencepiece (setup.py) ... error
error: subprocess-exited-with-error

So I read this article about Python wheels and figured I'd force pip to use wheels instead of source distributions for sentencepiece. Hope this helps!

[shaun-scale]

None of this worked until I found this comment: #608 (comment)

brew install sentencepiece

seems to have finally done it...

So

brew install cmake
brew install pkg-config
brew install sentencepiece
pip install sentencepiece

and then finally for Huggingface pip3 install 'transformers[tf-cpu]' to work, I had to also install Rust next...

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
pip3 install 'transformers[tf-cpu]'

Finally......!

[roberto-butti]

I confirm that installing cmake on Macos Ventura

brew install cmake

it allows me to install correctly sentencepiece python module

pip3 install  sentencepiece

Thank you!

[nishantkumar1292]

I solved this first by running the below command

brew install sentencepiece

and then running pip install sentencepiece again.

[beingPurple]

For anyone still having this issue on Windows, this fixed things for me!

[MoElaSec]

For anyone on Mac M1 this indeed fixed the issue:

I first tried:

pip install sentencepiece --use-pep517

didn't work so then I did:

brew install cmake protobuf pkg-config
pip install sentencepiece

and indeed it worked 🎉🎉🎉

[vishvanth]

For Apple M1 / ARM: Run these on default terminal (not sure if all packages are required):

brew install cmake
brew install gperftools
brew install pkg-config

Then run this on rosetta emulated terminal (maybe the default terminal will work too): pip3 install sentencepiece

Thanks Buddy this worked for Macbook Air M1

[egamaral]

None of this worked until I found this comment: #608 (comment)

brew install sentencepiece

seems to have finally done it...

So

brew install cmake
brew install pkg-config
brew install sentencepiece
pip install sentencepiece

and then finally for Huggingface pip3 install 'transformers[tf-cpu]' to work, I had to also install Rust next...

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
pip3 install 'transformers[tf-cpu]'

Finally......!

I have a M2 Macbook Pro and works after apply this thread!

[Eugene-Mokrushin]

I am Windows with Intel, but this also solved the issue for me:
pip install cmake

[jason-liang]

Just brew install cmake fixed the issue for me too

[gaodada123456]

I missed the same issue,and i demote my python vision <=3.8,the issue canbe settled