Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Installing spaCy with OpenMP support on OS X #267
I have successfully built and installed spaCy 0.100.7 with OpenMP on OS X for brew installed python. For the impatient among us, here are instructions on how to do it yourself.
Choose your adventure:
Note that these instructions are for brew installed python.
WARNING: Compiling with GCC as of spaCy 0.100.5 may result in a segfault (#266).
Thanks for the pointers and making it work.
I am not a big fan of these silent up/downgrades depending on what dependencies you had installed. PIL comes to my mind as a particular bad example of this style. I would rather prefer passing some args/envs to setup.py and fail if dependencies are not available or ship a separate package. We'll definitely look into making this more accessible.
I can confirm clang-omp to work. We had another thread some time ago about compilation problems with gcc on OSX (#237). Its unfortunate, but there is very little support on detecting compilers with setuptools and even if you could do that reliably you might still be out of luck in case the user's Python got built with non-gcc flags (AFAIK they are not visible from within setup.py). What I want to say is: supporting gcc on osx is really tough while not being that much appreciated in case it does work.
Great news on clang-omp. I've been experimenting with many native extensions for Python, R, and NodeJS. I've had so many issues with native extensions and compilers that I now just install whichever compiler works best... Once Apple updates XCode with LLVM 3.7, there will be much less need for supporting GCC on OS X.
Practically speaking, the clang-omp solution works for me and I'm okay leaving it at that for now. I'm sure you guys have plenty of other interesting improvements to make to spaCy!
referenced this issue
Mar 10, 2016
I see you are using the Anaconda Python. I wrote the instructions for the homebrew Python. What is the compiler used to build Anaconda Python?
Just running the interpreter on my machine printed this message:
If you are using GGC-built Python, try using GCC to build spacy instead:
brew install gcc --without-multilib export CC=gcc-5 export CXX=g++-5
I've been told that the
If you are successful, please let us know!
Update: tried with GCC and successfully got the segfault :)
So then I tried with homebrew Python. Please note that I was getting an issue trying to clone the repo after running the export statements, so I revised the order a little:
And then....when downloading the English pack
Perhaps at this point it makes more sense for me to spin up a Linux box on AWS and bypass the issue.
I actually downloaded the English pack using pip-installed spacy and symlinked the directory. I'm on a slow connection and neglected to test that command. Symlinking the data dir might work.
I've used the spot instances for cheap, but expect your instance to be killed during peak hours.
As an alternative to
Hey all, looks like
For final reference:
Thanks so much to each of you.
added a commit
Nov 6, 2016
@mikepb : Took forever to circle back to this. Much appreciated.
I've integrated your patch into the setup.py here: 36bcd46
I'm running a bit blind on this, so hopefully I haven't done anything wrong. We're not fully supporting wheels at the moment, due to resource constraints.
@gushecht: Thanks a lot for your snippet. I've referenced this thread in the install docs for MacOS / OSX.
I'm closing this for now to signal that I'm not aware of further action here. Not 100% confident this is fully resolved yet, as I don't feel very across the issue. Please reopen if there's more to do.
As per today
Environment variables I used before pip installation:
export CC=/usr/local/opt/llvm/bin/clang export CXX=/usr/local/opt/llvm/bin/clang++ export PATH=/usr/local/opt/llvm/bin:$PATH export C_INCLUDE_PATH=/usr/local/opt/llvm/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/usr/local/opt/llvm/include:$CPLUS_INCLUDE_PATH export LIBRARY_PATH=/usr/local/opt/llvm/lib:$LIBRARY_PATH export DYLD_LIBRARY_PATH=/usr/local/opt/llvm/lib:$DYLD_LIBRARY_PATH
Got about 2X speed up after this.
The patch seems to be "wrong" (maybe I'm missing something, therefore the quotes).
Removing the line, instead, enables building and installation of the multithreading version of spaCy. I still find an issue when specifying
I saw that directory in the egg from @mikepb, but that's about it.