Windows build issue

[MonkeyChap]

Hi,

I'm trying to build with mingw32 on windows (Anaconda python v3.6) and getting errors that look like they're in the python rather than the C:

C:\annoy>python setup.py build --compiler=mingw32
running build
running build_py
running build_ext
Traceback (most recent call last):
File "setup.py", line 69, in
setup_requires=['nose>=1.0']
File "C:\Users\Ian\Anaconda3\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Users\Ian\Anaconda3\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Users\Ian\Anaconda3\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\Ian\Anaconda3\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "C:\Users\Ian\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\Users\Ian\Anaconda3\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\Ian\Anaconda3\lib\site-packages\setuptools\command\build_ext.py", line 75, in run
_build_ext.run(self)
File "C:\Users\Ian\Anaconda3\lib\site-packages\Cython\Distutils\old_build_ext.py", line 185, in run
_build_ext.build_ext.run(self)
File "C:\Users\Ian\Anaconda3\lib\distutils\command\build_ext.py", line 308, in run
force=self.force)
File "C:\Users\Ian\Anaconda3\lib\distutils\ccompiler.py", line 1031, in new_compiler
return klass(None, dry_run, force)
File "C:\Users\Ian\Anaconda3\lib\distutils\cygwinccompiler.py", line 282, in init
CygwinCCompiler.init (self, verbose, dry_run, force)
File "C:\Users\Ian\Anaconda3\lib\distutils\cygwinccompiler.py", line 126, in init
if self.ld_version >= "2.10.90":
TypeError: '>=' not supported between instances of 'NoneType' and 'str'

Any ideas what's causing it ?

[erikbern]

sorry, but seems this is something in cygwin/anaconda rather than annoy?

[MonkeyChap]

Hi Erik,

After a bit of pain, I set up a new VM (I'm a Mac user) with no Anaconda, and just a vanilla mingw32 install. Still doesn't work, and the errors also seem to be in the disutils library. Is it possible that the problem is a python 3.6 compatibility thing ?

C:\annoy>python setup.py build --compiler=mingw32
running build
running build_py
running build_ext
Traceback (most recent call last):
File "setup.py", line 69, in
setup_requires=['nose>=1.0']
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cor
e.py", line 148, in setup
dist.run_commands()
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\dis
t.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\dis
t.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\com
mand\build.py", line 135, in run
self.run_command(cmd_name)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cmd
.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\dis
t.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\site-packages
\setuptools\command\build_ext.py", line 75, in run
_build_ext.run(self)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\com
mand\build_ext.py", line 308, in run
force=self.force)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cco
mpiler.py", line 1031, in new_compiler
return klass(None, dry_run, force)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cyg
winccompiler.py", line 282, in init
CygwinCCompiler.init (self, verbose, dry_run, force)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cyg
winccompiler.py", line 157, in init
self.dll_libraries = get_msvcr()
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36-32\lib\distutils\cyg
winccompiler.py", line 86, in get_msvcr
raise ValueError("Unknown MS Compiler version %s " % msc_ver)
ValueError: Unknown MS Compiler version 1900

PS - there's no cygwin on this VM, so the cygwin errors look a bit odd.

[MonkeyChap]

OK, so some progress, but still doesn't work. For anyone else hitting this problem, you need to patch the distutils section in your Python install (ugly) as follows:

https://stackoverflow.com/questions/34135280/valueerror-unknown-ms-compiler-version-1900

That said, it's still not compiling...

writing build\temp.win-amd64-3.6\Release\src\annoylib.cp36-win_amd64.def
C:\MinGW\bin\g++.exe -shared -s build\temp.win-amd64-3.6\Release\src\annoymodule
.o build\temp.win-amd64-3.6\Release\src\annoylib.cp36-win_amd64.def -LC:\Users\I
an\AppData\Local\Programs\Python\Python36\libs -LC:\Users\Ian\AppData\Local\Prog
rams\Python\Python36\PCbuild\amd64 -lpython36 -lvcruntime140 -o build\lib.win-am
d64-3.6\annoy\annoylib.cp36-win_amd64.pyd
c:/mingw/bin/../lib/gcc/mingw32/5.3.0/../../../../mingw32/bin/ld.exe: cannot fin
d -lvcruntime140
collect2.exe: error: ld returned 1 exit status
error: command 'C:\MinGW\bin\g++.exe' failed with exit status 1

[MonkeyChap]

Hmmm...given what a PITA this is, is it worth putting a Windows binary up on the repo ?

[erikbern]

would be great if someone could figure out how to build it consistently

https://www.appveyor.com/ would be nice too

[MonkeyChap]

Agreed - Annoy works fantastically well on my Mac, but after hours and hours of trying everything I can think of, I still can't build it on Windows (7 or 10).

[tjrileywisc]

@MonkeyChap
I have a fork that compiles on Windows (comfirmed to compile for Win7&10/ python3.5 & 3.6 with VS 2015 compiler) at https://github.com/tjrileywisc/annoy if you want to try that. I confirmed that all examples run.

I'm trying to confirm that it still compiles in gcc before sending a PR (I only have access to bash subsystem for Windows so I'm not 100% sure it is the same though). Most changes were in the annoylib.h header but I had to fiddle with compiler arguments in setup.py as well.

[MonkeyChap]

Thanks - I'll give it a try ! Would be really great to just have something that installs with pip though :)

[erikbern]

very nice @tjrileywisc !

[MonkeyChap]

I can confirm it compiles (first time !) on windows with no messing about required.

I did find another another problem though which is totally unrelated to the Windows stuff. I'll dig into it some more and post another thread.

[MBetters]

Hello, I'm trying to build on windows and also had difficulty. I cloned @tjrileywisc's branch. @MonkeyChap, how did you compile it? I appreciate any help!

[tjrileywisc]

@MBetters

It should compile if you just run setup.py install.

[MBetters]

@tjrileywisc
I changed into the directory and ran python setup.py install, which gave me some errors...

running install
running bdist_egg
running egg_info
writing annoy.egg-info\PKG-INFO
writing top-level names to annoy.egg-info\top_level.txt
writing dependency_links to annoy.egg-info\dependency_links.txt
reading manifest file 'annoy.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'annoy.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_py
running build_ext
building 'annoy.annoylib' extension
C:\Users\591961\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Users\591961\Desktop\fusion\system\fef-tuning\ml_model\bin\venv\PC /Tpsrc/annoymodule.cc /Fobuild\temp.win32-2.7\Release\src/annoymodule.obj
annoymodule.cc
C:\Users\591961\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\xlocale(342) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
c:\users\591961\desktop\fusion\system\fef-tuning\ml_model\bin\annoy\src\annoylib.h(31) : fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory
error: command 'C:\\Users\\591961\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2

[tjrileywisc]

@MBetters
The stdint.h header file is missing (see the second to last line in your output). This file isn't included with VS 2008 or the VS C++ compiler for Python 2.7 installer.

You can get it from here, and since you're using the VS C++ compiler for Python 2.7 compiler, you'll want to put it into
'C:\Users<userprofile>\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include'.

You should be able to compile it after that.

[MBetters]

@tjrileywisc
So I added the header file. Now, Im able to run python setup.py install no problem. However, when I go to run the following code, the python interpreter crashes without a stack trace. It just says that python.exe has stopped working.

from annoy import AnnoyIndex
import random

f = 40
t = AnnoyIndex(f)  # Length of item vector that will be indexed
for i in xrange(1000):
    v = [random.gauss(0, 1) for z in xrange(f)]
    t.add_item(i, v)

t.build(10) # 10 trees
t.save('test.ann')

# ...

u = AnnoyIndex(f)
u.load('test.ann') # super fast, will just mmap the file
print(u.get_nns_by_item(0, 1000)) # will find the 1000 nearest neighbors

[MBetters]

Nevermind, I just updated my project to python 3.6 and installing and using annoy worked. Although, it would be nice to have this work with windows 10/python 2.7 in the future.

[tjrileywisc]

For anyone still monitoring this thread, Windows support from my fork was just merged (#233)

Only python 3 is working (3.6 is working, anything with a MSVC 2015 compiler should be okay). 2.7 not working yet.

[stefanbschneider]

Actually, I still have the error on Windows 10 using Python 3.6 and trying to install annoy via pip:

  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.6
  creating build\lib.win-amd64-3.6\annoy
  copying annoy\__init__.py -> build\lib.win-amd64-3.6\annoy
  running build_ext
  building 'annoy.annoylib' extension
  error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools"                                                                                                  : https://visualstudio.microsoft.com/downloads/
  ----------------------------------------
  ERROR: Failed building wheel for annoy
  Running setup.py clean for annoy
Failed to build annoy
Installing collected packages: psutil, tornado, MarkupSafe, Jinja2, bokeh, annoy, imageio, PyW                                                                                                  avelets, decorator, networkx, scikit-image, rl-coach, pyflakes, pycodestyle, mccabe, entrypoin                                                                                                  ts, flake8, coverage, nose2, isort, wrapt, lazy-object-proxy, typed-ast, astroid, pylint, rlsp
  Running setup.py install for annoy: started
    Running setup.py install for annoy: finished with status 'error'
    ERROR: Complete output from command 'c:\users\stefan\git-repos\work\swc\rl-coordination\ve                                                                                                  nv\scripts\python.exe' -u -c 'import setuptools, tokenize;__file__='"'"'C:\\Users\\Stefan\\App                                                                                                  Data\\Local\\Temp\\pip-install-sxvvudzx\\annoy\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'                                                                                                  "', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile                                                                                                  (code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Stefan\AppData\Local\Temp\pip-re                                                                                                  cord-3kn2ksv3\install-record.txt' --single-version-externally-managed --compile --install-head                                                                                                  ers 'c:\users\stefan\git-repos\work\swc\rl-coordination\venv\include\site\python3.6\annoy':
    ERROR: running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.6
    creating build\lib.win-amd64-3.6\annoy
    copying annoy\__init__.py -> build\lib.win-amd64-3.6\annoy
    running build_ext
    building 'annoy.annoylib' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tool                                                                                                  s": https://visualstudio.microsoft.com/downloads/
    ----------------------------------------
ERROR: Command "'c:\users\stefan\git-repos\work\swc\rl-coordination\venv\scripts\python.exe' -                                                                                                  u -c 'import setuptools, tokenize;__file__='"'"'C:\\Users\\Stefan\\AppData\\Local\\Temp\\pip-i                                                                                                  nstall-sxvvudzx\\annoy\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code                                                                                                  =f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'ex                                                                                                  ec'"'"'))' install --record 'C:\Users\Stefan\AppData\Local\Temp\pip-record-3kn2ksv3\install-re                                                                                                  cord.txt' --single-version-externally-managed --compile --install-headers 'c:\users\stefan\git                                                                                                  -repos\work\swc\rl-coordination\venv\include\site\python3.6\annoy'" failed with error code 1 i                                                                                                  n C:\Users\Stefan\AppData\Local\Temp\pip-install-sxvvudzx\annoy\

Seems like it's not fixed after all?

[stefanbschneider]

And I do have Visual C++ installed:

[tjrileywisc]

@stefanbschneider Do you have Visual Studio 2015 or later installed? The redistributables are needed to run binaries, but annoy doesn't ship with a binary, so Visual Studio itself is needed to compile it.

You can get Visual Studio community 2015 from here:
https://visualstudio.microsoft.com/vs/older-downloads/

VS 2019 may work as well, though I haven't tried this myself. You'd have to make sure the compiler version matches what your python binary was originally made with.

[stefanbschneider]

Yes, I have VS2019 installed.

You'd have to make sure the compiler version matches what your python binary was originally made with.

How do I do that?

[stefanbschneider]

Ok, interesting. pip install annoy works fine with Python 3.7; it's only failing for Python 3.6, which I was using before.

So I guess, for Py3.6, I'd have to install VS2015.