New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Update Windows installation / support #364

Closed
ahartikainen opened this Issue Jul 7, 2017 · 22 comments

Comments

Projects
None yet
6 participants
@ahartikainen
Collaborator

ahartikainen commented Jul 7, 2017

Update the installation notes to use conda distribution with MSYS2 GCC compiler (m2w64 toolchain). Add disclaimer/warning concerning MSVC + bugs.

We should aim for something similar that theano uses: http://deeplearning.net/software/theano/install_windows.html

  • Anaconda / Miniconda installer
    • example with conda environment
  • GCC step: conda install -c msys2 m2w64-toolchain=5.3.0 to install MSYS2 GCC
  • Other packages either in the first step or with explicit example with conda-forge
  • TODO: Do we suggest to use distutils.cfg (this can leak to other environments) or some flag in source code? CC='gcc'?

Examples

# Optional - create a conda environment
conda create -n stan_env python=3.6
source activate stan_env

#dependencies
conda install numpy cython matplotlib scipy pandas

# gcc compiler components
conda install libpython
conda install -c msys2 m2w64-toolchain=5.3.0

# step with distutils.cfg
... (can we somehow automate this?)

# pip / conda
pip install pystan

We could also try to do conda step with pip.

pip install conda
conda install libpython
conda install -c msys2 m2w64-toolchain=5.3.0

@ariddell

This comment has been minimized.

Show comment
Hide comment
@ariddell

ariddell Jul 10, 2017

Member

If you've verified this works, we should certainly add it now. Thanks for digging into this.

Member

ariddell commented Jul 10, 2017

If you've verified this works, we should certainly add it now. Thanks for digging into this.

@lwiklendt

This comment has been minimized.

Show comment
Hide comment
@lwiklendt

lwiklendt Jul 15, 2017

Fantastic. Looks like n_jobs > 1 works too.

Fantastic. Looks like n_jobs > 1 works too.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Jul 18, 2017

Collaborator

We have to either remove the lines 286-287 in Model.py or update the if statement to recognize if the compiler is msvc or gcc.

if platform.platform().startswith('Win'):
    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']

Do we have any reliable way to do it before compilation or can we add a keyword if msvc compiler is used?

Also at the same time we could update line 29 so it would first try to import Extension from setuptools and if that fails then from distutils. This would fix the issue with msvc 14.1 / 2017 if user has updated the setuptools.

In my opinion we should not break the msvc compilation but only give warning?

Collaborator

ahartikainen commented Jul 18, 2017

We have to either remove the lines 286-287 in Model.py or update the if statement to recognize if the compiler is msvc or gcc.

if platform.platform().startswith('Win'):
    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']

Do we have any reliable way to do it before compilation or can we add a keyword if msvc compiler is used?

Also at the same time we could update line 29 so it would first try to import Extension from setuptools and if that fails then from distutils. This would fix the issue with msvc 14.1 / 2017 if user has updated the setuptools.

In my opinion we should not break the msvc compilation but only give warning?

@dimitry12

This comment has been minimized.

Show comment
Hide comment
@dimitry12

dimitry12 Jul 18, 2017

For me the instructions didn't work when using 64-bit Anaconda 4.2.2 Python 3.5 on Windows:

I ended-up switching to Anaconda 4.4 Python 3.6 where these instructions worked.

Let me know if you want me to reproduce and get more details. Thanks!

For me the instructions didn't work when using 64-bit Anaconda 4.2.2 Python 3.5 on Windows:

I ended-up switching to Anaconda 4.4 Python 3.6 where these instructions worked.

Let me know if you want me to reproduce and get more details. Thanks!

@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Aug 29, 2017

@ahartikainen I have followed your instructions precisely.
All the installation process went well.

INFO:pystan:COMPILING THE C++ CODE FOR MODEL

still not working. First error says:

DistutilsExecError: command 'C:\Program Files\Python\Anaconda3\Library\mingw-w64\bin\gcc.exe' failed with exit status 1

The other are CompileErrors
Any suggestions?


Windows 10, Anaconda 4.4

sursu commented Aug 29, 2017

@ahartikainen I have followed your instructions precisely.
All the installation process went well.

INFO:pystan:COMPILING THE C++ CODE FOR MODEL

still not working. First error says:

DistutilsExecError: command 'C:\Program Files\Python\Anaconda3\Library\mingw-w64\bin\gcc.exe' failed with exit status 1

The other are CompileErrors
Any suggestions?


Windows 10, Anaconda 4.4

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 29, 2017

Collaborator

Hi, did you comment out the lines in pystan/model.py file?

C:\Program Files\Python\Anaconda3\Lib\site-packages\pystan\model.py

Change lines 286-287 from

if platform.platform().startswith('Win'):
    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']

to

# if platform.platform().startswith('Win'):
#    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']
Collaborator

ahartikainen commented Aug 29, 2017

Hi, did you comment out the lines in pystan/model.py file?

C:\Program Files\Python\Anaconda3\Lib\site-packages\pystan\model.py

Change lines 286-287 from

if platform.platform().startswith('Win'):
    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']

to

# if platform.platform().startswith('Win'):
#    extra_compile_args = ['/EHsc', '-DBOOST_DATE_TIME_NO_LIB']
@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Aug 29, 2017

Done that. No success. Same errors.

sursu commented Aug 29, 2017

Done that. No success. Same errors.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 29, 2017

Collaborator

Ok, before we go further, could you give all the errors you get. Usually the 'last' error is just saying that something went wrong. Before that (usually the first error in the 'list') are the more meaningfull ones.

Like let's say the problem was '/EHsc' that is meant for the msvc and not gcc. Then there is an error message saying that it did not understand the '/EHsc'.

Also, when you say that you followed the instructions. Did you create a new conda environment or did you use the main Anaconda env? If you did create a new conda environment then the pystan/model.py file is in another directory found under main Anaconda folder (Anaconda/envs/envname/...).

Collaborator

ahartikainen commented Aug 29, 2017

Ok, before we go further, could you give all the errors you get. Usually the 'last' error is just saying that something went wrong. Before that (usually the first error in the 'list') are the more meaningfull ones.

Like let's say the problem was '/EHsc' that is meant for the msvc and not gcc. Then there is an error message saying that it did not understand the '/EHsc'.

Also, when you say that you followed the instructions. Did you create a new conda environment or did you use the main Anaconda env? If you did create a new conda environment then the pystan/model.py file is in another directory found under main Anaconda folder (Anaconda/envs/envname/...).

@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Aug 30, 2017

Hi, irrespective to where I install it, in the stan_env or Anaconda env (at first I indeed installed it in the latter env because source activate stan_env didn't run properly) when I run the following code, I get errors:

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code)
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_5944b02c79788fa0db5b3a93728ca2bf NOW.



---------------------------------------------------------------------------

DistutilsExecError                        Traceback (most recent call last)

C:\Program Files\Python\Anaconda3\lib\distutils\cygwinccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    172                 self.spawn(self.compiler_so + cc_args + [src, '-o', obj] +
--> 173                            extra_postargs)
    174             except DistutilsExecError as msg:


C:\Program Files\Python\Anaconda3\lib\distutils\ccompiler.py in spawn(self, cmd)
    908     def spawn(self, cmd):
--> 909         spawn(cmd, dry_run=self.dry_run)
    910 


C:\Program Files\Python\Anaconda3\lib\distutils\spawn.py in spawn(cmd, search_path, verbose, dry_run)
     37     elif os.name == 'nt':
---> 38         _spawn_nt(cmd, search_path, dry_run=dry_run)
     39     else:


C:\Program Files\Python\Anaconda3\lib\distutils\spawn.py in _spawn_nt(cmd, search_path, verbose, dry_run)
     80             raise DistutilsExecError(
---> 81                   "command %r failed with exit status %d" % (cmd, rc))
     82 


DistutilsExecError: command 'C:\\Program Files\\Python\\Anaconda3\\Library\\mingw-w64\\bin\\gcc.exe' failed with exit status 1


During handling of the above exception, another exception occurred:


CompileError                              Traceback (most recent call last)

<ipython-input-6-15d1abb9774a> in <module>()
----> 1 model = pystan.StanModel(model_code=model_code)


C:\Program Files\Python\Anaconda3\lib\site-packages\pystan\model.py in __init__(self, file, charset, model_name, model_code, stanc_ret, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args)
    309 
    310         try:
--> 311             build_extension.run()
    312         finally:
    313             if redirect_stderr:


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in run(self)
    337 
    338         # Now actually compile and link everything.
--> 339         self.build_extensions()
    340 
    341     def check_extensions_list(self, extensions):


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in build_extensions(self)
    446             self._build_extensions_parallel()
    447         else:
--> 448             self._build_extensions_serial()
    449 
    450     def _build_extensions_parallel(self):


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in _build_extensions_serial(self)
    471         for ext in self.extensions:
    472             with self._filter_build_errors(ext):
--> 473                 self.build_extension(ext)
    474 
    475     @contextlib.contextmanager


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in build_extension(self, ext)
    531                                          debug=self.debug,
    532                                          extra_postargs=extra_args,
--> 533                                          depends=ext.depends)
    534 
    535         # XXX outdated variable, kept here in case third-part code


C:\Program Files\Python\Anaconda3\lib\distutils\ccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends)
    572             except KeyError:
    573                 continue
--> 574             self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    575 
    576         # Return *all* object filenames, not just the ones we just built.


C:\Program Files\Python\Anaconda3\lib\distutils\cygwinccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    173                            extra_postargs)
    174             except DistutilsExecError as msg:
--> 175                 raise CompileError(msg)
    176 
    177     def link(self, target_desc, objects, output_filename, output_dir=None,


CompileError: command 'C:\\Program Files\\Python\\Anaconda3\\Library\\mingw-w64\\bin\\gcc.exe' failed with exit status 1

sursu commented Aug 30, 2017

Hi, irrespective to where I install it, in the stan_env or Anaconda env (at first I indeed installed it in the latter env because source activate stan_env didn't run properly) when I run the following code, I get errors:

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code)
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_5944b02c79788fa0db5b3a93728ca2bf NOW.



---------------------------------------------------------------------------

DistutilsExecError                        Traceback (most recent call last)

C:\Program Files\Python\Anaconda3\lib\distutils\cygwinccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    172                 self.spawn(self.compiler_so + cc_args + [src, '-o', obj] +
--> 173                            extra_postargs)
    174             except DistutilsExecError as msg:


C:\Program Files\Python\Anaconda3\lib\distutils\ccompiler.py in spawn(self, cmd)
    908     def spawn(self, cmd):
--> 909         spawn(cmd, dry_run=self.dry_run)
    910 


C:\Program Files\Python\Anaconda3\lib\distutils\spawn.py in spawn(cmd, search_path, verbose, dry_run)
     37     elif os.name == 'nt':
---> 38         _spawn_nt(cmd, search_path, dry_run=dry_run)
     39     else:


C:\Program Files\Python\Anaconda3\lib\distutils\spawn.py in _spawn_nt(cmd, search_path, verbose, dry_run)
     80             raise DistutilsExecError(
---> 81                   "command %r failed with exit status %d" % (cmd, rc))
     82 


DistutilsExecError: command 'C:\\Program Files\\Python\\Anaconda3\\Library\\mingw-w64\\bin\\gcc.exe' failed with exit status 1


During handling of the above exception, another exception occurred:


CompileError                              Traceback (most recent call last)

<ipython-input-6-15d1abb9774a> in <module>()
----> 1 model = pystan.StanModel(model_code=model_code)


C:\Program Files\Python\Anaconda3\lib\site-packages\pystan\model.py in __init__(self, file, charset, model_name, model_code, stanc_ret, boost_lib, eigen_lib, verbose, obfuscate_model_name, extra_compile_args)
    309 
    310         try:
--> 311             build_extension.run()
    312         finally:
    313             if redirect_stderr:


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in run(self)
    337 
    338         # Now actually compile and link everything.
--> 339         self.build_extensions()
    340 
    341     def check_extensions_list(self, extensions):


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in build_extensions(self)
    446             self._build_extensions_parallel()
    447         else:
--> 448             self._build_extensions_serial()
    449 
    450     def _build_extensions_parallel(self):


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in _build_extensions_serial(self)
    471         for ext in self.extensions:
    472             with self._filter_build_errors(ext):
--> 473                 self.build_extension(ext)
    474 
    475     @contextlib.contextmanager


C:\Program Files\Python\Anaconda3\lib\distutils\command\build_ext.py in build_extension(self, ext)
    531                                          debug=self.debug,
    532                                          extra_postargs=extra_args,
--> 533                                          depends=ext.depends)
    534 
    535         # XXX outdated variable, kept here in case third-part code


C:\Program Files\Python\Anaconda3\lib\distutils\ccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends)
    572             except KeyError:
    573                 continue
--> 574             self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    575 
    576         # Return *all* object filenames, not just the ones we just built.


C:\Program Files\Python\Anaconda3\lib\distutils\cygwinccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    173                            extra_postargs)
    174             except DistutilsExecError as msg:
--> 175                 raise CompileError(msg)
    176 
    177     def link(self, target_desc, objects, output_filename, output_dir=None,


CompileError: command 'C:\\Program Files\\Python\\Anaconda3\\Library\\mingw-w64\\bin\\gcc.exe' failed with exit status 1
@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Aug 30, 2017

Collaborator

Ok, it tries to use mingw-w64. Edit/create ´distutils.cfg´ file and use mingw32 keyword

https://stackoverflow.com/questions/3297254/how-to-use-mingws-gcc-compiler-when-installing-python-package-using-pip

[build]
compiler = mingw32
Collaborator

ahartikainen commented Aug 30, 2017

Ok, it tries to use mingw-w64. Edit/create ´distutils.cfg´ file and use mingw32 keyword

https://stackoverflow.com/questions/3297254/how-to-use-mingws-gcc-compiler-when-installing-python-package-using-pip

[build]
compiler = mingw32
@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Aug 31, 2017

Interestingly, there already is a file distutils.cfg with that text in it both in PYTHONPATH\Lib\distutils and in PYTHONPATH\envs\stan_env\Lib\distutils.

Was it supposed to run the x86_64-w64-mingw32-gcc.exe?

Would it be possible/easier to make it work with clang or msvc.

sursu commented Aug 31, 2017

Interestingly, there already is a file distutils.cfg with that text in it both in PYTHONPATH\Lib\distutils and in PYTHONPATH\envs\stan_env\Lib\distutils.

Was it supposed to run the x86_64-w64-mingw32-gcc.exe?

Would it be possible/easier to make it work with clang or msvc.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 2, 2017

Collaborator

The problem is with the python + compiler.
msvc is broken (with Eigen). And python does not work with clang.

I did recheck and your compiler looks correct. One idea could be that you have spaces in your path to gcc. That could be a problem (did some googling and this could really be your problem). Can you install to a user folder / path without spaces?

Collaborator

ahartikainen commented Sep 2, 2017

The problem is with the python + compiler.
msvc is broken (with Eigen). And python does not work with clang.

I did recheck and your compiler looks correct. One idea could be that you have spaces in your path to gcc. That could be a problem (did some googling and this could really be your problem). Can you install to a user folder / path without spaces?

@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Sep 3, 2017

Ari, I've installed to a path without spaces, but I get precisely the same errors as above.

As you might have seen already, the jupyter terminal says:

gcc: error: /EHsc: No such file or directory

The pystan version which gets installed is 2.16.0.0.

sursu commented Sep 3, 2017

Ari, I've installed to a path without spaces, but I get precisely the same errors as above.

As you might have seen already, the jupyter terminal says:

gcc: error: /EHsc: No such file or directory

The pystan version which gets installed is 2.16.0.0.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 3, 2017

Collaborator

Oh, great. If that is the error, then you need to edit the pystan/model.py file manually (there are some options for msvc but gcc will fail with these). See my earlier reply.

Make sure you are editing the right path. To see your pystan path try this:

import pystan
print(pystan) #or pystan.__file__

After the edit. Restart your python session so you will import the edited version.

Collaborator

ahartikainen commented Sep 3, 2017

Oh, great. If that is the error, then you need to edit the pystan/model.py file manually (there are some options for msvc but gcc will fail with these). See my earlier reply.

Make sure you are editing the right path. To see your pystan path try this:

import pystan
print(pystan) #or pystan.__file__

After the edit. Restart your python session so you will import the edited version.

@sursu

This comment has been minimized.

Show comment
Hide comment
@sursu

sursu Sep 3, 2017

It finally worked!!

I have commented out the lines 286-287 previously, when I have noticed that error. But I did not restart the python session before checking for the effects.

Thanks a lot for your help.

sursu commented Sep 3, 2017

It finally worked!!

I have commented out the lines 286-287 previously, when I have noticed that error. But I did not restart the python session before checking for the effects.

Thanks a lot for your help.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 7, 2017

Collaborator

Hi,
some info on the progress.

I now have an idea how to implement this. There is an way to inject (force) gcc to Cython if it is found on the system (this will work even without the distutils.cfg) and still leave an option to use msvc. I'm probably going to put this under the kwargs. The only problem is, that the process needs to call subprocess to work with systems without gcc. If we implement the subprocess option, then we can print the compiler version to "screen" if verbose is set to true.

Basically we need to call gcc --version for gcc (and cl for msvc but this is optional and I'm not even sure if it works)

Also fixing msvc 14.1 (2017) is possible. (setuptools vs distutils problem).

I still need to test what versions of python do even work with m2w64-toolchain. (2.7, 3.4, 3.5, 3.6)

I will get back to this next week.

Collaborator

ahartikainen commented Sep 7, 2017

Hi,
some info on the progress.

I now have an idea how to implement this. There is an way to inject (force) gcc to Cython if it is found on the system (this will work even without the distutils.cfg) and still leave an option to use msvc. I'm probably going to put this under the kwargs. The only problem is, that the process needs to call subprocess to work with systems without gcc. If we implement the subprocess option, then we can print the compiler version to "screen" if verbose is set to true.

Basically we need to call gcc --version for gcc (and cl for msvc but this is optional and I'm not even sure if it works)

Also fixing msvc 14.1 (2017) is possible. (setuptools vs distutils problem).

I still need to test what versions of python do even work with m2w64-toolchain. (2.7, 3.4, 3.5, 3.6)

I will get back to this next week.

@ariddell

This comment has been minimized.

Show comment
Hide comment
@ariddell

ariddell Sep 18, 2017

Member

Steve Downer has a informative blog post which might be helpful for people:

Member

ariddell commented Sep 18, 2017

Steve Downer has a informative blog post which might be helpful for people:

@reijerh

This comment has been minimized.

Show comment
Hide comment
@reijerh

reijerh Sep 22, 2017

The solution in the OP works, just make sure you edit the correct model.py file, I had to make sure to edit the one in my created environment's path: C:\Users\XXX\Anaconda2\envs\stan_env\Lib\site-packages\pystan (where stan_env is the environment)

reijerh commented Sep 22, 2017

The solution in the OP works, just make sure you edit the correct model.py file, I had to make sure to edit the one in my created environment's path: C:\Users\XXX\Anaconda2\envs\stan_env\Lib\site-packages\pystan (where stan_env is the environment)

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 22, 2017

Collaborator

Hi, current development branch (2.17 released in any these days) works without editing model.py file.

Collaborator

ahartikainen commented Sep 22, 2017

Hi, current development branch (2.17 released in any these days) works without editing model.py file.

@ariddell

This comment has been minimized.

Show comment
Hide comment
@ariddell

ariddell Sep 22, 2017

Member

@ahartikainen Are there any quick edits that need to be made to http://pystan.readthedocs.io/en/latest/windows.html before releasing 2.17?

Member

ariddell commented Sep 22, 2017

@ahartikainen Are there any quick edits that need to be made to http://pystan.readthedocs.io/en/latest/windows.html before releasing 2.17?

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 22, 2017

Collaborator

I created a PR #377 with updated installation instructions.

Collaborator

ahartikainen commented Sep 22, 2017

I created a PR #377 with updated installation instructions.

@ahartikainen

This comment has been minimized.

Show comment
Hide comment
@ahartikainen

ahartikainen Sep 24, 2017

Collaborator

Updated in #371 & #377

MSVC 2017 support comes when Python updates the distutils.

Collaborator

ahartikainen commented Sep 24, 2017

Updated in #371 & #377

MSVC 2017 support comes when Python updates the distutils.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment