New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precompiled wheels for Windows #5461

Closed
matysek opened this Issue Nov 6, 2015 · 30 comments

Comments

Projects
None yet
@matysek

matysek commented Nov 6, 2015

It would be very helpful to have binary wheel packages for Windows platform.

I think it might work to build those wheels using Appveyor service: http://www.appveyor.com/

@rgommers

This comment has been minimized.

Member

rgommers commented Nov 6, 2015

Would be nice, but there's no way the builds are going to be fast enough for Appveyor. Even for numpy it's quite hard to get Appveyor to work.

@hashtonmartyn

This comment has been minimized.

hashtonmartyn commented Nov 17, 2015

There are wheels for OSX on scipy version 0.16.1 but not for Windows, what's required to make this happen?

@pv

This comment has been minimized.

Member

pv commented Nov 18, 2015

More or less: find someone who is interested to spend effort on it, and knows how to do it (prepare a build setup that is reproducible by others and produces wheels working on the wide range of platforms).
The discussion on Numpy wheels probably is relevant here, the situation is similar:
numpy/numpy#5479

@ogrisel

This comment has been minimized.

Contributor

ogrisel commented Dec 15, 2015

Furthermore there is no fortran compiler on appveyor. There is the mingwpy that is promising but:

  • it does not support Python 3.5 yet (neither in upstream mingw-w64)
  • mingwpy is still very experimental and not packaged yet
@hofst

This comment has been minimized.

hofst commented Mar 23, 2016

I strongly support this request. As it currently stands, scipy is the only package that requires heavy manual workarounds when installing

@rgommers

This comment has been minimized.

Member

rgommers commented Mar 23, 2016

@hofst this is being worked on: you can follow the progress at http://mingwpy.github.io/ (see roadmap there, mailing list and https://github.com/mingwpy)

leycec added a commit to leycec/pyinstaller that referenced this issue Apr 29, 2016

Tests: test_scipy_special() added.
This test exercises a new hidden import solving pyinstaller#1908. For maintainability, all
SciPy tests have also been moved from "test_libraries.py" into a new
"test_hooks/test_scipy.py" script. Thanks to scipy/scipy#5461, these tests have
also been marked as failing under Windows.

Sadly, this test has also exposed pyinstaller#1919, a new non-trivial "ModuleGraph" issue.
Until addressed, the test_scipy_special() test must be unconditionally skipped.

@ev-br ev-br added the Build issues label May 18, 2016

@cas--

This comment has been minimized.

cas-- commented Mar 15, 2017

I am curious as to the state of this issue. Perhaps wheels could be build locally and pushed to PyPi for each release in interim until appvoyeur usage is available. Did you know Christoph Gohlke is providing a range of windows wheels?

So with numpy wheels available on pypi it would be useful for those using Windows who only need to install Python and pip install scipy therefore dropping the requirement of bloated freemium Anaconda that seems to be pushed far too much.

@pv

This comment has been minimized.

Member

pv commented Mar 15, 2017

You can read the discussion on the numpy ticket linked above to understand the issue in more detail.

@cas--

This comment has been minimized.

cas-- commented Mar 15, 2017

I did read some of it but there is no indication as to what is specifically the issue with providing scipy wheels since that ticket is closed and there are no recent status updates here. It would be good if you could just provide a quick summary?

I am asking because my concern is with the Python distribution ecosystem being fragmented by Anaconda. I was not aware of Anaconda until I started helping with introduction to Python workshops for scientists and was taken aback that they recommended Anaconda from the outset for an introduction series. Since Linux and OSX can use packages or pip wheels, only Windows users are left out now.

@rgommers

This comment has been minimized.

Member

rgommers commented Mar 15, 2017

I am asking because my concern is with the Python distribution ecosystem being fragmented by Anaconda.

That's a pretty wild mischaracterisation. Conda and pip have complementary strengths (as recognised by the whole Python/distutils-sig community, not just the scientific one), and Anaconda as a commercial distribution solves user issues that otherwise go unsolved. Note that the SciPy in conda-forge, which uses free compilers, also has no Windows builds.

The issue is simply with missing compiler support for Fortran on Windows, it can only be done reliably with Intel Fortran + Intel MKL at the moment which is $$-only. The most likely candidate to solve the compiler availability issue at this point is Flang, which will be open-sourced quite soon.

@ilayn

This comment has been minimized.

Member

ilayn commented Mar 15, 2017

As a summary, MKL license, compiler license, lack of support by MS for Fortran, community size (compared to GNU umbrella), runtime mania (look at Add/Remove programs for Visual C++ entries at least 17 of them). DLL hell and so on.

If @cgohlke stops providing those wheels, scientific stack would receive a major blow so s/he is doing an amazing service to the community.

@ogrisel

This comment has been minimized.

Contributor

ogrisel commented Mar 16, 2017

The most likely candidate to solve the compiler availability issue at this point is Flang, which will be open-sourced quite soon.

+1, once we have an opensource flang working on windows and generating binary extensions that are compatible with programs built with the MS 2015 C/C++ toolchain we can generate windows wheels for scipy and upload them to PyPI (for Python 3.5+).

This might take some time to get there though.

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented Mar 16, 2017

@rgommers

This comment has been minimized.

Member

rgommers commented Mar 27, 2017

Intel Fortran modifies its run-time license for us and gives us some licenses (see:
https://software.intel.com/en-us/comment/1881526#comment-1881526).

I'm pretty sure that that license won't change. Nathaniel also talked to some Intel people last year (more authoratative ones than on the support forum) and confirmed that building with Intel Fortran is not going to happen.

@ghost

This comment has been minimized.

ghost commented Apr 16, 2017

May I ask why @cgohlke is able to post the wheels on his website but numpy/scipy is not able to post the wheels on pip? Specifically, why numpy cannot post the MKL wheels on pip?

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented Apr 17, 2017

You may certainly ask :). We do post numpy wheels. Scipy wheels need Fortran, for which the only current option is Intel Fortran. That makes the wheels susceptible to the Intel Fortran run-time license, which makes the person releasing the wheels responsible for any legal fees incurred by Intel if someone sues Intel as a result of using Scipy (compiled with Intel Fortran). Christoph is very generously taking on that risk, but we (the scipy org) do not think we should or can take on that risk.

@ghost

This comment has been minimized.

ghost commented Apr 17, 2017

Another option is using f2c to translate to intermediate C that can then be compiled with MSVC. Performance is questionable probably but this can be used for 90% of users.

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented Apr 17, 2017

Yes, that would be an option if the Fortran was all f77, but I believe that is no longer the case.

@rgommers

This comment has been minimized.

Member

rgommers commented Apr 20, 2017

Yes, that would be an option if the Fortran was all f77, but I believe that is no longer the case.

AFAIK we haven't merged any Fortran 90/95 code yet, but there is at least one PR that would add such code, and we don't want to block adding more later.

@ghost

This comment has been minimized.

ghost commented Apr 20, 2017

AFAIK we haven't merged any Fortran 90/95 code yet, but there is at least one PR that would add such code, and we don't want to block adding more later.

Perhaps this could be placed in a separate folder so that only the f77 functions are available in the wheels. It's not an ideal solution but frankly the lack of available wheels is wasting everyone's time.

@rgommers

This comment has been minimized.

Member

rgommers commented Apr 20, 2017

Perhaps this could be placed in a separate folder so that only the f77 functions are available in the wheels.

I don't think we want to ship incomplete versions of scipy like that. Plus f2c is not a great solution anyway. Within the next month or so we expect an announcement of a Fortran front-end for Clang being open sourced. That is currently the most promising way forward.

It's not an ideal solution but frankly the lack of available wheels is wasting everyone's time.

It is a pain, but we have limited control over it. There's a reason that we strongly recommend users to use Anaconda, Canopy or similar. Windows + compilers + pip/wheels is a bad idea for scientific work even if we would have a wheel available.

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented May 24, 2017

See conda-forge/openblas-feedstock#2 (comment) for update on Flang. Summary: still some way off being useful to us, but at least it's open source now.

@ghost

This comment has been minimized.

ghost commented Jul 12, 2017

@matthew-brett Why not compile a dll with the gcc stack and then call wrapper c functions from the msvc stack?

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented Jul 13, 2017

Some possible progress on wheel building over at #7597

@matthew-brett

This comment has been minimized.

Contributor

matthew-brett commented Jul 13, 2017

@xoviat - could you expand on your suggestion? At the moment I am not sure what you mean.

@ghost

This comment has been minimized.

ghost commented Jul 14, 2017

@matthew-brett I actually found a blog post here that explains the idea pretty well (nothing in computer science is a new idea anymore).

Basically the only officially supported ABI bridge between gcc and msvc is through a DLL. It is therefore possible to automatically detect all calls to fortran code (because the caller is typically c or c++, not python which would make that impossible) and then generate wrappers:

  1. On the MSVC side, replace the fortran calls with wrapper functions that create a load-time dependency on an imaginary DLL that we will create. Simply put, __declspec(dllimport).
  2. On the gfortran side, wrap all functions with C code that exports from the DLL. Simply put, __declspec(dllexport).

Cmake apparently had this idea long before I did and makes this effortless. However, I am actually not familiar with the scipy build system so I am not sure what would be required here.

@ev-br ev-br marked this as a duplicate of #7645 Jul 23, 2017

Juanlu001 added a commit to Juanlu001/poliastro that referenced this issue Jul 31, 2017

Revert changes in AppVeyor conf
No SciPy wheels on Windows in the short term, see
scipy/scipy#5461

Keep using conda for conflicting packages, namely
NumPy, SciPy and numba.
@pv

This comment has been minimized.

Member

pv commented Sep 2, 2017

Prerelease wheels are available as explained in a post on the mailing list. Testing is welcome.

@Juanlu001

This comment has been minimized.

Contributor

Juanlu001 commented Sep 2, 2017

I did try them and work like a charm, no issues found.

@mikofski

This comment has been minimized.

Contributor

mikofski commented Sep 7, 2017

@rgommers @ogrisel @matthew-brett and others, any news on the open-source Flang LLVM option? Will the mingwpy + generic MSVCRT be the final solution? What are the differences between the two proposals? Is anyone working on the Flang LLVM option, or still waiting for it to be open-sourced? I've found numerous online references to Flang, but I can figure out what or which is the most significant effort, and what their states are.

There seem to be at least two versions of Flang:

I hope this doesn't come across as favoring one solution (flang v. mingwpy) over another. I am extremely grateful for any solution. I am just curious. Thanks!

@ghost

This comment has been minimized.

ghost commented Sep 7, 2017

The latter of the two links is the correct effort but someone needs to implement windows support. It's even worse than that, though because even with the fortran compiler, we still could not link OpenBLAS.

@pv pv closed this Oct 28, 2017

@pv pv added this to the 1.0.0 milestone Oct 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment