Skip to content

Mingw Gfortran on Windows? #2829

Open
pv opened this Issue Sep 3, 2013 · 61 comments
@pv
SciPy member
pv commented Sep 3, 2013

So far we've kept to Fortran 77 due to needing g77 on Windows.

However, it seems to me Mingw32 gfortran-4.8.1 can compile a working Scipy, so perhaps it could be used for the next release? This would also allow including Fortran 90 code.

@matthew-brett

That would be really good.

@pv pv referenced this issue Sep 3, 2013
Open

Odeint rewrite #2818

@josef-pkt
SciPy member

What Blas/Lapack version can we use?
the old binaries in numpy/vendor are for MinGW 3.14, gcc 3.4 + g77

That would be really good. to echo Matthew
and I would be able to build scipy again

@rgommers
SciPy member
rgommers commented Sep 4, 2013

I can't find @cournape's description of the issue right now, but the problem is what to do with the binaries. Apparently with MinGW 4.x we need extra dll's and have nowhere to install them in such a way that they can be found.

@cournape
SciPy member
cournape commented Sep 4, 2013

@rgommers that's correct. There are only bad solutions:

  • keep the current situation with out of date compilers
  • update to mingw 4.x with the corresponding Dll Hell.

Details: there is unfortunately no way to statically link the required runtimes on gcc 4.x, and short of putting the mingw dll in C:\Python, I don't see any solution. Putting the dll in C:\Python is really horrible, though, and prone to all kind of dll-hell situations.

I have an idea that could fix this long term (requires changes to python itself), but that won't fix anything until we can give up the current python...

@Juanlu001

Let me understand this, is it a problem with the recent versions of MinGW, gfortran specifically, distutils, Python itself, this charming operative system, everything at the same time?

@cournape
SciPy member
cournape commented Sep 4, 2013

@Juanlu001 It is a problem related to recent versions of mingw + windows-specific issues, in the sense that every other system has a sane way to ship "private" shared libraries.

@matthew-brett

Maybe this discussion should go to the {numpy or scipy} mailing list with the options laid out? Am I right in thinking this issue is now blocking us from using fortran 90 in scipy and providing Windows 64 bit builds for numpy?

@Juanlu001

For future references, this is the description @cournape gave on the mingw + windows issue, in the thread @matthew-brett linked:

http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065349.html

According to the documentation, g77 cannot compile Fortran 90 (the new features not available in FORTRAN 77)

http://gcc.gnu.org/onlinedocs/gcc-3.4.6/g77/Fortran-90-Support.html

So yes, this is blocking us from using Fortran 90 (see #2818).

@pv
SciPy member
pv commented Sep 9, 2013

I would perhaps just dive for the DLL-hell solution. At least build an example installer and put it available with the beta release and see how many people manage to blow a gasket.

Of course, this may be hitting not-so-nicely the downstream guys like Anaconda & EPD (but are they really also sticking with G77)? The package list on http://scipy.org/install.html should be checked.

@cournape
SciPy member
cournape commented Sep 9, 2013

It does not matter for us (Enthought) at least, since we're not using mingw at all for our own packages. Someone from Continuum should answer, but I suspect that's not a problem for them either.

@rgommers
SciPy member

@ilanschnell do you care about g77 support for Anaconda?

@rgommers
SciPy member

If we drop support for Windows XP and Vista for the binaries we provide ourselves, then we can go to MinGW 4.x without DLL hell, correct? That may the best option - XP support has ended back in 2009 and extended (paid) support is also ending in H1 2014. And no one runs Vista, right?

@pv
SciPy member
pv commented Sep 15, 2013

There was also the suggestion from Christoph Gohlke to preload the DLLs from a custom location using ctypes. If that works (he says he used it), it would seem like a portable solution.

@ilanschnell

We also do not use mingw to compile the packages that go into Anaconda. So no, g77 is not an issue.

@njsmith
njsmith commented Sep 16, 2013

Yeah, in case dropping XP and Vista support is unacceptable, @cgohlke's solutions are here: http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065357.html

It's not pretty, but it's hardly DLL-hell. DLL-minor-purgatory, maybe. The current mingw3-based system is what's really hellish...

@cournape
SciPy member

it is not just about being non pretty, it causes lots of problems in non-standard installation, or embedding. I actually prefer DLL-hell to that solution.

@njsmith
@josef-pkt
SciPy member
@cournape
SciPy member

@njsmith the pb with non standard installation is not where to look for the file, but pre-loading. Some packages do that, and it causes hard to track bugs, because it changes the rules of which dll are looked first (or worse how symbols are resolved).

Putting the dll in non standard location without pre-loading is actually the ideal solution :)

@pv
SciPy member
pv commented Sep 16, 2013

As Josef notes above, Mingw >= 4.7.1 have -static-libgcc and -static-libstdc++ flags which may be helpful.
Not sure if there is -static-libgfortran

EDIT: -static-libgfortran apparently seems to be there. We also don't necessarily need C++ exceptions.

@cournape
SciPy member

@josef-pkt I believe neither the C++ exception support (the *sljl.dll stuff) or the fortran runtimes can be statically linked. So while numpy may be fine, scipy will not. Last time I tried to force static linking, it crashed horribly at import times for scipy, but I have not tried for a while now.

@njsmith
@josef-pkt
SciPy member

From a search on my Windows computer, octave and several other programs have the gcc dlls locally.
R does not, some information I found http://www.stat.yale.edu/~jay/Admin3.3.pdf which looks like they do only static linking (there are Lapack.dll and others in the R bin directories, but no gcc/MingW related dlls AFAICS)

@carlkl
carlkl commented Sep 17, 2013

using recent mingw-w64 toolchains, i.e. mingw-builds "http://sourceforge.net/projects/mingwbuilds/" statically linking is possible with the following flags at the linking stage:

"-static" "-static-libgcc" "-static-libstdc++" "-static-libgfortran" depending on wether you use Fortran, C++, C. Result is that there is no dependency on any gcc DLLs. I'm not sure if "-static-libgfortran" has an effect. From my experience gfortran linking with "-static" "-static-libgcc" is sufficient.

Another possibility is using a toolchain with shared linking disabled , i.e. from "equation.com" or TDM's mingw toolchain. I personally prefer a toolchain with a comprehensible build process and patches, i.e. mingw-build (see "https://github.com/niXman/mingw-builds" ).
.
The resulting binaries and pyd libraries should be compressed with: "strip --strip-unneeded extension.pyd" as the mingw gnu linker is not very good stripping out not referenced object code. The only disadvantage for heavyly exception based C++ code (QT, Boost), that exceptions can not propagated out of DLL boundaries. Only TDM's mingw toolchain has some patches to get around this (with the disadvantge of ABI incompatbility)

I never encountered problems with static linking python extensions with mingw-w64 (mostly gfortran). The overhead is minimal after the pyd files are stripped.

Some other details which should be considered:

1) Use of mingw or mingw-w64 toolchain:
Today only mingw-w64 supports 32bit as well as 64bit. Linux cross compilers are based on mingw-w64 since quite some time.
There is a legal approval from the Red Hat lawyers: http://lists.fedoraproject.org/pipermail/mingw/2012-February/004533.html
"the mingw-w64 toolchain is no longer subject to any known legal blocks"

2) mingw-w64 distribution:
"mingw-builds" is the distribution of choice for many OSS developers. For the good and the bad you can get a lot of variants the mingw-w64 toolchain. See a discussion on: "http://qt-project.org/wiki/MinGW-64-bit"

3) Exception model:
32 bit: use sjlj exception model. use dwarf only for compiling QT or boost. For use with C and gfortran this option is not relevant.
64 bit: use the SEH exception model

4) posix or win32 threads:
use the win32 option; the posix option is for C++11 compatibility only.

5) major gcc version:
this is of course up to the user.

There is much more to say about mingw-w64 usage:

  • Use of the winpthread library in favour of pthreads-win32
  • Use of openBLAS instead of Atlas or MKL
  • Linking MSVCRT90 (MSVCRT100) instead of MSVCRT
  • creating Manifest files
@pv
SciPy member
pv commented Sep 17, 2013

Earlier, there were also some legal concerns with mingw-w64. These seem to have been resolved: https://lists.fedoraproject.org/pipermail/mingw/2012-February/004533.html

EDIT: ok, I see you noted this too...

@cournape
SciPy member

I have started looking into this yesterday night: I intend to build openblas with mingw+msys, and then look into numpy + scipy with the static options.

@carlkl
carlkl commented Sep 17, 2013

BTW: openblas has some binaries on sourceforge: "http://sourceforge.net/projects/openblas/files/v0.2.8/"
AFAIK this has been build with a mingw-w64 based gcc-4.7 cross compiler (win32 threads based), see:
https://groups.google.com/forum/#!topic/openblas-users/bGwJJWuvbdo
Blas, Lapack is contained in a huge static library. Also a dll with a MSVC compatible .lib file is provided.

mingw-w64 itself is on the way to make a new stable release (if I understand it correctly):
[Mingw-w64-public] Mass rebuild report for August 29 2013: http://sourceforge.net/mailarchive/forum.php?thread_name=521FAA0A.6070300%40redhat.com&forum_name=mingw-w64-public
[Mingw-w64-public] mingw-w64 v3 release calling for testers: http://sourceforge.net/mailarchive/forum.php?thread_name=523726C7.6040506%40codeweavers.com&forum_name=mingw-w64-public

I intend to build a fully statically gcc and maybe publish the binaries somewhere in near future. Things to include are missing libmoldnameXX.a libraries for msvcr80..100 versions as well as different spec files for linking with different msvcr runtimes.

Another hint is important for working with 64bit Windows and 64bit gcc: instead of msys it is recommended to use msys2 http://sourceforge.net/projects/msys2/files/Alpha-versions to get around the SysWOW64 problem http://en.wikipedia.org/wiki/WoW64 .

@josef-pkt
SciPy member

a piece of information from the python list (picked up by accident, in case it's not known)
--mno-cygwin is still in distutils but not accepted anymore by MingW since gcc 4.6
http://comments.gmane.org/gmane.comp.gnu.mingw.user/42092
http://stackoverflow.com/questions/6034390/compiling-with-cython-and-mingw-produces-gcc-error-unrecognized-command-line-o

@Tillsten

@josef-pkt only in 2.7, irc?

@josef-pkt
SciPy member

The patches are for all python, it looks like
http://bugs.python.org/issue12641

@Tillsten

You are right, i only remembered that they reject the patch because it was for python 2.7 and it was deemed as a non crtical patch, sorry.

@rgommers
SciPy member

Bumping milestone to 0.15.0

@rgommers rgommers modified the milestone: 0.15.0, 0.14.0 Feb 25, 2014
@sturlamolden

We can use MSVC for C and C++. Let us focus on gfortran and Fortran.

Sharing C++ exceptions between Python extension modules is a bad idea, even if we do not use g++, and I cannot imagine that SciPy does that. However, it is of no consequence for Fortran. What we need for f2py is gfortran and gcc.

-mno-cygwin is gone in Python 2.7.6, 3.3 and 3.4.

@Juanlu001 Juanlu001 referenced this issue in CamDavidsonPilon/lifelines Jun 3, 2014
Merged

Add concordance index function #57

@rgommers rgommers modified the milestone: 0.16.0, 0.15.0 Sep 16, 2014
@rgommers rgommers referenced this issue in rgommers/scipy Dec 28, 2014
Closed

There is a new ODRPACK #9

@rgommers rgommers modified the milestone: 0.17.0, 0.16.0 May 5, 2015
@rgommers
SciPy member
rgommers commented May 5, 2015

bumped to 0.17.0

@sturlamolden

This solves most of the problems discussed here:

https://github.com/numpy/numpy/wiki/Mingw-static-toolchain

What are the remaining problems?

@njsmith
@sturlamolden

So it solves the problems mentioned here but introduces a set of new ones... Great. :disappointed:

@carlkl
carlkl commented May 11, 2015

The situation is not that bad. The features of the toolchain are summarized here: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingwpy-2015-04-readme.pdf. A lot of long standing problems are solved with this adapted mingw-w64 right now.

What is the situation for mingw-w64 based numpy and scipy?

Remaining numpy and scipy failures are descriped in #4528 (comment). The OpenBLAS related errors are gone now during OpenBLAS development. Also OpenBLAS dgemv performance problem (xianyi/OpenBLAS#532) for smaller sizes is hopefully solved now. (the latter is relevant for typical theano cpu usage)

Lost of precision failures in scipy and NaN/Inf mismatch numpy/numpy#5194 (comment) is something that has to worked out in the math code of mingw-w64. The questions is, are the latter failures a show stopper to release openblas based numpy and scipy wheels? I made wheels for testing available at binstar: https://binstar.org/carlkl.

The easiest way right now to test the toolchain for extensions is to install the portable winpython distribution at https://winpython.github.io as it is configured to use mingw-w46 as default.

I try right now to reduce the number of hacks necessary and to put all changes into forked projects at https://github.com/mingwpy. This is not released yet due to lack of spare time.

@matthew-brett

Carl - it sounds as though now might be a good time to collect some effort to help you with the details here? How about a Google hangout in the next few days? We have a sprint on Wednesday in Berkeley, where we could devote some time to the problem. Cc'ing @ogrisel ...

@carlkl
carlkl commented May 12, 2015

wednesday or thursday next week?

@matthew-brett

Sorry - bad timing - but I am going to Cuba on Tuesday next week, to two weeks, and it's really impossible to do any internet voice thing from there.

@hiccup7
hiccup7 commented May 13, 2015

@carlkl , Does this issue also apply to SciPy & NumPy with OpenBLAS?
JuliaLang/julia#11258

@sturlamolden

"For an Intel CPU with hyperthreading, better OpenBLAS performance would occur by using the number of physical cores instead of logical cores. For example, blas_set_num_threads(CPU_CORES >> 1) For portability, I suggest Julia includes a new constant: CPU_PHYSICAL_CORES."

This is not even correct. If you do this, a thread can be scheduled to one of the idle logical cores. Setting the number of threads to the number of logical cores is what gives the best performance (given there is enough work). MKL uses the number of logical cores as well.

@sturlamolden

See here what happens when I go from 4 threads (number of physical cores) to 8 threads (number of logical cores):

http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21-simd.ipynb

Why? Because if I only have half as many threads as logical cores, each thread has a 50 % chance of being scheduled to an idle core.

@njsmith
@rgommers rgommers modified the milestone: 0.18.0, 0.17.0 Nov 22, 2015
@stonebig

Link to the "funding request" on the mingwpy subject: https://mingwpy.github.io/proposal_december2015.html

Is it still true that this problem has an impact on "odeint / sundials" progress ? #2818

@Juanlu001

Thanks for linking, I was not aware of its existence. Please keep us posted about NumFOCUS decision :)

@rgommers
SciPy member

@Juanlu001 the full proposal was just accepted. I just sent a PR to update http://mingwpy.github.io/roadmap.html with a sentence saying that.

@stonebig

wunderbar !

@Juanlu001

Congratulations! :tada:

@rgommers
SciPy member

Is it still true that this problem has an impact on "odeint / sundials" progress ? #2818

I was starting to type "yes, because ..." but then realized that we did drop the numpy-vendor (mingw32) based toolchain a few weeks ago. And that was the issue - we relied on g77 for Windows binaries. So even before we start using MingwPy for Windows wheels on PyPi (i.e. right now), we can start using Fortran 90/95 code I think.

Does anyone see a problem with that?

@njsmith
njsmith commented Jan 17, 2016

we can start using Fortran 90/95 code I think.

So the logic here is that for now, ifort will be the only supported fortran compiler for building scipy on windows, and eventually (if everything goes well) gfortran will also be supported? That sounds plausible, since I'm pretty sure @cgohlke and Anaconda (@ilanschnell?) are already using ifort, and WinPython IIUC is already using Carl's gfortran builds (right @stonebig?). I'm not as certain about Canopy (@cournape) or Python(x,y) (@mindw, @yuvalj) -- anyone want to object?

@mindw
mindw commented Jan 17, 2016

Python(x,y) uses uses ifort.

@rgommers
SciPy member

So the logic here is that for now, ifort will be the only supported fortran compiler for building scipy on windows, and eventually (if everything goes well) gfortran will also be supported?

Pretty much, but minor correct: gfortran is already supported, in the sense that it can build scipy just fine. Only distributing binaries is problematic, hence we don't do that and everybody else uses ifort to distribute their own builds.

@ilanschnell

Yes, we use ifort on Windows to compile scipy for Anaconda.

@stonebig

Currently, WinPython is mainly prepared with @cgohlke wheels for the scipy stack, as:

  • MKL is king of the numeric performance on Intel,
  • Scipy is very sensitive to any non-purity in its compilation/alignment options.

This bus factor of 1 is an uncomfortable blue pill.

@carlkl
carlkl commented Jan 18, 2016

It's true that scipy compiles with mingwpy/gfortran on the windows platform without compile or link errors. The prerequisite is a mingwpy compiled numpy build. Hence an officially scipy build compiled with mingwpy/gfortran will be available with the advent of a rigorously tested numpy based on mingwpy and OpenBLAS. The necessary code changes for numpy has been withdrawn due to some side effects some weeks ago. This numpy customization has to be corrected and resubmitted to the numpy repository. This is an essential part of the recently confirmed mingwpy proposal.

One nice feature of scipy-1.6.x is the Cython api for LAPACK and BLAS: #4021. It means in effect that a python package depending on specific BLAS/LAPACK features can be developed with the help of the Scipy Cython API without becoming dependant on the specific BLAS implementation of numpy. The hope is, that much less packgages with a dedicated numpy+MKL dependancy are around.

BTW the latests numpy, scipy test builds (based on local patches and mingpwy, OpenBLAS) are about 4 months old and can be found here:

Here: https://anaconda.org/carlkl/all are some more and somewhat older variants. numpy-1.9.3 and scipy-1.6.1 and newer releases are in the pipeline.

For the existings builds there are some test failures

  • FAIL: test_optimize.TestOptimize.test_minimize
  • FAIL: test_optimize.TestOptimize.test_powell
  • FAIL: test_ltisys.Test_bode.test_06
  • FAIL: test_ltisys.Test_freqresp.test_pole_zero

this happens only on 32 bit

  • FAIL: test_decomp_update.TestQRupdate_F.test_Mx1_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_Mx1_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_neg_strides_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_neg_strides_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_itemsize_strides_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_itemsize_strides_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_native_byte_order_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_native_byte_order_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_unit_strides_economic_rank_1
  • FAIL: test_decomp_update.TestQRupdate_F.test_non_unit_strides_economic_rank_p
  • FAIL: test_decomp_update.TestQRupdate_F.test_overwrite_qruv_rank_1_economic

this happens on python-3.4.amd64

  • ERROR: test_fortran.test_fortranfiles_read
  • ERROR: test_fortran.test_fortranfiles_write
    • OSError: Sizes do not agree in the header and footer for this record - check header dtype

this errors pops up because there is an additional byte at the end of the footer

This picture may change with future builds of numpy scipy

@mrslezak
mrslezak commented Feb 9, 2016

Not sure if this is in the right discussion, but I saw Anaconda people speaking of compiling with ifort.exe. I can compile NumPy with MKL or OpenBLAS no problem w/ MSVC14, but on SciPy I get an error "no fortan compiler found" about halfway through the build. Obviously ifort is available. Do you modify the build files at all in SciPy to get it to compile? Thanks.

@cournape
SciPy member
cournape commented Feb 9, 2016

Building scipy with MSVC and Intel Fortran compiler is supported. Just be sure to pass --fcompiler=intelvem (64 bits) to the build_ commands. If that does not work, please send a message to the scipy ML, as this is off-topic for this issue.

@gfyoung
gfyoung commented Feb 14, 2016

@mrslezak : You need to have ifort.exe in your PATH. In addition, make sure that the appropriate environment is set up for your build by running psxevars.bat <arch> before doing so.

@cournape : It may actually not be necessary to pass in the fcompiler arg if you have your environment set up appropriately. I can't say for sure since I can't test it, but I don't have to pass in any fcompiler arg on 32-bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.