Mingw Gfortran on Windows?

[pv]

So far we've kept to Fortran 77 due to needing g77 on Windows.

However, it seems to me Mingw32 gfortran-4.8.1 can compile a working Scipy, so perhaps it could be used for the next release? This would also allow including Fortran 90 code.

[matthew-brett]

That would be really good.

[josef-pkt]

What Blas/Lapack version can we use?
the old binaries in numpy/vendor are for MinGW 3.14, gcc 3.4 + g77

That would be really good. to echo Matthew
and I would be able to build scipy again

[rgommers]

I can't find @cournape's description of the issue right now, but the problem is what to do with the binaries. Apparently with MinGW 4.x we need extra dll's and have nowhere to install them in such a way that they can be found.

[cournape]

@rgommers that's correct. There are only bad solutions:

• keep the current situation with out of date compilers
• update to mingw 4.x with the corresponding Dll Hell.

Details: there is unfortunately no way to statically link the required runtimes on gcc 4.x, and short of putting the mingw dll in C:\Python_, I don't see any solution. Putting the dll in C:\Python_ is really horrible, though, and prone to all kind of dll-hell situations.

I have an idea that could fix this long term (requires changes to python itself), but that won't fix anything until we can give up the current python...

[astrojuanlu]

Let me understand this, is it a problem with the recent versions of MinGW, gfortran specifically, distutils, Python itself, this charming operative system, everything at the same time?

[cournape]

@Juanlu001 It is a problem related to recent versions of mingw + windows-specific issues, in the sense that every other system has a sane way to ship "private" shared libraries.

[matthew-brett]

Maybe this discussion should go to the {numpy or scipy} mailing list with the options laid out? Am I right in thinking this issue is now blocking us from using fortran 90 in scipy and providing Windows 64 bit builds for numpy?

[matthew-brett]

Previous discussion on this thread : http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065339.html

[astrojuanlu]

For future references, this is the description @cournape gave on the mingw + windows issue, in the thread @matthew-brett linked:

http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065349.html

According to the documentation, g77 cannot compile Fortran 90 (the new features not available in FORTRAN 77)

http://gcc.gnu.org/onlinedocs/gcc-3.4.6/g77/Fortran-90-Support.html

So yes, this is blocking us from using Fortran 90 (see #2818).

[pv]

I would perhaps just dive for the DLL-hell solution. At least build an example installer and put it available with the beta release and see how many people manage to blow a gasket.

Of course, this may be hitting not-so-nicely the downstream guys like Anaconda & EPD (but are they really also sticking with G77)? The package list on http://scipy.org/install.html should be checked.

[cournape]

It does not matter for us (Enthought) at least, since we're not using mingw at all for our own packages. Someone from Continuum should answer, but I suspect that's not a problem for them either.

[rgommers]

@ilanschnell do you care about g77 support for Anaconda?

[rgommers]

If we drop support for Windows XP and Vista for the binaries we provide ourselves, then we can go to MinGW 4.x without DLL hell, correct? That may the best option - XP support has ended back in 2009 and extended (paid) support is also ending in H1 2014. And no one runs Vista, right?

[pv]

There was also the suggestion from Christoph Gohlke to preload the DLLs from a custom location using ctypes. If that works (he says he used it), it would seem like a portable solution.

[ilanschnell]

We also do not use mingw to compile the packages that go into Anaconda. So no, g77 is not an issue.

[njsmith]

Yeah, in case dropping XP and Vista support is unacceptable, @cgohlke's solutions are here: http://mail.scipy.org/pipermail/numpy-discussion/2013-February/065357.html

It's not pretty, but it's hardly DLL-hell. DLL-minor-purgatory, maybe. The current mingw3-based system is what's really hellish...

[cournape]

it is not just about being non pretty, it causes lots of problems in non-standard installation, or embedding. I actually prefer DLL-hell to that solution.

[njsmith]

Can you give an example of such a problem?

Installation in a non-standard location seems like a non-issue to me -- if
Python can find the full path to numpy/core/multiarray.dll, then
numpy.init can find the full path to numpy/core/extra_dlls/. (Generally
just dirname(path) + "/core/extra_dlls/", right?)

The way embedding tools handle .dll's like multiarray.dll is that they just
pack them into the binary and then uncompress them into a temp dir at run
time, right? So they could just do the same with the runtime dll's? It's no
more problematic than trying to embed something like lxml, and much easier
than embedding something like wxwidgets, qt, or matplotlib, and people
manage those all the time.

I might be missing something...

On Mon, Sep 16, 2013 at 11:15 AM, David Cournapeau <notifications@github.com

wrote:

it is not just about being non pretty, it causes lots of problems in
non-standard installation, or embedding. I actually prefer DLL-hell to that
solution.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2829#issuecomment-24500358
.

[josef-pkt]

Which MingW dlls are still necessary?
My impression from a google search is that gcclib (and maybe others) can
now be statically linked in recent versions of MingW.

Josef

[cournape]

@njsmith the pb with non standard installation is not where to look for the file, but pre-loading. Some packages do that, and it causes hard to track bugs, because it changes the rules of which dll are looked first (or worse how symbols are resolved).

Putting the dll in non standard location without pre-loading is actually the ideal solution :)

[pv]

As Josef notes above, Mingw >= 4.7.1 have -static-libgcc and -static-libstdc++ flags which may be helpful.
Not sure if there is -static-libgfortran

EDIT: -static-libgfortran apparently seems to be there. We also don't necessarily need C++ exceptions.

[cournape]

@josef-pkt I believe neither the C++ exception support (the *sljl.dll stuff) or the fortran runtimes can be statically linked. So while numpy may be fine, scipy will not. Last time I tried to force static linking, it crashed horribly at import times for scipy, but I have not tried for a while now.

[njsmith]

From
http://sourceforge.net/p/mingw-w64/discussion/723798/thread/c2c28bd7/#8eff:
"If you link a final executable with "-static" and the toolchain was built
with -enable-static, you get zero dependency on libgcc, libstdc++, and
libgfortran dll's, and instead have the code linked in statically." It's
obnoxiously difficult to find solid information on these flags, but my
impression from reading around is that the C++ exception stuff can be
statically linked, at the cost of not supporting cross-DLL exception
throws. I don't know if that's a problem for us or not. Might be worth a
try anyway to see if things work with new MinGW. If so then great, and if
not then it still seems like we have reasonable options.

@cournape: To be clear when I'm referring to Cristoph's solutions I mean
either the preloading or the PATH manipulation, either way. Both seem
better than nothing to me -- is swapping the order that libgcc and
multiarray get loaded really going to trigger a bunch of symbol conflicts
in practice? -- but maybe PATH manipulation is the better option of the
two? We could do even set PATH to point to our extra DLLs at the top of
numpy.init/scipy.init, and then restore it at the bottom.

On Mon, Sep 16, 2013 at 1:11 PM, David Cournapeau
notifications@github.comwrote:

@josef-pkt https://github.com/josef-pkt I believe neither the C++
exception support (the *sljl.dll stuff) or the fortran runtimes can be
statically linked. So while numpy may be fine, scipy will not. Last time I
tried to force static linking, it crashed horribly at import times for
scipy, but I have not tried for a while now.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2829#issuecomment-24505328
.

[josef-pkt]

From a search on my Windows computer, octave and several other programs have the gcc dlls locally.
R does not, some information I found http://www.stat.yale.edu/~jay/Admin3.3.pdf which looks like they do only static linking (there are Lapack.dll and others in the R bin directories, but no gcc/MingW related dlls AFAICS)

[carlkl]

using recent mingw-w64 toolchains, i.e. mingw-builds "http://sourceforge.net/projects/mingwbuilds/" statically linking is possible with the following flags at the linking stage:

"-static" "-static-libgcc" "-static-libstdc++" "-static-libgfortran" depending on wether you use Fortran, C++, C. Result is that there is no dependency on any gcc DLLs. I'm not sure if "-static-libgfortran" has an effect. From my experience gfortran linking with "-static" "-static-libgcc" is sufficient.

Another possibility is using a toolchain with shared linking disabled , i.e. from "equation.com" or TDM's mingw toolchain. I personally prefer a toolchain with a comprehensible build process and patches, i.e. mingw-build (see "https://github.com/niXman/mingw-builds" ).
.
The resulting binaries and pyd libraries should be compressed with: "strip --strip-unneeded extension.pyd" as the mingw gnu linker is not very good stripping out not referenced object code. The only disadvantage for heavyly exception based C++ code (QT, Boost), that exceptions can not propagated out of DLL boundaries. Only TDM's mingw toolchain has some patches to get around this (with the disadvantge of ABI incompatbility)

I never encountered problems with static linking python extensions with mingw-w64 (mostly gfortran). The overhead is minimal after the pyd files are stripped.

Some other details which should be considered:

1. Use of mingw or mingw-w64 toolchain:
   Today only mingw-w64 supports 32bit as well as 64bit. Linux cross compilers are based on mingw-w64 since quite some time.
   There is a legal approval from the Red Hat lawyers: http://lists.fedoraproject.org/pipermail/mingw/2012-February/004533.html
   "the mingw-w64 toolchain is no longer subject to any known legal blocks"

2. mingw-w64 distribution:
   "mingw-builds" is the distribution of choice for many OSS developers. For the good and the bad you can get a lot of variants the mingw-w64 toolchain. See a discussion on: "http://qt-project.org/wiki/MinGW-64-bit"

3. Exception model:
   32 bit: use sjlj exception model. use dwarf only for compiling QT or boost. For use with C and gfortran this option is not relevant.
   64 bit: use the SEH exception model

4. posix or win32 threads:
   use the win32 option; the posix option is for C++11 compatibility only.

5. major gcc version:
   this is of course up to the user.

There is much more to say about mingw-w64 usage:

• Use of the winpthread library in favour of pthreads-win32
• Use of openBLAS instead of Atlas or MKL
• Linking MSVCRT90 (MSVCRT100) instead of MSVCRT
• creating Manifest files

[pv]

Earlier, there were also some legal concerns with mingw-w64. These seem to have been resolved: https://lists.fedoraproject.org/pipermail/mingw/2012-February/004533.html

EDIT: ok, I see you noted this too...

[cournape]

I have started looking into this yesterday night: I intend to build openblas with mingw+msys, and then look into numpy + scipy with the static options.

[carlkl]

BTW: openblas has some binaries on sourceforge: "http://sourceforge.net/projects/openblas/files/v0.2.8/"
AFAIK this has been build with a mingw-w64 based gcc-4.7 cross compiler (win32 threads based), see:
https://groups.google.com/forum/#!topic/openblas-users/bGwJJWuvbdo
Blas, Lapack is contained in a huge static library. Also a dll with a MSVC compatible .lib file is provided.

mingw-w64 itself is on the way to make a new stable release (if I understand it correctly):
[Mingw-w64-public] Mass rebuild report for August 29 2013: http://sourceforge.net/mailarchive/forum.php?thread_name=521FAA0A.6070300%40redhat.com&forum_name=mingw-w64-public
[Mingw-w64-public] mingw-w64 v3 release calling for testers: http://sourceforge.net/mailarchive/forum.php?thread_name=523726C7.6040506%40codeweavers.com&forum_name=mingw-w64-public

I intend to build a fully statically gcc and maybe publish the binaries somewhere in near future. Things to include are missing libmoldnameXX.a libraries for msvcr80..100 versions as well as different spec files for linking with different msvcr runtimes.

Another hint is important for working with 64bit Windows and 64bit gcc: instead of msys it is recommended to use msys2 http://sourceforge.net/projects/msys2/files/Alpha-versions to get around the SysWOW64 problem http://en.wikipedia.org/wiki/WoW64 .

[sturlamolden]

See here what happens when I go from 4 threads (number of physical cores) to 8 threads (number of logical cores):

http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21-simd.ipynb

Why? Because if I only have half as many threads as logical cores, each thread has a 50 % chance of being scheduled to an idle core.

[njsmith]

Is your claim that if you have 2 threads, 2 physical cores, and 4 total
logical cores (hyperthreads), that this is bad because there's a 50% chance
that both threads will be scheduled on the same physical core while leaving
the other one idle? Because that is not true at all, e.g. I know for a fact
that the Linux scheduler in this case will be sure to assign the two
threads to different physical cores whenever possible.

Determining the fastest number of threads for a given workload on modern
CPUs is a very complex matter, which depends on the cache miss rate, the
size of the shared and unshared caches between different execution units,
the extent to which different threads have overlapping working sets, etc.
Sometimes you are even best off doing things like running one thread per
socket (because multiple cores in the same physical package generally
share -- and can end up fighting over -- L3 cache).

Anyway, this is all 100% a topic for the OpenBLAS bug tracker, not numpy or
Julia's...
On May 13, 2015 12:57 PM, "Sturla Molden" notifications@github.com wrote:

See here what happens when we go from 4 threads (number of physical cores)
to 8 threads (number of logical cores):

http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21-simd.ipynb

Why? Because if you only have half as many threads as logical cores, each
thread has a 50 % chance of being scheduled to an idle core.

—
Reply to this email directly or view it on GitHub
#2829 (comment).

[stonebig]

Link to the "funding request" on the mingwpy subject: https://mingwpy.github.io/proposal_december2015.html

Is it still true that this problem has an impact on "odeint / sundials" progress ? #2818

[astrojuanlu]

Thanks for linking, I was not aware of its existence. Please keep us posted about NumFOCUS decision :)

[rgommers]

@Juanlu001 the full proposal was just accepted. I just sent a PR to update http://mingwpy.github.io/roadmap.html with a sentence saying that.

[stonebig]

wunderbar !

[astrojuanlu]

Congratulations! 🎉

[rgommers]

Is it still true that this problem has an impact on "odeint / sundials" progress ? #2818

I was starting to type "yes, because ..." but then realized that we did drop the numpy-vendor (mingw32) based toolchain a few weeks ago. And that was the issue - we relied on g77 for Windows binaries. So even before we start using MingwPy for Windows wheels on PyPi (i.e. right now), we can start using Fortran 90/95 code I think.

Does anyone see a problem with that?

[njsmith]

we can start using Fortran 90/95 code I think.

So the logic here is that for now, ifort will be the only supported fortran compiler for building scipy on windows, and eventually (if everything goes well) gfortran will also be supported? That sounds plausible, since I'm pretty sure @cgohlke and Anaconda (@ilanschnell?) are already using ifort, and WinPython IIUC is already using Carl's gfortran builds (right @stonebig?). I'm not as certain about Canopy (@cournape) or Python(x,y) (@mindw, @yuvalj) -- anyone want to object?

[mindw]

Python(x,y) uses uses ifort.

[rgommers]

So the logic here is that for now, ifort will be the only supported fortran compiler for building scipy on windows, and eventually (if everything goes well) gfortran will also be supported?

Pretty much, but minor correct: gfortran is already supported, in the sense that it can build scipy just fine. Only distributing binaries is problematic, hence we don't do that and everybody else uses ifort to distribute their own builds.

[ilanschnell]

Yes, we use ifort on Windows to compile scipy for Anaconda.

[stonebig]

Currently, WinPython is mainly prepared with @cgohlke wheels for the scipy stack, as:

• MKL is king of the numeric performance on Intel,
• Scipy is very sensitive to any non-purity in its compilation/alignment options.

This bus factor of 1 is an uncomfortable blue pill.

[carlkl]

It's true that scipy compiles with mingwpy/gfortran on the windows platform without compile or link errors. The prerequisite is a mingwpy compiled numpy build. Hence an officially scipy build compiled with mingwpy/gfortran will be available with the advent of a rigorously tested numpy based on mingwpy and OpenBLAS. The necessary code changes for numpy has been withdrawn due to some side effects some weeks ago. This numpy customization has to be corrected and resubmitted to the numpy repository. This is an essential part of the recently confirmed mingwpy proposal.

One nice feature of scipy-1.6.x is the Cython api for LAPACK and BLAS: #4021. It means in effect that a python package depending on specific BLAS/LAPACK features can be developed with the help of the Scipy Cython API without becoming dependant on the specific BLAS implementation of numpy. The hope is, that much less packgages with a dedicated numpy+MKL dependancy are around.

BTW the latests numpy, scipy test builds (based on local patches and mingpwy, OpenBLAS) are about 4 months old and can be found here:

• https://anaconda.org/mingwpy/numpy (1.9.2)
• https://anaconda.org/mingwpy/scipy (1.6.0)

Here: https://anaconda.org/carlkl/all are some more and somewhat older variants. numpy-1.9.3 and scipy-1.6.1 and newer releases are in the pipeline.

For the existings builds there are some test failures

• FAIL: test_optimize.TestOptimize.test_minimize
• FAIL: test_optimize.TestOptimize.test_powell
• FAIL: test_ltisys.Test_bode.test_06
• FAIL: test_ltisys.Test_freqresp.test_pole_zero

this happens only on 32 bit

• FAIL: test_decomp_update.TestQRupdate_F.test_Mx1_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_Mx1_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_neg_strides_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_neg_strides_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_non_itemsize_strides_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_non_itemsize_strides_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_non_native_byte_order_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_non_native_byte_order_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_non_unit_strides_economic_rank_1
• FAIL: test_decomp_update.TestQRupdate_F.test_non_unit_strides_economic_rank_p
• FAIL: test_decomp_update.TestQRupdate_F.test_overwrite_qruv_rank_1_economic

this happens on python-3.4.amd64

• ERROR: test_fortran.test_fortranfiles_read
• ERROR: test_fortran.test_fortranfiles_write
  • OSError: Sizes do not agree in the header and footer for this record - check header dtype

this errors pops up because there is an additional byte at the end of the footer

This picture may change with future builds of numpy scipy

[mrslezak]

Not sure if this is in the right discussion, but I saw Anaconda people speaking of compiling with ifort.exe. I can compile NumPy with MKL or OpenBLAS no problem w/ MSVC14, but on SciPy I get an error "no fortan compiler found" about halfway through the build. Obviously ifort is available. Do you modify the build files at all in SciPy to get it to compile? Thanks.

[cournape]

Building scipy with MSVC and Intel Fortran compiler is supported. Just be sure to pass --fcompiler=intelvem (64 bits) to the build_ commands. If that does not work, please send a message to the scipy ML, as this is off-topic for this issue.

[gfyoung]

@mrslezak : You need to have ifort.exe in your PATH. In addition, make sure that the appropriate environment is set up for your build by running psxevars.bat <arch> before doing so.

@cournape : It may actually not be necessary to pass in the fcompiler arg if you have your environment set up appropriately. I can't say for sure since I can't test it, but I don't have to pass in any fcompiler arg on 32-bit.

[rgommers]

Removed the 0.18.0 milestone - this is not tied to a particular release anymore. Once we can build binary wheels for Windows, we'll upload them for the latest existing releases.

The g77 vs. gfortran thing is resolved: g77 is not supported anymore. Fortran 90/95 can be used in Scipy.

[pv]

This issue is resolved (pending only merging the numpy distutils patch), closing.