Windows install: python3

[jdtournier]

Using the current installation instructions on a fresh MSYS2 install leads to the following error:

$ ./configure
/usr/bin/env: ‘python’: No such file or directory

But everything works fine if I install python as a standalone package first:

$ pacman -S python

But what I don't get is that the version numbers differ between the python and mingw-w64-x86_64-python3 packages:

python:

$ python --version
Python 3.4.3

mingw-w64-x86_64-python3:

$ python3 --version
Python 3.5.0

If I use the mingw-w64-x86_64-python3 version explicitly, configure fails:

$ python3 configure

MRtrix build type requested: release

Detecting OS: windows
Checking for C++11 compliant compiler [g++]: 5.4.0 - tested ok
Detecting pointer size: 64 bit
Detecting byte order: little-endian
Checking for variable-length array support: yes
Checking for non-POD variable-length array support: yes
Checking for zlib compression library: 1.2.8
Checking for Eigen 3 library:
ERROR:  error compiling Eigen application!

  MRtrix3 was unable to compile a test program involving Eigen3.

  Set the EIGEN_CFLAGS environment variable to inform 'configure' of
  the flags it must provide to the compiler in order to compile
  programs that use Eigen3 functionality; this may include the path to
  the Eigen3 include files, as well as any required flags.
  For example:
    $ export EIGEN_CFLAGS="-isystem /usr/include/eigen3"
    $./configure
  (amend with the actual path to the Eigen3 include files on your system)

But it works fine with the standard python version (or if invoked using ./configure). The error is a failure of pkg-config to find the eigen3 flags. Here's the relevant excerpt from configure.log:

REPORT: Checking for Eigen 3 library:
EXEC <<
CMD: pkg-config --cflags eigen3
EXIT: 1
STDERR:
Package eigen3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `eigen3.pc'
to the PKG_CONFIG_PATH environment variable
No package 'eigen3' found
>>

error running on pkg-config --cflags eigen3

But invoking that same command directly works fine (and obviously also works fine using the other version of python):

$  pkg-config --cflags eigen3
-I/mingw64/include/eigen3

I have no idea what's going on with this version of python...

---

So overall I'd be happy to replace mingw-w64-x86_64-python3 with python on the list of suggested packages, but the problem is that the scripts also rely on numpy, and the only version available is mingw-w64-x86_64-python3-numpy:

$ pacman -Qs numpy
local/mingw-w64-x86_64-python3-numpy 1.11.0-1
    Scientific tools for Python (mingw-w64)

which depends on mingw-w64-x86_64-python3:

$ pacman -Qi mingw-w64-x86_64-python3-numpy
Name            : mingw-w64-x86_64-python3-numpy
Version         : 1.11.0-1
Description     : Scientific tools for Python (mingw-w64)
Architecture    : any
URL             : http://www.numpy.org/
Licenses        : BSD
Groups          : None
Provides        : None
Depends On      : mingw-w64-x86_64-openblas  mingw-w64-x86_64-python3
Optional Deps   : mingw-w64-x86_64-python3-nose: testsuite
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 15.69 MiB
Packager        : Alexey Pavlov <alexpux@gmail.com>
Build Date      : Wed, Apr 6, 2016 10:30:16 AM
Install Date    : Tue, Jun 21, 2016 2:53:06 PM
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

So it looks like we're going to have to include both. It would make things simpler if we didn't rely on numpy... But really I just can't figure out why there are two versions of Python3 available, why the most recent one doesn't work, and whether this is just a temporary glitch in MSYS2 that will eventually come good with the next update. Maybe they're in the process of revamping their package management...?

---

In a similar vibe: there is also a gcc package available via MSYS2's pacman (provide GCC 5.3.0, as opposed to the 5.4.0 version provided by the ming64 version). I just had a go at running that instead, and this time it's the zlib stage that configure falls over on. I can fix that by installing the zlib-devel package, but then it falls over at the Qt5 configure stage... Error this time is:

CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -march=native -DMRTRIX_WORD64 -isystem /mingw64/include/eigen3 -pipe -fno-keep-inline-dllexport -march=nocona -mtune=core2 -Wa,-mbig-obj -O2 -std=gnu++0x -frtti -Wall -Wextra -fexceptions -mthreads -DUNICODE -DQT_NO_DEBUG -DQT_OPENGL_LIB -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_NEEDS_QMAIN -isystem /mingw64/include/QtOpenGL -isystem /mingw64/include/QtSvg -isystem /mingw64/include/QtWidgets -isystem /mingw64/include/QtGui -isystem /mingw64/include/QtCore -isystem release -isystem /mingw64/share/qt5/mkspecs/win32-g++ qt.cpp -o qt.o
EXIT: 1
STDERR:
In file included from /mingw64/include/QtCore/QObject:1:0,
                 from qt.h:1,
                 from qt.cpp:2:
/mingw64/include/QtCore/qobject.h:40:32: fatal error: QtCore/qobjectdefs.h: No such file or directory
compilation terminated.

If anyone has any idea why things are in such a mess, please enlighten me...

[Lestropie]

But what I don't get is that the version numbers differ between the python and mingw-w64-x86_64-python3 packages

The mingw-w64-x86_64-python2 and mingw-w64-x86_64-python3 packages are maintained and provided by MinGW64; python is from MSYS. Different versions that get installed to different locations. :-/

If I use the mingw-w64-x86_64-python3 version explicitly, configure fails:

I recall at some point in my frustrating Windows installation debugging that EIGEN_CFLAGS=-isystem /path/to/eigen3 didn't work, but EIGEN_CFLAGS=-I/path/to/eigen3 did. Don't recall the exact circumstances though.

So overall I'd be happy to replace mingw-w64-x86_64-python3 with python on the list of suggested packages, but the problem is that the scripts also rely on numpy, and the only version available is mingw-w64-x86_64-python3-numpy:

Yes, I hit that myself too. Removed the numpy include from dwiintensitynorm since it wasn't actually being used, and had just gotten used to population_template throwing an error during documentation generation. But that's maybe not a solution for everyone. See if the -I/path/to/eigen3 trick works, and if it does maybe we should recommend installing that version and have clear instructions in the documentation.

If anyone has any idea why things are in such a mess, please enlighten me...

Don't even get me started on trying to solve Qt issues on Windows 😨

[jdtournier]

OK, I think this makes some kind of sense... I'm assuming all these issues will sort themselves out in due course. But in the meantime, MSYS2 is in a bit of a mess. I don't know if any of you tried to update your installation recently (pacman -Syu), but all the shortcuts to start the various shells stop working. And this is using the latest installer. Currently our instructions will fail because of this. All you get when you update MSYS2 is a message right at the end telling you what commands your shortcuts should have in them - not exactly the most obvious way to present the information, or particularly easy to fix for your average user...

I'm not sure how to fix this in the immediate future. I guess once we have stable releases, we can provide sanitised and up to date versions of MSYS2? But this seems like a lot of work to fix something that really ought to just work...

[Lestropie]

Yes, their first update warnings are not great. It does say somewhere that that step shouldn't be required after some point in time. But really all it should need is a warning on our docs page saying to pay attention to the warnings that are provided during that step.

[jdtournier]

Not great is an understatement... I reinstalled MSYS2 about 3 times in a row before finally noticing that message. I knew about the whole 'close the window, don't return to shell' issue, but that message was actually quite a few lines previous to that bit, and very hard to notice if you're just going through the motions.

And besides, the message itself is going to completely stump most users. Even if you know you need to do something to fix the shortcuts, it's not obvious where to find the shortcuts in the first place (right-click on the icon in the start menu, click on 'more options', then 'open file location'), it's not obvious how to edit them (right-click and properties), and not obvious what to replace the commands with, since the instructions provided say to replace the commands with MSYSROOT\msys_shell.cmd ... or something like that - a straight copy/paste doesn't work, you need to replace MSYSROOT with whatever the appropriate folder is (typically C:\msys64 or something). That's a lot of stuff to figure out for your average Windows user... I don't understand why they don't provide a more up to date installer where all this is actually fixed up - no update_core script, and the correct shortcuts...

Anyway, enough ranting for now...

[jdtournier]

Sorry, the reason I'm mentioning this again is that while your recent commit (0b1c4fa) does mention that this needs to be dealt with, I don't see most users figuring out how to do this. Either we need to provide much more detailed instructions, or supply a fixed up version, or wait for the MSYS2 devs to fix it, or something else I can't think of right now. But it's going to be a real hurdle for most Windows users.

[Lestropie]

Yeah I agree. I wanted to do something while I thought of it, but I was timid about giving full details about modifying shortcuts because I wasn't sure how much would change between Windows versions. At the very least it will make people more aware of the issue as it occurs, and maybe have a go at figuring it out, bearing in mind that it's not actually an MRtrix problem so maybe they ask MSYS or a tech-savvy friend or something.

I don't think we should be providing a fixed version: between my Windows 7 and 8.1 machines, using the exact same installer gives a very different installation, so we might just cause even more problems trying to support the 'fixed' version across a lot of systems. I'd rather have a go at more detailed instructions, and redo an installation so I can grab text dumps of the actual update_core outputs and put them there. Hell, it could even be its own page.

[jdtournier]

Pretty much what I was thinking. One option would I think be to post some instructions on the community site, so at least we can direct people there if they struggle, without necessarily modifying the main docs to provide instructions that are likely to change from one day to the next... What do you reckon?