New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pkg-config to get fftw3 compiler flags #1090

Merged
merged 11 commits into from Aug 24, 2017

Conversation

Projects
None yet
6 participants
@bjeurissen
Member

bjeurissen commented Aug 13, 2017

No description provided.

@bjeurissen bjeurissen requested a review from jdtournier Aug 13, 2017

@bjeurissen

This comment has been minimized.

Show comment
Hide comment
@bjeurissen

bjeurissen Aug 19, 2017

Member

Tested on Linux, Mac and Windows. Should be good to go if someone approves it.

Member

bjeurissen commented Aug 19, 2017

Tested on Linux, Mac and Windows. Should be good to go if someone approves it.

@jdtournier

Seems good to me - go for it.

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 21, 2017

Member

Just out of interest: you say tested on Mac - did you not encounter the issues with Qt5 described in #1094?

Member

jdtournier commented Aug 21, 2017

Just out of interest: you say tested on Mac - did you not encounter the issues with Qt5 described in #1094?

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 21, 2017

Member

Only other thing is I'm not sure whether this should go out on master - it's not a bug fix per se? But then I guess it will become a bug fix as soon as someone tries to compile with a FFTW installation in a non-standard location...?

Member

jdtournier commented Aug 21, 2017

Only other thing is I'm not sure whether this should go out on master - it's not a bug fix per se? But then I guess it will become a bug fix as soon as someone tries to compile with a FFTW installation in a non-standard location...?

@bjeurissen

This comment has been minimized.

Show comment
Hide comment
@bjeurissen

bjeurissen Aug 21, 2017

Member

I did not encounter issues like #1094 on any of my macs. I am using macports though and not homebrew.

The reason I use macports and not homebrew is that homebrew tries to reuse as many of the "native" mac os x libraries as possible when resolving the dependencies. So if your mac comes with some library by default it will use that library to build other libraries. Macports on the other hand, will always use the libraries from its own repository, making it more self-consistent. In my experience the macports approach is less likely to cause library related problems.

Member

bjeurissen commented Aug 21, 2017

I did not encounter issues like #1094 on any of my macs. I am using macports though and not homebrew.

The reason I use macports and not homebrew is that homebrew tries to reuse as many of the "native" mac os x libraries as possible when resolving the dependencies. So if your mac comes with some library by default it will use that library to build other libraries. Macports on the other hand, will always use the libraries from its own repository, making it more self-consistent. In my experience the macports approach is less likely to cause library related problems.

@bjeurissen

This comment has been minimized.

Show comment
Hide comment
@bjeurissen

bjeurissen Aug 21, 2017

Member

The reason I created this "bugfix" was that the fftw option was not working on my mac, because it could not find the header.

Member

bjeurissen commented Aug 21, 2017

The reason I created this "bugfix" was that the fftw option was not working on my mac, because it could not find the header.

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 21, 2017

Member

OK, good to know. I guess this is worth pushing to master given that it is likely to turn out to be a relatively critical bug, at least for those users who do want to use FFTW. I'm happy enough for it to be merged direct to master. Everyone else happy with this?

Member

jdtournier commented Aug 21, 2017

OK, good to know. I guess this is worth pushing to master given that it is likely to turn out to be a relatively critical bug, at least for those users who do want to use FFTW. I'm happy enough for it to be merged direct to master. Everyone else happy with this?

@maxpietsch

This comment has been minimized.

Show comment
Hide comment
@maxpietsch

maxpietsch Aug 21, 2017

Member

All good to me. Unfortunately, this does not fix the linking issue related to homebrew's tiff libraries (#1094).

Member

maxpietsch commented Aug 21, 2017

All good to me. Unfortunately, this does not fix the linking issue related to homebrew's tiff libraries (#1094).

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 21, 2017

Member

Ok, just had a play with this with @dchristiaens - seems the issue with libTIFF was also down to not using pkg-config... I think this last commit should also resolve #1094 - @maxpietsch, can you confirm?

Member

jdtournier commented Aug 21, 2017

Ok, just had a play with this with @dchristiaens - seems the issue with libTIFF was also down to not using pkg-config... I think this last commit should also resolve #1094 - @maxpietsch, can you confirm?

@maxpietsch

This comment has been minimized.

Show comment
Hide comment
@maxpietsch

maxpietsch Aug 21, 2017

Member

This solves #1094 on my mac 👍

Member

maxpietsch commented Aug 21, 2017

This solves #1094 on my mac 👍

@Lestropie

This comment has been minimized.

Show comment
Hide comment
@Lestropie

Lestropie Aug 21, 2017

Member

Happy for it to go if it's resolved issues.

One thing that piqued my interest here (but isn't specific to these changes) is the forced conversion from "-I" to "-isystem" of the pkg-config output. On one of my Windows systems, "export EIGEN_CFLAGS=-I/usr/include/eigen3" works but "export EIGEN_CFLAGS=-isystem /usr/include/eigen3" does not. It also doesn't work out of the box without this explicit environment variable; I'm guessing that's because of this back-end conversion of the pkg-config output (which I hadn't noticed previously). So it might be worth having a look into this.

Member

Lestropie commented Aug 21, 2017

Happy for it to go if it's resolved issues.

One thing that piqued my interest here (but isn't specific to these changes) is the forced conversion from "-I" to "-isystem" of the pkg-config output. On one of my Windows systems, "export EIGEN_CFLAGS=-I/usr/include/eigen3" works but "export EIGEN_CFLAGS=-isystem /usr/include/eigen3" does not. It also doesn't work out of the box without this explicit environment variable; I'm guessing that's because of this back-end conversion of the pkg-config output (which I hadn't noticed previously). So it might be worth having a look into this.

docs installation update: mac_install.rst
add instructions for installing tiff and fftw libraries via homebrew or macports.
@chunhungyeh

This comment has been minimized.

Show comment
Hide comment
@chunhungyeh

chunhungyeh Aug 22, 2017

Member

Homebrew and Macports use different library names for TIFF and FFTW. So, I took this chance to update the installation docs.

(PS: As Macports has fftw and fftw-3, the library would not be found when running configuration if the former (which is an older version of fftw) is installed.)

Member

chunhungyeh commented Aug 22, 2017

Homebrew and Macports use different library names for TIFF and FFTW. So, I took this chance to update the installation docs.

(PS: As Macports has fftw and fftw-3, the library would not be found when running configuration if the former (which is an older version of fftw) is installed.)

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 22, 2017

Member

Thanks all for the feedback and other improvements.

@Lestropie: the replacement of -I with -isystem is not actually required. What it does is signal to the compiler that headers from this location are system headers over which you have no control, and to mute any warnings that originate from there. This helped a great deal to avoid a barrage of irrelevant warnings from some versions of Eigen3 and Qt. But I'd be happy to revert back if that's a problem. In many ways, it would be best to stick with exactly what pkg-config reports, and it might be that current versions of these packages don't produce warnings any more anyway...

Member

jdtournier commented Aug 22, 2017

Thanks all for the feedback and other improvements.

@Lestropie: the replacement of -I with -isystem is not actually required. What it does is signal to the compiler that headers from this location are system headers over which you have no control, and to mute any warnings that originate from there. This helped a great deal to avoid a barrage of irrelevant warnings from some versions of Eigen3 and Qt. But I'd be happy to revert back if that's a problem. In many ways, it would be best to stick with exactly what pkg-config reports, and it might be that current versions of these packages don't produce warnings any more anyway...

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 22, 2017

Member

OK, I had a go at removing the replacement of -I with -isystem - this doesn't produce any undesired warnings on my system. I've also taken the liberty of cleaning up configure somewhat, by providing convenience functions for tests that were very similar - see the commit diff for details. It all works fine on my system, but given that the changes are now a bit more extensive, I'd appreciate if everyone could have a quick check that everything is still working as it should on their systems...

I've also modified the zlib configuration to use pkg-config, by the way - seems like a cleaner solution...

Member

jdtournier commented Aug 22, 2017

OK, I had a go at removing the replacement of -I with -isystem - this doesn't produce any undesired warnings on my system. I've also taken the liberty of cleaning up configure somewhat, by providing convenience functions for tests that were very similar - see the commit diff for details. It all works fine on my system, but given that the changes are now a bit more extensive, I'd appreciate if everyone could have a quick check that everything is still working as it should on their systems...

I've also modified the zlib configuration to use pkg-config, by the way - seems like a cleaner solution...

@maxpietsch

This comment has been minimized.

Show comment
Hide comment
@maxpietsch

maxpietsch Aug 22, 2017

Member

Works on macOS with homebrew and on ubuntu 14.04. The latter produces tons of warnings for Eigen:

/usr/include/eigen3/Eigen/src/Core/products/SelfadjointMatrixVector.h:150:5: warning: 'register' storage class specifier is deprecated and incompatible with C++1z [-Wdeprecated-register]

Member

maxpietsch commented Aug 22, 2017

Works on macOS with homebrew and on ubuntu 14.04. The latter produces tons of warnings for Eigen:

/usr/include/eigen3/Eigen/src/Core/products/SelfadjointMatrixVector.h:150:5: warning: 'register' storage class specifier is deprecated and incompatible with C++1z [-Wdeprecated-register]

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 22, 2017

Member

The latter produces tons of warnings for Eigen

Yes, that's what I was worried about... I think I'd rather revert to using -isystem in this case, otherwise you guys are not going to be able to spot genuine warnings...

@Lestropie, any chance you could give a bit more info about the issues you're having on your Windows install? What shows up in configure.log? In particular, does the pkg-config step work OK?

Also, can I just double-check that you did put quotes in the proper place in your export line, like this:

 export EIGEN_CFLAGS=”-isystem /usr/include/eigen3”

There were no such quotes in your previous post, that would definitely not work...

Member

jdtournier commented Aug 22, 2017

The latter produces tons of warnings for Eigen

Yes, that's what I was worried about... I think I'd rather revert to using -isystem in this case, otherwise you guys are not going to be able to spot genuine warnings...

@Lestropie, any chance you could give a bit more info about the issues you're having on your Windows install? What shows up in configure.log? In particular, does the pkg-config step work OK?

Also, can I just double-check that you did put quotes in the proper place in your export line, like this:

 export EIGEN_CFLAGS=”-isystem /usr/include/eigen3”

There were no such quotes in your previous post, that would definitely not work...

@bjeurissen

This comment has been minimized.

Show comment
Hide comment
@bjeurissen

bjeurissen Aug 22, 2017

Member

For me -isystem works fine on Windows (using msys2)

Member

bjeurissen commented Aug 22, 2017

For me -isystem works fine on Windows (using msys2)

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 22, 2017

Member

For me -isystem works fine on Windows (using msys2)

Yes, it's always worked fine for me too. And this has been in place for quite some time (original commit is b075514, from Dec 2015). But @Lestropie says this only affects one of his Windows systems, so it's likely to be specific to that particular setup. It would be good to figure out why it doesn't work in this particularly instance, just to make sure this isn't going to affect others too...

@Lestropie: could the problem simply be that pkg-config is not installed...?

Member

jdtournier commented Aug 22, 2017

For me -isystem works fine on Windows (using msys2)

Yes, it's always worked fine for me too. And this has been in place for quite some time (original commit is b075514, from Dec 2015). But @Lestropie says this only affects one of his Windows systems, so it's likely to be specific to that particular setup. It would be good to figure out why it doesn't work in this particularly instance, just to make sure this isn't going to affect others too...

@Lestropie: could the problem simply be that pkg-config is not installed...?

@bjeurissen

This comment has been minimized.

Show comment
Hide comment
@bjeurissen

bjeurissen Aug 22, 2017

Member

Btw, Qt also supports pkg-config:

pkg-config --list-all | grep Qt

lists

Qt5Test               Qt5 Test - Qt Unit Testing Library
Qt5Network            Qt5 Network - Qt Network module
Qt5OpenGLExtensions   Qt5 OpenGLExtensions - Qt OpenGLExtensions module
Qt5Sql                Qt5 Sql - Qt Sql module
Qt5Widgets            Qt5 Widgets - Qt Widgets module
Qt5Core               Qt5 Core - Qt Core module
Qt5Xml                Qt5 Xml - Qt Xml module
Qt5Concurrent         Qt5 Concurrent - Qt Concurrent module
Qt5Gui                Qt5 Gui - Qt Gui module
Qt5DBus               Qt5 DBus - Qt DBus module
Qt5Svg                Qt5 Svg - Qt Svg module
Qt5OpenGL             Qt5 OpenGL - Qt OpenGL module
Qt5PrintSupport       Qt5 PrintSupport - Qt PrintSupport module

Could simplify configure but at the expense of making pkg-config almost indispensable for building the gui.

Member

bjeurissen commented Aug 22, 2017

Btw, Qt also supports pkg-config:

pkg-config --list-all | grep Qt

lists

Qt5Test               Qt5 Test - Qt Unit Testing Library
Qt5Network            Qt5 Network - Qt Network module
Qt5OpenGLExtensions   Qt5 OpenGLExtensions - Qt OpenGLExtensions module
Qt5Sql                Qt5 Sql - Qt Sql module
Qt5Widgets            Qt5 Widgets - Qt Widgets module
Qt5Core               Qt5 Core - Qt Core module
Qt5Xml                Qt5 Xml - Qt Xml module
Qt5Concurrent         Qt5 Concurrent - Qt Concurrent module
Qt5Gui                Qt5 Gui - Qt Gui module
Qt5DBus               Qt5 DBus - Qt DBus module
Qt5Svg                Qt5 Svg - Qt Svg module
Qt5OpenGL             Qt5 OpenGL - Qt OpenGL module
Qt5PrintSupport       Qt5 PrintSupport - Qt PrintSupport module

Could simplify configure but at the expense of making pkg-config almost indispensable for building the gui.

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 22, 2017

Member

Btw, Qt also supports pkg-config:

That is really tempting... I'd go for it if it wasn't for the need for the matching versions of rcc and moc to also be in the PATH. I think I'd rather stick with the current explicit qmake probe, since that's very likely to be the same version as the rcc & moc executables. Also, as you say, not relying on pkg-config allows a non-standard (manual) install of Qt to be used. I guess we'll stick with the current setup if it's not a problem currently.

Member

jdtournier commented Aug 22, 2017

Btw, Qt also supports pkg-config:

That is really tempting... I'd go for it if it wasn't for the need for the matching versions of rcc and moc to also be in the PATH. I think I'd rather stick with the current explicit qmake probe, since that's very likely to be the same version as the rcc & moc executables. Also, as you say, not relying on pkg-config allows a non-standard (manual) install of Qt to be used. I guess we'll stick with the current setup if it's not a problem currently.

@Lestropie

This comment has been minimized.

Show comment
Hide comment
@Lestropie

Lestropie Aug 22, 2017

Member

On one of my Windows systems, "export EIGEN_CFLAGS=-I/usr/include/eigen3" works but "export EIGEN_CFLAGS="-isystem /usr/include/eigen3"" does not.

Unable to reproduce. I think this was back with a manual install of the Eigen3.3 beta, whereas I now have it installed through pacman. Thought I tested it a bit at the time and had triple-checked the quotations. But if nobody else has encountered the same, forget it was raised.

Member

Lestropie commented Aug 22, 2017

On one of my Windows systems, "export EIGEN_CFLAGS=-I/usr/include/eigen3" works but "export EIGEN_CFLAGS="-isystem /usr/include/eigen3"" does not.

Unable to reproduce. I think this was back with a manual install of the Eigen3.3 beta, whereas I now have it installed through pacman. Thought I tested it a bit at the time and had triple-checked the quotations. But if nobody else has encountered the same, forget it was raised.

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 23, 2017

Member

@Lestropie: thanks for confirming. I guess this is good to go then, as soon as I get confirmation that my latest commit doesn't bork anything up on anyone's system... 😁

Member

jdtournier commented Aug 23, 2017

@Lestropie: thanks for confirming. I guess this is good to go then, as soon as I get confirmation that my latest commit doesn't bork anything up on anyone's system... 😁

@maxpietsch

This comment has been minimized.

Show comment
Hide comment
@maxpietsch

maxpietsch Aug 23, 2017

Member

All good on ubuntu 14.04 and macOS with homebrew.

Member

maxpietsch commented Aug 23, 2017

All good on ubuntu 14.04 and macOS with homebrew.

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 23, 2017

Member

Seems good on my Win10 laptop under MSYS2. Also works fine on my Arch Linux setup. Any other exotic setups that people can test on...?

Member

jdtournier commented Aug 23, 2017

Seems good on my Win10 laptop under MSYS2. Also works fine on my Arch Linux setup. Any other exotic setups that people can test on...?

@dchristiaens

This comment has been minimized.

Show comment
Hide comment
@dchristiaens

dchristiaens Aug 23, 2017

Member

On Ubuntu 16.04, I got fftw3 and libtiff support to work only after installing the *-dev versions of both packages. (Even though manually setting the path for (non-dev) FFTW had previously worked fine.) Any reason why pkg-config wouldn't pick up on non-dev system packages?

Member

dchristiaens commented Aug 23, 2017

On Ubuntu 16.04, I got fftw3 and libtiff support to work only after installing the *-dev versions of both packages. (Even though manually setting the path for (non-dev) FFTW had previously worked fine.) Any reason why pkg-config wouldn't pick up on non-dev system packages?

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Aug 23, 2017

Member

On Ubuntu 16.04, I got fftw3 and libtiff support to work only after installing the *-dev versions of both packages.

OK, would be worth amending the docs to mention this then - although it is an optional dependency at this point.

Any reason why pkg-config wouldn't pick up on non-dev system packages?

Yes: pkg-config basically reads the relevant settings from *.pc files distributed with the software. On Ubuntu, these files are contained in the corresponding *-dev package (e.g. fftw3-dev on 16.04) - which makes sense given that that's the whole point of the *-dev package.

(Even though manually setting the path for (non-dev) FFTW had previously worked fine.)

That does not make sense... The ffftw3.h header is also included in the fftw3-dev package - I don't see how you could have compiled against FFTW without it being installed already... Unless you were using a manually downloaded version of it before...?

Member

jdtournier commented Aug 23, 2017

On Ubuntu 16.04, I got fftw3 and libtiff support to work only after installing the *-dev versions of both packages.

OK, would be worth amending the docs to mention this then - although it is an optional dependency at this point.

Any reason why pkg-config wouldn't pick up on non-dev system packages?

Yes: pkg-config basically reads the relevant settings from *.pc files distributed with the software. On Ubuntu, these files are contained in the corresponding *-dev package (e.g. fftw3-dev on 16.04) - which makes sense given that that's the whole point of the *-dev package.

(Even though manually setting the path for (non-dev) FFTW had previously worked fine.)

That does not make sense... The ffftw3.h header is also included in the fftw3-dev package - I don't see how you could have compiled against FFTW without it being installed already... Unless you were using a manually downloaded version of it before...?

@dchristiaens

This comment has been minimized.

Show comment
Hide comment
@dchristiaens

dchristiaens Aug 23, 2017

Member

The fftw3.h header is also included in the fftw3-dev package - I don't see how you could have compiled against FFTW without it being installed already... Unless you were using a manually downloaded version of it before...?

That must have been the case then, even though I don't find it anywhere else on my system. Not important now, just need to document the *-dev requirement as you say.

Member

dchristiaens commented Aug 23, 2017

The fftw3.h header is also included in the fftw3-dev package - I don't see how you could have compiled against FFTW without it being installed already... Unless you were using a manually downloaded version of it before...?

That must have been the case then, even though I don't find it anywhere else on my system. Not important now, just need to document the *-dev requirement as you say.

@jdtournier

Guess I'll approve my changes then...

@jdtournier jdtournier merged commit b30ad75 into master Aug 24, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jdtournier jdtournier deleted the pkgconfig-fftw3 branch Aug 24, 2017

@Lestropie

This comment has been minimized.

Show comment
Hide comment
@Lestropie

Lestropie Sep 29, 2017

Member

Managed to reproduce the issue I mentioned earlier here.

TL;DR: Windows system doesn't work with EIGEN_CFLAGS="-isystem /usr/include/eigen3", but does work with EIGEN_CFLAGS="-I/usr/include/eigen3". pkg-config also appears to not work regardless of whether the mingw or msys version is installed.


 ./configure -verbose
EXEC <<
CMD: pkg-config --cflags eigen3
EXIT: 1
STDERR:
Package eigen3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `eigen3.pc'
to the PKG_CONFIG_PATH environment variable
No package 'eigen3' found
>>

error running "pkg-config --cflags eigen3"

Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpvokl7q.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -isystem /usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpvokl7q.cpp -o c:/users/rob/appdata/local/temp/tmpvokl7q.o
EXIT: 1
STDERR:
c:/users/rob/appdata/local/temp/tmpvokl7q.cpp:3:10: fatal error: Eigen/Core: No such file or directory
 #include <Eigen/Core>
          ^~~~~~~~~~~~
compilation terminated.
>>

error deleting temporary file "c:/users/rob/appdata/local/temp/tmpvokl7q.o": The system cannot find the file specified
ERROR: error compiling Eigen3 application!

    MRtrix3 was unable to compile a test program involving Eigen3.

  Set the EIGEN_CFLAGS environment variable to inform 'configure' of
  the flags it must provide to the compiler in order to compile
  programs that use Eigen3 functionality; this may include the path to
  the Eigen3 include files, as well as any required flags.
  For example:
    $ export EIGEN_CFLAGS="-isystem /usr/include/eigen3"
    $./configure
  (amend with the actual path to the Eigen3 include files on your system)

EIGEN_CFLAGS="-isystem /usr/include/eigen3" ./configure -verbose
Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpohsg5p.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -isystem /usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpohsg5p.cpp -o c:/users/rob/appdata/local/temp/tmpohsg5p.o
EXIT: 1
STDERR:
c:/users/rob/appdata/local/temp/tmpohsg5p.cpp:3:10: fatal error: Eigen/Core: No such file or directory
 #include <Eigen/Core>
          ^~~~~~~~~~~~
compilation terminated.
>>

error deleting temporary file "c:/users/rob/appdata/local/temp/tmpohsg5p.o": The system cannot find the file specified
ERROR: error compiling Eigen3 application!

    MRtrix3 was unable to compile a test program involving Eigen3.

  Set the EIGEN_CFLAGS environment variable to inform 'configure' of
  the flags it must provide to the compiler in order to compile
  programs that use Eigen3 functionality; this may include the path to
  the Eigen3 include files, as well as any required flags.
  For example:
    $ export EIGEN_CFLAGS="-isystem /usr/include/eigen3"
    $./configure
  (amend with the actual path to the Eigen3 include files on your system)

EIGEN_CFLAGS="-I/usr/include/eigen3" ./configure -verbose
Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpzwtkhd.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -IC:/msys64/usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpzwtkhd.cpp -o c:/users/rob/appdata/local/temp/tmpzwtkhd.o
EXIT: 0
STDERR:
( ... MASSIVE LIST OF COMPILER WARNINGS ... )
>>

EXEC <<
CMD: g++ c:/users/rob/appdata/local/temp/tmpzwtkhd.o -Wl,-O1,--sort-common,--as-needed -pthread -Wl,--allow-multiple-definition -o a.out
EXIT: 0
>>

EXEC <<
CMD: ./a.out
EXIT: 0
STDOUT:
3.3.1
>>

3.3.1
Member

Lestropie commented Sep 29, 2017

Managed to reproduce the issue I mentioned earlier here.

TL;DR: Windows system doesn't work with EIGEN_CFLAGS="-isystem /usr/include/eigen3", but does work with EIGEN_CFLAGS="-I/usr/include/eigen3". pkg-config also appears to not work regardless of whether the mingw or msys version is installed.


 ./configure -verbose
EXEC <<
CMD: pkg-config --cflags eigen3
EXIT: 1
STDERR:
Package eigen3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `eigen3.pc'
to the PKG_CONFIG_PATH environment variable
No package 'eigen3' found
>>

error running "pkg-config --cflags eigen3"

Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpvokl7q.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -isystem /usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpvokl7q.cpp -o c:/users/rob/appdata/local/temp/tmpvokl7q.o
EXIT: 1
STDERR:
c:/users/rob/appdata/local/temp/tmpvokl7q.cpp:3:10: fatal error: Eigen/Core: No such file or directory
 #include <Eigen/Core>
          ^~~~~~~~~~~~
compilation terminated.
>>

error deleting temporary file "c:/users/rob/appdata/local/temp/tmpvokl7q.o": The system cannot find the file specified
ERROR: error compiling Eigen3 application!

    MRtrix3 was unable to compile a test program involving Eigen3.

  Set the EIGEN_CFLAGS environment variable to inform 'configure' of
  the flags it must provide to the compiler in order to compile
  programs that use Eigen3 functionality; this may include the path to
  the Eigen3 include files, as well as any required flags.
  For example:
    $ export EIGEN_CFLAGS="-isystem /usr/include/eigen3"
    $./configure
  (amend with the actual path to the Eigen3 include files on your system)

EIGEN_CFLAGS="-isystem /usr/include/eigen3" ./configure -verbose
Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpohsg5p.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -isystem /usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpohsg5p.cpp -o c:/users/rob/appdata/local/temp/tmpohsg5p.o
EXIT: 1
STDERR:
c:/users/rob/appdata/local/temp/tmpohsg5p.cpp:3:10: fatal error: Eigen/Core: No such file or directory
 #include <Eigen/Core>
          ^~~~~~~~~~~~
compilation terminated.
>>

error deleting temporary file "c:/users/rob/appdata/local/temp/tmpohsg5p.o": The system cannot find the file specified
ERROR: error compiling Eigen3 application!

    MRtrix3 was unable to compile a test program involving Eigen3.

  Set the EIGEN_CFLAGS environment variable to inform 'configure' of
  the flags it must provide to the compiler in order to compile
  programs that use Eigen3 functionality; this may include the path to
  the Eigen3 include files, as well as any required flags.
  For example:
    $ export EIGEN_CFLAGS="-isystem /usr/include/eigen3"
    $./configure
  (amend with the actual path to the Eigen3 include files on your system)

EIGEN_CFLAGS="-I/usr/include/eigen3" ./configure -verbose
Checking for Eigen3 library:
COMPILE c:/users/rob/appdata/local/temp/tmpzwtkhd.cpp:
---

#include <cstddef>
#include <Eigen/Core>
#include <iostream>

int main (int argc, char* argv[]) {
  std::cout << EIGEN_WORLD_VERSION << "." << EIGEN_MAJOR_VERSION << "." << EIGEN_MINOR_VERSION << "\n";
  return 0;
}

---
EXEC <<
CMD: g++ -c -std=c++11 -pthread -DMRTRIX_WINDOWS -mms-bitfields -Wa,-mbig-obj -D_FILE_OFFSET_BITS=64 -DMRTRIX_WORD64 -IC:/msys64/usr/include/eigen3 c:/users/rob/appdata/local/temp/tmpzwtkhd.cpp -o c:/users/rob/appdata/local/temp/tmpzwtkhd.o
EXIT: 0
STDERR:
( ... MASSIVE LIST OF COMPILER WARNINGS ... )
>>

EXEC <<
CMD: g++ c:/users/rob/appdata/local/temp/tmpzwtkhd.o -Wl,-O1,--sort-common,--as-needed -pthread -Wl,--allow-multiple-definition -o a.out
EXIT: 0
>>

EXEC <<
CMD: ./a.out
EXIT: 0
STDOUT:
3.3.1
>>

3.3.1
@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Sep 29, 2017

Member

OK, this sounds similar to an issue I had a while back - I'd filed an issue on the MSYS2 repo, but I can't find any trace of it now - might have something to do with them having disabled the issues feature on that repo...

The gist of it is that MinGW applications are pure Windows applications, with no runtime dependence on MSYS as such. This means they don't perform path translation to Windows filenames. But these applications will typically be launched from the MSYS2 terminal, which will do path translation prior to invoking the executable - so you can use MinGW executables with Unix-styles paths from the command-line and expect them to behave appropriately most of the time. This translation relies on various heuristics, but there's a good chance that it catches the -I/usr/lib/filename case better than the "-isystem /usr/lib/filename" case. Which would I think explain the discrepancy - note that gcc was invoked with a Windows path in the -I case, whereas in the -isystem case, it was left unchanged as a Unix path.

The solution in this case is to make sure you are using the MSYS2 version of python - not the MinGW version. Basically make sure the only python package installed is python - not mingw-w64-x86_64-python or anything like that...

Member

jdtournier commented Sep 29, 2017

OK, this sounds similar to an issue I had a while back - I'd filed an issue on the MSYS2 repo, but I can't find any trace of it now - might have something to do with them having disabled the issues feature on that repo...

The gist of it is that MinGW applications are pure Windows applications, with no runtime dependence on MSYS as such. This means they don't perform path translation to Windows filenames. But these applications will typically be launched from the MSYS2 terminal, which will do path translation prior to invoking the executable - so you can use MinGW executables with Unix-styles paths from the command-line and expect them to behave appropriately most of the time. This translation relies on various heuristics, but there's a good chance that it catches the -I/usr/lib/filename case better than the "-isystem /usr/lib/filename" case. Which would I think explain the discrepancy - note that gcc was invoked with a Windows path in the -I case, whereas in the -isystem case, it was left unchanged as a Unix path.

The solution in this case is to make sure you are using the MSYS2 version of python - not the MinGW version. Basically make sure the only python package installed is python - not mingw-w64-x86_64-python or anything like that...

@jdtournier

This comment has been minimized.

Show comment
Hide comment
@jdtournier

jdtournier Oct 6, 2017

Member

Just to update on the compile failure on Windows - I just came across the same problem as I was about to answer this forum post. Turns out the problem is what I thought: the wrong version of python is being invoked:

$ which python.exe
/mingw64/bin/python.exe

Which is odd since I'm pretty sure I made sure I hadn't installed it...

So I did a bit of detective work... Which package owns this executable:

$ pacman -Qo /mingw64/bin/python.exe
/mingw64/bin/python.exe is owned by mingw-w64-x86_64-python2 2.7.14-1

What happens if I try to remove it:

$ pacman -R mingw-w64-x86_64-python2
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-glib2: removing mingw-w64-x86_64-python2 breaks dependency 'mingw-w64-x86_64-python2'

OK, let's try removing that too:

$ pacman -R mingw-w64-x86_64-glib2
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-dbus: removing mingw-w64-x86_64-glib2 breaks dependency 'mingw-w64-x86_64-glib2'
:: mingw-w64-x86_64-harfbuzz: removing mingw-w64-x86_64-glib2 breaks dependency 'mingw-w64-x86_64-glib2'

Let's try removing mingw-w64-x86_64-harfbuzz:

$ pacman -R mingw-w64-x86_64-harfbuzz
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-freetype: removing mingw-w64-x86_64-harfbuzz breaks dependency 'mingw-w64-x86_64-harfbuzz'
:: mingw-w64-x86_64-qt5: removing mingw-w64-x86_64-harfbuzz breaks dependency 'mingw-w64-x86_64-harfbuzz'

So installing Qt5 brings in the MinGW version of python2. Great... And that location appears first in the PATH.

Obviously, there's no simple solution here. The simplest workaround is to invoke a known-good version of python directly:

$ /usr/bin/python ./configure 
$ /usr/bin/python ./build

A better solution would be to ensure these scripts work regardless - but that's a potentially difficult thing to do well though: we'd need to intercept the output of pkg-config and translate anything that looks like a path to the Windows equivalent... And some of the MRtrix3 scripts may also be affected in unexpected ways - although to be fair that's probably unlikely.

I'll try to think of something here - maybe MSYS2 provide a utility to do its path translation that we can invoke...?

Member

jdtournier commented Oct 6, 2017

Just to update on the compile failure on Windows - I just came across the same problem as I was about to answer this forum post. Turns out the problem is what I thought: the wrong version of python is being invoked:

$ which python.exe
/mingw64/bin/python.exe

Which is odd since I'm pretty sure I made sure I hadn't installed it...

So I did a bit of detective work... Which package owns this executable:

$ pacman -Qo /mingw64/bin/python.exe
/mingw64/bin/python.exe is owned by mingw-w64-x86_64-python2 2.7.14-1

What happens if I try to remove it:

$ pacman -R mingw-w64-x86_64-python2
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-glib2: removing mingw-w64-x86_64-python2 breaks dependency 'mingw-w64-x86_64-python2'

OK, let's try removing that too:

$ pacman -R mingw-w64-x86_64-glib2
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-dbus: removing mingw-w64-x86_64-glib2 breaks dependency 'mingw-w64-x86_64-glib2'
:: mingw-w64-x86_64-harfbuzz: removing mingw-w64-x86_64-glib2 breaks dependency 'mingw-w64-x86_64-glib2'

Let's try removing mingw-w64-x86_64-harfbuzz:

$ pacman -R mingw-w64-x86_64-harfbuzz
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: mingw-w64-x86_64-freetype: removing mingw-w64-x86_64-harfbuzz breaks dependency 'mingw-w64-x86_64-harfbuzz'
:: mingw-w64-x86_64-qt5: removing mingw-w64-x86_64-harfbuzz breaks dependency 'mingw-w64-x86_64-harfbuzz'

So installing Qt5 brings in the MinGW version of python2. Great... And that location appears first in the PATH.

Obviously, there's no simple solution here. The simplest workaround is to invoke a known-good version of python directly:

$ /usr/bin/python ./configure 
$ /usr/bin/python ./build

A better solution would be to ensure these scripts work regardless - but that's a potentially difficult thing to do well though: we'd need to intercept the output of pkg-config and translate anything that looks like a path to the Windows equivalent... And some of the MRtrix3 scripts may also be affected in unexpected ways - although to be fair that's probably unlikely.

I'll try to think of something here - maybe MSYS2 provide a utility to do its path translation that we can invoke...?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment