Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

scipy install fails with 'ld: library not found for -lgfortran' #8

Closed
aisipos opened this Issue · 15 comments

2 participants

@aisipos

On attempting to install scipy, I get this failure:

» brew install scipy
==> Downloading http://sourceforge.net/projects/scipy/files/scipy/0.11.0/scipy-0.11.0.tar.gz
Already downloaded: /Library/Caches/Homebrew/scipy-0.11.0.tar.gz
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> python -s setup.py --no-user-cfg --verbose build --fcompiler=gnu95 install --force --install-scripts=/usr/local/Cellar/scipy/0.11.0/share/python --install-lib=/usr/local/Ce
ld: library not found for -lgfortran
error: Command "/usr/bin/gfortran -Wall -arch x86_64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.8-x86_64-2.7/build/src.macosx-10.8-x86_64-2.7/scipy/fftpack/_fftpackmodule.o build/temp.macosx-10.8-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.macosx-10.8-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.macosx-10.8-x86_64-2.7/scipy/fftpack/src/zrfft.o build/temp.macosx-10.8-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.8-x86_64-2.7/build/src.macosx-10.8-x86_64-2.7/scipy/fftpack/src/dct.o build/temp.macosx-10.8-x86_64-2.7/build/src.macosx-10.8-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.macosx-10.8-x86_64-2.7/build/src.macosx-10.8-x86_64-2.7/fortranobject.o -Lbuild/temp.macosx-10.8-x86_64-2.7 -ldfftpack -lfftpack -lgfortran -o build/lib.macosx-10.8-x86_64-2.7/scipy/fftpack/_fftpack.so" failed with exit status 1

Admittedly, I have no experience with gfortran and perhaps I've missed something trivial, but I don't know how to proceed with fixing this. How can I help diagnose this issue?

@samueljohn samueljohn was assigned
@samueljohn
Owner

Can you provide me the output of your brew --config and brew doctor?

@samueljohn
Owner

If you have brew install gfortran, there should be a gfortran executable in the PATH. That is whay scipy looks for.

@aisipos

Thanks for your help:

» brew --config
HOMEBREW_VERSION: 0.9.3
HEAD: a00f7e96bfd998d2a938fa7c3d37aba65a1adcb6
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
CPU: quad-core 64-bit sandybridge
OS X: 10.8.2-x86_64
Xcode: 4.5.2
CLT: 4.5.0.0.1.1249367152
LLVM-GCC: build 2336
Clang: 4.1 build 421
X11: 2.7.4 => /opt/X11
System Ruby: 1.8.7-358
Perl: /usr/bin/perl
Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/local/bin/ruby => /usr/local/Cellar/ruby/1.9.3-p194/bin/ruby

anton@airliner:~ (!)
» brew doctor
Error: Failed to import: composer-requirement
Error: Failed to import: homebrew-php-requirement
Error: Failed to import: phar-building-requirement
Error: Failed to import: phar-requirement
Error: Failed to import: php-meta-requirement
Error: Failed to import: xhgui53-requirement
Error: Failed to import: xhgui54-requirement
Error: Failed to import: xhprof53-requirement
Error: Failed to import: xhprof54-requirement
Your system is raring to brew.

anton@airliner:~ (!)
» which gfortran
/usr/local/bin/gfortran

anton@airliner:~ (!)
» gfortran --version
GNU Fortran (GCC) 4.7.2
Copyright (C) 2012 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING
@aisipos

I think I am being bit by this issue:
Homebrew/homebrew#8539
(Library libgfortran not found after installing gfortran)

I'll peek at that issue and see if I can report any findings here

@aisipos

Here are all the locations in /usr where libgfortran is located on my machine:

» sudo find /usr -iname 'libgfortran*.a'
/usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/i386/libgfortranbegin.a
/usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/libgfortranbegin.a
/usr/local/Cellar/gfortran/4.7.2/gfortran/lib/i386/libgfortran.a
/usr/local/Cellar/gfortran/4.7.2/gfortran/lib/libgfortran.a
/usr/local/gfortran/lib/gcc/x86_64-apple-darwin11/4.6.2/i386/libgfortranbegin.a
/usr/local/gfortran/lib/gcc/x86_64-apple-darwin11/4.6.2/libgfortranbegin.a
/usr/local/gfortran/lib/i386/libgfortran.a
/usr/local/gfortran/lib/libgfortran.a
/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/libgfortranbegin.a
/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64/libgfortranbegin.a
/usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/libgfortranbegin.a
/usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/ppc64/libgfortranbegin.a
/usr/local/lib/ppc64/libgfortran.a
/usr/local/lib/x86_64/libgfortran.a
@aisipos

Based on advice in Homebrew/homebrew#8539 , I even tried:

export LIBRARY_PATH=/usr/local/lib/gcc

but I still get the same error:

ld: library not found for -lgfortran

A little more investigation, here are the shasums of my libgfortrans:

» sudo find /usr -iname 'libgfortran*.a' | xargs shasum
Password:
4e56b92bb77e5971c245c4e5d8803e5aa0f24c18  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/i386/libgfortranbegin.a
3e45a1008510ef2cbf6b03cd6af2e3f6bbd19718  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/libgfortranbegin.a
d602dde9e05654ef73b8c879df0cffb8c41bff00  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/i386/libgfortran.a
4bb0e5793c848e40a888aa80341cda1ed2cfca86  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/libgfortran.a
e0689d42df8a9fdfe1b524290644122bff3ce085  /usr/local/gfortran/lib/gcc/x86_64-apple-darwin11/4.6.2/i386/libgfortranbegin.a
dc2fe446ea3804e70104279ee1e24696d056d7d7  /usr/local/gfortran/lib/gcc/x86_64-apple-darwin11/4.6.2/libgfortranbegin.a
371461e357993eb21328f16613c395261bebc21b  /usr/local/gfortran/lib/i386/libgfortran.a
46c454ad30ccc76d3a0dccc5b70793a60d7586e4  /usr/local/gfortran/lib/libgfortran.a
804b0a4dd08a14120fcc9101e7028e98d365fd0f  /usr/local/lib/gcc/i686-apple-darwin8/4.2.3/libgfortranbegin.a
b86aace212cfc55aefcc96f2481232e9cd1e0f2d  /usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64/libgfortranbegin.a
1d10ec81d4d8f007caf67913c43cc8cee2b6711b  /usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/libgfortranbegin.a
52846cf932e670da1890934792cd29a1d9a17edb  /usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/ppc64/libgfortranbegin.a
shasum: /usr/local/lib/ppc64/libgfortran.a: No such file or directory
shasum: /usr/local/lib/x86_64/libgfortran.a: No such file or directory

Turns out the last 2 are broken links!

» ls -la /usr/local/lib/x86_64/libgfo*
lrwxr-xr-x 1 anton wheel 16 Jul 26  2011 /usr/local/lib/x86_64/libgfortran.a -> ../libgfortran.a

» ls -la /usr/local/lib/libgfo*
ls: cannot access /usr/local/lib/libgfo*: No such file or directory

I don't know if brew install gfortran got me in this situation or not, I had run that some time ago. Reinstalling the gfortran brew doesn't seem to help.

I could link one of the libgfortran.a files to /usr/local/lib, but I'm not sure now which one is right. Then presumably I would need to set LIBRARY_PATH = /usr/local/lib, right?

This may also be relevant, I do have ARCHFLAGS set:

» echo $ARCHFLAGS 
-arch x86_64
@samueljohn
Owner

ARCHFLAGS might indeed be an issue. unset ARCHFLAGS - homebrew always attempts to build 64bit (unless you specify --universal, which is available for some).

@samueljohn
Owner
find /homebrew -iname 'libgfortran*.a' | xargs shasum
12c2b9fce41c95a6971b3eacecf2c0851ec706ad  /homebrew/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/i386/libgfortranbegin.a
1ff4d7bd70a5782a4759d1858955bec610568648  /homebrew/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/libgfortranbegin.a
3c71820a9e9865987c4c1dd4434fc3a6b7eb87ef  /homebrew/Cellar/gfortran/4.7.2/gfortran/lib/i386/libgfortran.a
b1054a8ef78c750fa8e8bb593035994dacdbef72  /homebrew/Cellar/gfortran/4.7.2/gfortran/lib/libgfortran.a

Besides, I have my test homebrew at /homebrew, it seems you had an older/mismatching gfortran from elsewhere. I would try to remove all gfortran remains and try to brew rm gfortran, brew install gfortran. Then, first remove numpy (because scipy re-uses the distutils settings from numpy, too).
So: brew rm numpy, then brew install numpy and brew install scipy.

Btw. homebrew's stuff in /usr/local does not require sudo, so this is evidence for a pkg or whatever that installed gfortran.

@samueljohn
Owner

The hashes are a bit different in my /usr/local - homebrew (due to rpath or other bits different)

find /usr -iname 'libgfortran*.a' | xargs shasum
74f41043171ced9d4f1e81ffa0c33a30e568c414  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/i386/libgfortranbegin.a
1b14d1e5fd2d53a011bfe40fdfa4ad74abd3d945  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/libgfortranbegin.a
5f2251b4419001fe15bb000ed6c609818c296225  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/i386/libgfortran.a
d8af17fd7de5a325aed6b8209bf4c03f4772aa31  /usr/local/Cellar/gfortran/4.7.2/gfortran/lib/libgfortran.a
@samueljohn
Owner

perhaps also remove the gcc stuff (brew rm gcc).

@aisipos

I've cleaned my gfortran install as much as I think I could:

unset ARCHFLAGS
brew rm numpy
pip uninstall numpy
brew rm gfortran
brew rm gcc
sudo rm -rf /usr/local/gfortran
find /usr/local -iname '*gfortran*' -depth -delete

I then

brew install gfortran
brew install numpy --with-openblas

I get:

» brew install numpy --with-openblas
==> Installing numpy dependency: openblas
==> Downloading http://github.com/xianyi/OpenBLAS/zipball/v0.2.5
Already downloaded: /Library/Caches/Homebrew/openblas-0.2.5.zip
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading http://www.netlib.org/lapack/lapack-3.4.2.tgz
Already downloaded: /Library/Caches/Homebrew/lapack-3.4.2.tgz
==> Using LAPACK: /Library/Caches/Homebrew/lapack-3.4.2.tgz
==> make FC=/usr/bin/gfortran
i686-apple-darwin11-gfortran-4.2.1: error trying to exec 'f951': execvp: No such file or directory
make[1]: *** [dblat1.o] Error 1
i686-apple-darwin11-gfortran-4.2.1: error trying to exec 'f951': execvp: No such file or directory
make[1]: *** [cblat1.o] Error 1
make: *** [tests] Error 2

I can install numpy without --openblas. Again gfortran is causing problems, f951 is present in /usr/local/bin but is a broken symlink:

» file /usr/local/bin/f951
/usr/local/bin/f951: broken symbolic link to /usr/local/gfortran/libexec/gcc/x86_64-apple-darwin11/4.6.2/f951

If do a plain:

brew install numpy
brew link --overwrite numpy
brew install scipy

Numpy does install, but then scipy install fails with the same f951 error:

» brew install scipy
==> Downloading http://sourceforge.net/projects/scipy/files/scipy/0.11.0/scipy-0.11.0.tar.gz
Already downloaded: /Library/Caches/Homebrew/scipy-0.11.0.tar.gz
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> python -s setup.py --no-user-cfg --verbose build --fcompiler=gnu95 install --force --install-scripts=/usr/local/Cellar/scipy/0.11.0/share/python --install-lib=/usr/local/Ce
i686-apple-darwin11-gfortran-4.2.1: error trying to exec 'f951': execvp: No such file or directory
_update_environment(...)
i686-apple-darwin11-gfortran-4.2.1: error trying to exec 'f951': execvp: No such file or directory
error: Command "/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -O3 -funroll-loops -I/usr/local/lib/python2.7/site-packages/numpy/core/include -c -c scipy/fftpack/src/dfftpack/dcosqb.f -o build/temp.macosx-10.8-x86_64-2.7/scipy/fftpack/src/dfftpack/dcosqb.o" failed with exit status 1
removing /var/folders/c_/6xbq8h9s0xbcqfkrwm_c06h80000gn/T/tmpbb9bTe

I've tried to make as clean a hombrew installed gfortran as I could, but there are still broken symlinks (f951 in this case). Should I try with the Apple installed gfortran?

@aisipos

Fixed - there were still multiple files remaining from an older gfortran version, not installed from hombrew. (As an aside, I should look into checkinstall on osx to avoid this problem). After cleaning more of them, most importantly /usr/bin/gfortran, brew install scipy now works.

It seems the ENV machinery for gfortran assumes that if which gfortran returns anything, that this is a homebrew installed gfortran. This is not necessarily the case, it should really look for /usr/local/bin/gfortran. Having another gfortran installed can really confuse homebrew recipes, and perhaps should have a caveat about this in the formula or a check in brew doctor. Might be worth a bug in the gfortran formula.

@aisipos aisipos closed this
@samueljohn
Owner

Ahhh - I am really glad, you found that files! I will keep that in mind.

What is checkinstall?

It's always a balance between choice and stability. For example, we allow to use the Python from OS X or any other Python (though we can't support and test all combinations) additionally to brew install python. Often this causes trouble - but in the python case, I hope the effort is worth it. For most dependencies, we favor to hard-code and install the dependency (e.g. we don't allow Python being built against any libreadline, but we provide one and everybody seems to be happy about that).
For (g)fortran, we had the same discussion. People may want to download any other fortran (e.g. from Intel). That is why homebrew allows to set the FC variable. But numpy on OS X ignores the FC and looks for a fortran itself. I wrote the Requirement (in the numpy formula) to check for a gfortran, because I had reports that this is the best option on OS X. However numpy (stupidly) prefers f95 if that is available. This is not a homebrew issue.

I have pushed a fix to the numpy formula that removes the f95 from the list of fortran compiler numpy looks for. So the issue should probably be fixed.

However, I still want to allow people installing gfortran from other sources (perhaps a manual build of gcc with special options or something). This will still be possible.

Having said that, I agree that most Python are no hardcore gfortran users and just want numpy to work.

PS: I suggest not to use --with-openblas right now, I found that some LAPACK functions (eigenvalue decomp.) are 5-fold slower (or more) because OpenBLAS does not optimize nor parallelize them. But Accelerate.framework does. The "BLAS" performance of OpenBLAS is really good, however.

@aisipos

Thanks for all your help!

Checkinstall is a way on linux to install code from source, but turn it into a package that can be uninstalled. Since I installed gfortran originally from source, and it spilled files everywhere, there was no easy way to uninstall it. Checkinstall is designed to fix cases like this, but it seems tied to package managers on unix distributions, so I doubt it would have helped here on osx.

Hmm, thanks for the tip on --with-openblas . I'll compile without it.

@samueljohn
Owner

Great, then. Thanks for the report.

I didn't knew checkinstall. You can never know where a Makefile puts a file. All you can do is to specify a --prefix or something (as homebrew does). The only thing possible would be to observe the file system modifications originating from the make install process. This should be possible on OS X.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.