Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiles with MKL on OS X, but 'make testall' fails calling mkl_blas_zdotc and elsewhere #17946

Closed
y9KAaixxegjR opened this issue Aug 10, 2016 · 8 comments

Comments

@y9KAaixxegjR
Copy link

@y9KAaixxegjR y9KAaixxegjR commented Aug 10, 2016

I am using OS X and followed the MKL installation instructions for the 0.5 release (Version 0.5.0-rc1+1 (2016-08-05 15:23 UTC), Commit acfd04c (5 days old release-0.5)). The build completed (output attached), but several tests failed (also attached). I also tried this with the 0.4.6 release, and I get the same mkl_blas_zdotc error in the arnoldi code.

My bash profile contains:

source /opt/intel/mkl/bin/mklvars.sh intel64 ilp64
MKLPATH=/opt/intel/mkl

My Maker.user file is:

USE_INTEL_MKL = 1
USE_INTEL_MKL_FFT = 1
USE_INTEL_LIBM = 0
USE_INTEL_JITEVENTS ?= 0
USEICC = 0
USEIFC = 0

My environment variables are:

echo $MKLROOT
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl
echo $DYLD_LIBRARY_PATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/tbb/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
echo $LIBRARY_PATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/tbb/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
echo $NLSPATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib/locale/%l_%t/%N
echo $MANPATH
nothing...
echo $CPATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/include

MakeTestallOutput.txt

@kshyatt

This comment has been minimized.

Copy link
Contributor

@kshyatt kshyatt commented Aug 10, 2016

Hi @y9KAaixxegjR, thank you for providing so much information. I've edited your OP to be a little easier to read.

@tkelman

This comment has been minimized.

Copy link
Contributor

@tkelman tkelman commented Aug 11, 2016

So you're trying to use MKL with clang and gfortran instead of the Intel compilers? I think there are Fortran calling convention differences depending on whether it uses the libmkl_intel_ilp64.dylib vs libmkl_gnu_ilp64.dylib. Since we use the single dynamic mkl_rt library, I don't think there's a way of getting it to use the gfortran convention. Try recompiling with ifort?

@y9KAaixxegjR

This comment has been minimized.

Copy link
Author

@y9KAaixxegjR y9KAaixxegjR commented Aug 12, 2016

Thanks for the prompt response!

Intel MKL is free on OS X through the community license, but the icc and ifort compilers are not. In fact, the compilers are each sold separately. Using 30-day free trials for the compilers, all permutations of USEICC = 0/1 and USEIFC = 0/1 are failing for me.

If USEICC=1, I cannot run make and get this error.

julia/Make.inc:408: *** USE_LIBCPP only supported with clang. Try setting USE_LIBCPP=0.  Stop.

Setting USE_LIBCPP=0 in the Make.user file does not change the result.

If USEICC = 0 and USEIFC = 1, then I get the attached output from 'make testall'
makeTestallWithifortranOnly.txt

@m-j-w

This comment has been minimized.

Copy link
Contributor

@m-j-w m-j-w commented Mar 24, 2017

It appears I have the same issue on Linux, Ubuntu 16.04 with MKL 2017.0.098 with today's master of Julia 0.6 (but also before). The failing call appears to be happening from the Julia bundled libarpack.so to libmkl_rt.so, although everything is compiled with same MKL settings and linked properly to same libmkl_rt.so.
Is it possible that the failing function zdotc which is invoked by arpack has a different signature in MKL?
Or any other ideas on where to start looking and how to limit the scope of the error?
(All other Julia tests appear to run fine when invoked manually.)

@tkelman

This comment has been minimized.

Copy link
Contributor

@tkelman tkelman commented Mar 24, 2017

You would need to rebuild with the same fortran calling convention that mkl uses.

@m-j-w

This comment has been minimized.

Copy link
Contributor

@m-j-w m-j-w commented Mar 24, 2017

Here's the explanation for this somewhat common problem when linking gfortran with MKL:
Fortran compilers may use two different calling conventions which affect complex number return values. Functions can be rewritten as subroutines with an additional argument that represents the function result, which in turn yields this segfault in particular for complex numbers being returned, or in the present case the zdotc function.

There are two solutions if such a segfault occurs:
a) pass "-ff2c" flag to gfortran which enforces the standard compliant calling convention, or
b) link explicitly against libmkl_gf_ilp64.so when using 64bit default integers, or libmkl_gf_lp64.so for 32bit default integers which are the gfortran compatible MKL libraries; check in Julia by printing Base.LinAlg.BlasInt which is either an Int64 or Int32.

See e.g. scipy/scipy#3193 (comment) or https://software.intel.com/en-us/articles/mkl-single-dynamic-library-libmkl-rtso-does-not-conform-to-the-gfortran-calling-convention

Is there interest in a PR, either as change of documentation or possibly to autodetect this unlucky combination, or both?

@tkelman

This comment has been minimized.

Copy link
Contributor

@tkelman tkelman commented Mar 25, 2017

using mkl along with gfortran built libraries has never really been supported with julia. if you're using mkl, will have better chances of things working if you build all fortran libraries with ifort. libmkl_gf_ilp64 does not link to its dependencies and isn't suited for using dynamically with ccall the way the single file libmkl_rt is. If mkl provided a single file libmkl_rt_gf that could potentially work.

@m-j-w

This comment has been minimized.

Copy link
Contributor

@m-j-w m-j-w commented Mar 25, 2017

It appears the Julia arpack tests Base.runtests("linalg/arnoldi") succeed when adding FFLAGS=-ff2c to Make.user for the combination of gfortran with MKL libmkl_rt.so.

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-pre.alpha.250 (2017-03-25 05:22 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 43fb9f0* (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> Base.LinAlg.BLAS.vendor()
:mkl

julia> Base.LinAlg.BlasInt
Int64

julia> Base.runtests("linalg/arnoldi")
Test (Worker)      | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
linalg/arnoldi (1) |   24.50  |  0.43  |  1.8 | 1201.37    | 324.24  

Test Summary: | Pass  Total
  Overall     |   76     76
    SUCCESS

The tests make check-arpack succeed as well in this configuration.

Thus, a possible fix could be to add to deps/tools/common.mk something like

ifeq  ($(USE_INTEL_MKL), 1)
ifneq ($(USEIFC),1)
USE_BLAS_FFLAGS += -ff2c
endif
endif
andreasnoack added a commit that referenced this issue Dec 28, 2017
andreasnoack added a commit that referenced this issue Dec 30, 2017
)

Fixes #17946
ararslan added a commit that referenced this issue Apr 26, 2018
)

Fixes #17946

Ref #25298
(cherry picked from commit dad96fb)
ararslan added a commit that referenced this issue Apr 26, 2018
)

Fixes #17946

Ref #25298
(cherry picked from commit dad96fb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.