Compiles with MKL on OS X, but 'make testall' fails calling mkl_blas_zdotc and elsewhere

[Divisible8737]

I am using OS X and followed the MKL installation instructions for the 0.5 release (Version 0.5.0-rc1+1 (2016-08-05 15:23 UTC), Commit acfd04c (5 days old release-0.5)). The build completed (output attached), but several tests failed (also attached). I also tried this with the 0.4.6 release, and I get the same mkl_blas_zdotc error in the arnoldi code.

My bash profile contains:

source /opt/intel/mkl/bin/mklvars.sh intel64 ilp64
MKLPATH=/opt/intel/mkl

My Maker.user file is:

USE_INTEL_MKL = 1
USE_INTEL_MKL_FFT = 1
USE_INTEL_LIBM = 0
USE_INTEL_JITEVENTS ?= 0
USEICC = 0
USEIFC = 0

My environment variables are:

echo $MKLROOT
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl
echo $DYLD_LIBRARY_PATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/tbb/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
echo $LIBRARY_PATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/tbb/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
echo $NLSPATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib/locale/%l_%t/%N
echo $MANPATH
nothing...
echo $CPATH
/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/include

MakeTestallOutput.txt

[kshyatt]

Hi @y9KAaixxegjR, thank you for providing so much information. I've edited your OP to be a little easier to read.

[tkelman]

So you're trying to use MKL with clang and gfortran instead of the Intel compilers? I think there are Fortran calling convention differences depending on whether it uses the libmkl_intel_ilp64.dylib vs libmkl_gnu_ilp64.dylib. Since we use the single dynamic mkl_rt library, I don't think there's a way of getting it to use the gfortran convention. Try recompiling with ifort?

[Divisible8737]

Thanks for the prompt response!

Intel MKL is free on OS X through the community license, but the icc and ifort compilers are not. In fact, the compilers are each sold separately. Using 30-day free trials for the compilers, all permutations of USEICC = 0/1 and USEIFC = 0/1 are failing for me.

If USEICC=1, I cannot run make and get this error.

julia/Make.inc:408: *** USE_LIBCPP only supported with clang. Try setting USE_LIBCPP=0.  Stop.

Setting USE_LIBCPP=0 in the Make.user file does not change the result.

If USEICC = 0 and USEIFC = 1, then I get the attached output from 'make testall'
makeTestallWithifortranOnly.txt

[m-j-w]

It appears I have the same issue on Linux, Ubuntu 16.04 with MKL 2017.0.098 with today's master of Julia 0.6 (but also before). The failing call appears to be happening from the Julia bundled libarpack.so to libmkl_rt.so, although everything is compiled with same MKL settings and linked properly to same libmkl_rt.so.
Is it possible that the failing function zdotc which is invoked by arpack has a different signature in MKL?
Or any other ideas on where to start looking and how to limit the scope of the error?
(All other Julia tests appear to run fine when invoked manually.)

[tkelman]

You would need to rebuild with the same fortran calling convention that mkl uses.

[m-j-w]

Here's the explanation for this somewhat common problem when linking gfortran with MKL:
Fortran compilers may use two different calling conventions which affect complex number return values. Functions can be rewritten as subroutines with an additional argument that represents the function result, which in turn yields this segfault in particular for complex numbers being returned, or in the present case the zdotc function.

There are two solutions if such a segfault occurs:
a) pass "-ff2c" flag to gfortran which enforces the standard compliant calling convention, or
b) link explicitly against libmkl_gf_ilp64.so when using 64bit default integers, or libmkl_gf_lp64.so for 32bit default integers which are the gfortran compatible MKL libraries; check in Julia by printing Base.LinAlg.BlasInt which is either an Int64 or Int32.

See e.g. scipy/scipy#3193 (comment) or https://software.intel.com/en-us/articles/mkl-single-dynamic-library-libmkl-rtso-does-not-conform-to-the-gfortran-calling-convention

Is there interest in a PR, either as change of documentation or possibly to autodetect this unlucky combination, or both?

[tkelman]

using mkl along with gfortran built libraries has never really been supported with julia. if you're using mkl, will have better chances of things working if you build all fortran libraries with ifort. libmkl_gf_ilp64 does not link to its dependencies and isn't suited for using dynamically with ccall the way the single file libmkl_rt is. If mkl provided a single file libmkl_rt_gf that could potentially work.

[m-j-w]

It appears the Julia arpack tests Base.runtests("linalg/arnoldi") succeed when adding FFLAGS=-ff2c to Make.user for the combination of gfortran with MKL libmkl_rt.so.

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-pre.alpha.250 (2017-03-25 05:22 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 43fb9f0* (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> Base.LinAlg.BLAS.vendor()
:mkl

julia> Base.LinAlg.BlasInt
Int64

julia> Base.runtests("linalg/arnoldi")
Test (Worker)      | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
linalg/arnoldi (1) |   24.50  |  0.43  |  1.8 | 1201.37    | 324.24  

Test Summary: | Pass  Total
  Overall     |   76     76
    SUCCESS

The tests make check-arpack succeed as well in this configuration.

Thus, a possible fix could be to add to deps/tools/common.mk something like

ifeq  ($(USE_INTEL_MKL), 1)
ifneq ($(USEIFC),1)
USE_BLAS_FFLAGS += -ff2c
endif
endif