Numpy MKL errors during installation

[adamjstewart]

I'm trying to build numexpr 2.6.9 from source using Python 3.7.4 and numpy 1.17.0, but I get the following error message during configuration:

mkl_info:
/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py:773: UserWarning: Specified path /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64;/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/compiler/lib/intel64 is invalid.
  return self.get_paths(self.section, key)
/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py:782: UserWarning: Specified path /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include is invalid.
  return self.get_paths(self.section, key)
Traceback (most recent call last):
  File "setup.py", line 228, in <module>
    setup_package()
  File "setup.py", line 224, in setup_package
    setup(**metadata)
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/core.py", line 137, in setup
    config = configuration()
  File "setup.py", line 122, in configuration
    mkl_config_data = config.get_info('mkl')
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/misc_util.py", line 2069, in get_info
    dict_append(info_dict,**get_info(a))
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 444, in get_info
    return cl().get_info(notfound_action)
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 690, in get_info
    self.calc_info()
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 1139, in calc_info
    opt = self.get_option_single('mkl_libs', 'libraries')
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-tt562t4vqgt2jc6j4gkk5rdj6mbakig2/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 659, in get_option_single
    section=self.section, options='[{}]'.format(', '.join(options))))
numpy.distutils.system_info.AliasedOptionError:
    Aliases entries in config files should not be existing.
    In section 'mkl' we found multiple appearances of options [mkl_libs, libraries].

It seems like something might be wrong with my numpy installation, but it passed all of its unit tests. Is numexpr 2.6.9 not compatible with numpy 1.17.0? How can I debug this?

[robbmcleod]

More likely a configuration error related to the Intel MKL, which might be difficult for me to diagnose since you are on OSX. Maybe @FrancescAlted can provide some insight. Can you provide the text in your site.cfg file and your install location of the MKL library? What build commands are you using? What's your version of setuptools and/or pip?

[adamjstewart]

I'm actually not trying to use MKL, I'm trying to use OpenBLAS. My numpy site.cfg looks like:

[openblas]
libraries=openblas
library_dirs=/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/openblas-0.3.6-3yts3o25cqf437oxaa4qzfhiw3swc7j2/lib
rpath=/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/openblas-0.3.6-3yts3o25cqf437oxaa4qzfhiw3swc7j2/lib
[mkl]
libraries=
library_dirs=
rpath=
[atlas]
libraries=
library_dirs=
rpath=

I'm using setuptools 41.0.1. My build command for numexpr is:

$ python3.7 -s setup.py --no-user-cfg build

I'm using the Spack package manager to handle the installation. I'm a Spack developer, so if this is a bug on Spack's side of things I can fix it.

[robbmcleod]

My recommendation is you build numpy first, then, as the error doesn't seem to lie with NumExpr, and you probably want to explicitly know what options the numpy requirement is getting. Building NumExpr without MKL support should be trivial once you get NumPy to build. My first step would be to completely delete the mkl part of the NumPy site.cfg.

[adamjstewart]

It looks like removing the [mkl] section from site.cfg solved the problem. This used to work in numpy 1.16, not sure what changes in numpy 1.17. Thanks for your help!

[adamjstewart]

Okay, this bug magically reappeared. I built numpy 1.17.0 with Python 3.7.4 and Intel MKL 2019.4. It passed all of its unit tests. The site.cfg is as follows:

[mkl]
libraries = mkl_rt
library_dirs = /opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib:/usr/lib
rpath = /opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib:/usr/lib
include_dirs = /opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/include

When I build numexpr 2.6.9, I see the following error message:

/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py:773: UserWarning: Specified path /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64;/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/compiler/lib/intel64 is invalid.
  return self.get_paths(self.section, key)
/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py:782: UserWarning: Specified path /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include is invalid.
  return self.get_paths(self.section, key)
Traceback (most recent call last):
  File "setup.py", line 228, in <module>
    setup_package()
  File "setup.py", line 224, in setup_package
    setup(**metadata)
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/core.py", line 137, in setup
    config = configuration()
  File "setup.py", line 122, in configuration
    mkl_config_data = config.get_info('mkl')
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/misc_util.py", line 2069, in get_info
    dict_append(info_dict,**get_info(a))
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 444, in get_info
    return cl().get_info(notfound_action)
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 690, in get_info
    self.calc_info()
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 1139, in calc_info
    opt = self.get_option_single('mkl_libs', 'libraries')
  File "/Users/Adam/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.17.0-3hrw57rz33kzrhxbsj7apu3jyvgiwu2l/lib/python3.7/site-packages/numpy/distutils/system_info.py", line 659, in get_option_single
    section=self.section, options='[{}]'.format(', '.join(options))))
numpy.distutils.system_info.AliasedOptionError: 
    Aliases entries in config files should not be existing.
    In section 'mkl' we found multiple appearances of options [mkl_libs, libraries].

[adamjstewart]

@rgommers do you have any idea why this bug might be occurring? Trying to bring in some numpy experts who know how the build system works.

[adamjstewart]

For the record, Scipy installs just fine with this Numpy installation:

lapack_opt_info:
lapack_mkl_info:
customize UnixCCompiler
customize UnixCCompiler
customize UnixCCompiler
  FOUND:
    libraries = ['mkl_rt', 'pthread', 'mkl_rt']
    library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/include', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    runtime_library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']

  FOUND:
    libraries = ['mkl_rt', 'pthread', 'mkl_rt']
    library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/include', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    runtime_library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']

Splitting linalg.interpolative Fortran source files
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
customize UnixCCompiler
customize UnixCCompiler
  FOUND:
    libraries = ['mkl_rt', 'pthread', 'mkl_rt']
    library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/include', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    runtime_library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']

  FOUND:
    libraries = ['mkl_rt', 'pthread', 'mkl_rt']
    library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/include', '/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']
    runtime_library_dirs = ['/opt/intel/compilers_and_libraries_2019.4.233/mac/mkl/lib']

So either numexpr tries to access different MKL info, or numexpr uses it in a different way. I have no idea where the /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64 paths are coming from.

[rgommers]

It looks like removing the [mkl] section from site.cfg solved the problem. This used to work in numpy 1.16, not sure what changes in numpy 1.17. Thanks for your help!

not sure about actual behavior, but I'm not surprised it's undefined - a site.cfg section should only be uncommented if it has actual content.

In section 'mkl' we found multiple appearances of options [mkl_libs, libraries].

I've never seen this before, but this is potentially a change in numpy.distutils. What I can imagine is that you have one entry in site.cfg, and a second one is inserted by mkl_info.calc_info (in numpy/distutils/system_info.py, it's the MKL auto-detection).

I'm not sure if that changed in numpy 1.17.0 though; if SciPy builds fine while numexpr doesn't, it could have always worked like that and numexpr does something unusual.

[adamjstewart]

I'm able to reproduce this problem on a Cray CNL5 cluster using GCC 6.3.0. Same /Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64 path. Let me try with another BLAS/LAPACK provider and see if that helps.

[adamjstewart]

numexpr + numpy + openblas builds fine, as does numexpr + numpy with no BLAS/LAPACK, so it seems this problem is specific to MKL.