NumPy windows build is broken

[SteveDiamond]

The numpy.distutils.system_info.get_info function is broken on the windows build of NumPy. I need this function to get the location of BLAS and LAPACK installations. But when I call get_info with the Anaconda NumPy build I just get an error:

>>> from numpy.distutils.system_info import get_info
>>> get_info('blas_opt')
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\lib is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\include is invalid
.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
46: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
55: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
58: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)

For the full details, see this issue: cvxgrp/scs#72

[bodono]

Here is a copy-paste of my findings from the linked bug. Let me know if I have misunderstood how it's meant to work:

As far as I can tell this is an issue with the numpy conda package. SCS uses numpy.distutils.system_info function get_info to tell it where the blas/lapack libraries it should link against are located. Numpy stores that information in a site.cfg file. But it appears this is broken in the conda numpy package. On a clean windows machine I installed anaconda, then installed numpy via conda install numpy. Then in a python interpreter I run:

>>> from numpy.distutils.system_info import get_info
>>> get_info('blas_opt')
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\lib is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\include is invalid
.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
46: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
55: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
58: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)

This is before I've even downloaded SCS. It's looking for the libs and includes in C:\Minonda\envs\_build, which doesn't exist on my machine.

After some digging I found this part of the guide in conda about making packages relocatable, which says that the build prefix is replaced in any file that contains it with a placeholder and the file is added to the list in info/has_prefix. If you examine the packages from the conda package repo you can inspect the windows and linux numpy packages manually. In the linux one, it has an info/has_prefix file which has

/opt/anaconda1anaconda2anaconda3 text lib/python2.7/site-packages/numpy/distutils/site.cfg

and the site.cfg file has

[mkl]
library_dirs = /opt/anaconda1anaconda2anaconda3/lib
include_dirs = /opt/anaconda1anaconda2anaconda3/include
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5

In other words the has_prefix file is telling conda to replace the placeholder string /opt/anaconda1anaconda2anaconda3 with the real prefix when it is installing it (since that's when it will be known).

However the latest windows numpy packages (e.g. numpy-1.11.1-py27_1.tar.bz2) do not have a has_prefix file, so it doesn't do the prefix swapping. The site.cfg file simply has

[mkl]
include_dirs = C:\Minonda\envs\_build\Library\include
library_dirs = C:\Minonda\envs\_build\Library\lib
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll

But that directory doesn't exist, and the prefix is not getting replaced, so it ends up with the error above.

However, an older windows version of numpy, numpy-1.11.0-py27_2 does have an info/has_prefix file, that has

"C:\Minonda\envs\_build" text "Lib/site-packages/numpy/distutils/site.cfg"

Which is saying to swap out the C:\Minonda\envs\_build for the real prefix in the site.cfg file. So you would think that would replace the path correctly, however it still throws an error for me because the path that it replaces it with is also invalid (doesn't exist, and mixes forwards and backwards slashes in the path).

I think the message here is that the conda numpy packages for windows are untested and broken, and appear to have been broken in different ways for a long time. I guess the right thing to do is flag it with the anaconda team. I never even got to the point of installing SCS on the system!