make numpy and scipy use pkg-config to find blas/lapack #20157

kiwifb · 2016-03-04T01:46:46Z

Since we want the choice of blas/lapack to be configurable by .pc files we need to have numpy and scipy obey to pkg-config.

This is complicated by the fact that numpy/scipy's build system will look for

mkl
openblas
threaded atlas 3.10.x (tatlas.so)
single thread atlas 3.10.x (satlas.so)
threaded atlas (any versions)
single thread atlas (any versions)
accelerate on OS X
blas/lapack configuration provided by the user
blas/lapack source provided by the user

In that order from (1) to (9). Note that the user's wishes are ignored unless everything else fails. Getting numpy/scipy to obey the user wishes, whatever they are, means using not well documented variables to disable the auto-detections steps 1 to 6.

CC: @jdemeyer

Component: packages: standard

Author: François Bissey

Branch/Commit: af70376

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/20157

The text was updated successfully, but these errors were encountered:

kiwifb · 2016-03-04T03:07:07Z

comment:1

Actually

export {ATLAS,PTATLAS,BLAS,LAPACK,OPENBLAS,MKL}=None

shuts down all detection so there is hope things can be taken done selectively.

kiwifb · 2016-03-04T03:49:20Z

comment:2

Setting everything to None except for BLAS and LAPACK and then giving those some values is not enough. I have got something quite ugly.

kiwifb · 2016-03-04T10:28:05Z

Branch: u/fbissey/numpy_scipy_pc

kiwifb · 2016-03-04T10:28:05Z

Commit: 3237f38

kiwifb · 2016-03-04T10:28:05Z

Author: François Bissey

kiwifb · 2016-03-04T10:28:05Z

New commits:

`3237f38`	`Get numpy and scipy to use pc files`

kiwifb · 2016-03-05T19:19:14Z

Work Issues: version bumps

sagetrac-git · 2016-03-05T19:33:46Z

Branch pushed to git repo; I updated commit sha1. New commits:

`158fb8d`	`Version bump to trigger rebuilds`

sagetrac-git · 2016-03-05T19:33:46Z

Changed commit from 3237f38 to 158fb8d

kiwifb · 2016-03-05T19:34:24Z

Changed work issues from version bumps to none

vbraun · 2016-03-05T22:53:26Z

Reviewer: Volker Braun

vbraun · 2016-03-06T10:24:03Z

comment:10

Full log: http://build.sagedev.org/release/builders/%20%20fast%20Volker%20Desktop%20%28Fedora%2022%20x86_64%29%20incremental/builds/458/steps/compile/logs/stdio

/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/numpy-1.10.4.p1/src/numpy/distutils/system_info.py:1651: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/numpy-1.10.4.p1/src/numpy/distutils/system_info.py:635: UserWarning: Specified path  is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C compiler: cc

creating /tmp/tmpX0K4bQ/tmp
creating /tmp/tmpX0K4bQ/tmp/tmpX0K4bQ
compile options: '-c'
cc: /tmp/tmpX0K4bQ/source.c
cc /tmp/tmpX0K4bQ/tmp/tmpX0K4bQ/source.o -L/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib -lcblas -o /tmp/tmpX0K4bQ/a.out
/usr/bin/ld: warning: libatlas.so.3, needed by /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib/libcblas.so, not found (try using -rpath or -rpath-link)

It is missing the -latlas for some reason, though it is in the cblas.pc

$ pkg-config --libs cblas
-L/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib -lcblas -latlas

kiwifb · 2016-03-06T10:27:38Z

comment:11

Can you look at the generated site.cfg file?

kiwifb · 2016-03-06T10:31:48Z

comment:12

I notice poseidon also has a problem but look slightly different.

vbraun · 2016-03-06T10:39:45Z

comment:13

$ cat /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/numpy-1.10.4.p1/src/site.cfg
[DEFAULT]
library_dirs = /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib
include_dirs = /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/include
[blas]
include_dirs = 
library_dirs = /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib
blas_libs = atlas,f77blas,cblas
[lapack]
library_dirs = /mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib
lapack_libs = lapack,f77blas,cblas,atlas

sagetrac-git · 2016-03-13T22:13:27Z

Changed commit from f9658ba to 050dff9

sagetrac-git · 2016-03-13T22:13:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`050dff9`	`more targeted output`

kiwifb · 2016-03-13T22:13:53Z

comment:31

Still no review please.

kiwifb · 2016-03-14T20:57:44Z

Changed commit from 050dff9 to fea3a2a

kiwifb · 2016-03-14T20:57:44Z

Dependencies: 20208

kiwifb · 2016-03-14T20:57:44Z

Changed branch from u/fbissey/numpy_scipy_pc_dbg to u/fbissey/numpy_scipy_pc

kiwifb · 2016-03-14T20:57:44Z

comment:32

Random failures should now be solved by #20208.

vbraun · 2016-03-14T21:43:56Z

Changed dependencies from 20208 to #20208

sagetrac-git · 2016-03-22T00:24:51Z

Changed commit from fea3a2a to 38007ae

sagetrac-git · 2016-03-22T00:24:51Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

sagetrac-git · 2016-03-22T01:52:50Z

Changed commit from 38007ae to b2c95cb

sagetrac-git · 2016-03-22T01:52:50Z

Branch pushed to git repo; I updated commit sha1. New commits:

`20e1893`	`Do not strip out system libs in pkg-config`
`b2c95cb`	`Merge branch 'keep_system_libraries_in_pkgconf' into numpy_scipy_pc`

kiwifb · 2016-03-22T01:54:07Z

comment:37

Clean up and merge #20208 so it does build on poseidon out of the box. Cross fingers.

kiwifb · 2016-03-22T20:38:32Z

comment:38

And poseidon still fails because pkgconf hasn't been rebuild before numpy..... I guess I will have to wait until #20208 is in a beta.

sagetrac-git · 2016-03-25T03:07:16Z

Changed commit from b2c95cb to 41a7bde

sagetrac-git · 2016-03-25T03:07:16Z

Branch pushed to git repo; I updated commit sha1. New commits:

`41a7bde`	`Merge branch 'develop' into numpy_scipy_pc`

kiwifb · 2016-03-25T03:08:56Z

comment:40

Rebased on 7.2.beta0 which includes #20208.

kiwifb · 2016-03-25T03:08:56Z

Changed dependencies from #20208 to none

sagetrac-git · 2016-03-28T22:07:29Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a7869d8`	`Merge branch 'develop' into numpy_scipy_pc`
`af70376`	`try another strategy to get things going on poseidon`

sagetrac-git · 2016-03-28T22:07:29Z

Changed commit from 41a7bde to af70376

kiwifb · 2016-03-29T05:00:39Z

comment:42

Finally building on poseidon. Theory: #20208 fixes the case where there is no global pkg-config installed and sage's pkgconf is used and in that case having it not strip default library directories when present in the .pc file. But poseidon still failed, probably because it has a global pkg-config command available and it happens to be pkgconf. Because it is provided by the system #20208 doesn't fix it.

The lapack.conf script is much more robust as a result of the latest commit.

Putting back to positive review.

vbraun · 2016-03-30T14:05:49Z

Changed branch from u/fbissey/numpy_scipy_pc to af70376

kiwifb added this to the sage-7.1 milestone Mar 4, 2016

kiwifb added p: major / 3 labels Mar 4, 2016

This comment has been minimized.

Sign in to view

kiwifb added the s: needs review label Mar 5, 2016

kiwifb added s: needs work and removed s: needs review labels Mar 5, 2016

kiwifb added s: needs review and removed s: needs work labels Mar 5, 2016

vbraun added s: positive review and removed s: needs review labels Mar 5, 2016

vbraun added s: needs work and removed s: positive review labels Mar 6, 2016

kiwifb modified the milestones: sage-7.1, sage-7.2 Mar 21, 2016

kiwifb added s: positive review and removed s: needs review labels Mar 29, 2016

vbraun removed the s: positive review label Mar 30, 2016

vbraun closed this as completed in 899e822 Mar 30, 2016

embray mentioned this issue Oct 4, 2021

Slight annoyance wrt Numpy and ATLAS #20507

Closed

mkoeppe mentioned this issue Jan 31, 2020

Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections #29051

Closed

make numpy and scipy use pkg-config to find blas/lapack #20157

make numpy and scipy use pkg-config to find blas/lapack #20157

Comments

kiwifb commented Mar 4, 2016

kiwifb commented Mar 4, 2016

This comment has been minimized.

kiwifb commented Mar 4, 2016

This comment has been minimized.

kiwifb commented Mar 4, 2016

kiwifb commented Mar 4, 2016

kiwifb commented Mar 4, 2016

kiwifb commented Mar 4, 2016

kiwifb commented Mar 5, 2016

sagetrac-git mannequin commented Mar 5, 2016

sagetrac-git mannequin commented Mar 5, 2016

kiwifb commented Mar 5, 2016

vbraun commented Mar 5, 2016

vbraun commented Mar 6, 2016

kiwifb commented Mar 6, 2016

kiwifb commented Mar 6, 2016

vbraun commented Mar 6, 2016

sagetrac-git mannequin commented Mar 13, 2016

sagetrac-git mannequin commented Mar 13, 2016

kiwifb commented Mar 13, 2016

kiwifb commented Mar 14, 2016

kiwifb commented Mar 14, 2016

kiwifb commented Mar 14, 2016

kiwifb commented Mar 14, 2016

vbraun commented Mar 14, 2016

sagetrac-git mannequin commented Mar 22, 2016

sagetrac-git mannequin commented Mar 22, 2016

sagetrac-git mannequin commented Mar 22, 2016

sagetrac-git mannequin commented Mar 22, 2016

kiwifb commented Mar 22, 2016

kiwifb commented Mar 22, 2016

sagetrac-git mannequin commented Mar 25, 2016

sagetrac-git mannequin commented Mar 25, 2016

kiwifb commented Mar 25, 2016

kiwifb commented Mar 25, 2016

sagetrac-git mannequin commented Mar 28, 2016

sagetrac-git mannequin commented Mar 28, 2016

kiwifb commented Mar 29, 2016

vbraun commented Mar 30, 2016