Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

ENH: Respect blas_opt and lapack_opt groups in site.cfg. #2751

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
6 participants
Contributor

bfroehle commented Nov 18, 2012

Do not merge (yet).

Previously blas_opt and lapack_opt would only be merged into the autodetected BLAS and LAPACK libraries. This prevented using custom LAPACK and BLAS implementation if a system installed version of ATLAS (or similar) was present.

Now the presence of a blas_opt or lapack_opt section completely overrides the respective autodetection mechanisms, as the site.cfg documentation would already lead you to believe.

I'm opening this for discussion. There's a lot more backstory and some basic scripts (to be used to test the behavior of the autodetection) in #2728. I'm not convinced in any way that this is the "correct" approach to the problem. Also it doesn't appear that the language parameter can be set this way -- but is it even necessary to set?

@bfroehle bfroehle ENH: Respect blas_opt and lapack_opt groups in site.cfg.
Previously blas_opt and lapack_opt would only be merged into the
autodetected BLAS and LAPACK libraries. This prevented using custom
LAPACK and BLAS implementation if a system installed version of
ATLAS (or similar) was present.

Now the presence of a blas_opt or lapack_opt section completely
overrides the respective autodetection mechanisms, as the site.cfg
documentation would already lead you to believe.

Closes #2728.
781292a
Owner

njsmith commented Nov 18, 2012

You should send a ping to the mailing list with a description of the issues
here. Most people who use numpy in weirder build environments don't get
automatic notifications about PRs.
On 18 Nov 2012 22:34, "Bradley M. Froehle" notifications@github.com wrote:

Do not merge (yet).

Previously blas_opt and lapack_opt would only be merged into the
autodetected BLAS and LAPACK libraries. This prevented using custom LAPACK
and BLAS implementation if a system installed version of ATLAS (or similar)
was present.

Now the presence of a blas_opt or lapack_opt section completely overrides
the respective autodetection mechanisms, as the site.cfg documentation
would already lead you to believe.

I'm opening this for discussion. There's a lot more backstory and some
basic scripts (to be used to test the behavior of the autodetection) in
#2728 #2728. I'm not convinced in
any way that this is the "correct" approach to the problem. Also it doesn't
appear that the language parameter can be set this way -- but is it even

necessary to set?

You can merge this Pull Request by running:

git pull https://github.com/bfroehle/numpy site_cfg_opt

Or view, comment on, or merge it at:

#2751
Commit Summary

  • ENH: Respect blas_opt and lapack_opt groups in site.cfg.

File Changes

  • M numpy/distutils/system_info.py (12)

Patch Links

Contributor

nouiz commented Nov 19, 2012

I agree with the problem, but I didn't looked at the solution.

Contributor

bfroehle commented Nov 30, 2012

@njsmith: I've pinged the mailing list as you suggested. Thanks.

@y-p y-p referenced this pull request Dec 4, 2012

Merged

Backports Travis-CI fix #2786

Contributor

bfroehle commented Dec 6, 2012

After some discussion in the mailing list, I have discovered that a site.cfg which explicitly disables ATLAS would suffice. However this leads to the other big issue which is that there is no way to tell the build system that you have CBLAS (and hence build numpy.core._dotblas). The pull request did an end-run around that by letting the user specify the blas_opt and lapack_opt arguments very explicitly and in doing so carefully inform numpy about all of the capabilities of the system libraries.

# site.cfg
[atlas]
atlas_libs = 
library_dirs =

[blas]
blas_libs = cblas,acml
library_dirs = /opt/acml5.2.0/gfortan64_fma4/lib

[lapack]
blas_libs = cblas,acml
library_dirs = /opt/acml5.2.0/gfortan64_fma4/lib
>>> import numpy
>>> numpy.show_config()
blas_info:
    libraries = ['cblas', 'acml']
    library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
    language = f77
lapack_info:
    libraries = ['cblas', 'acml']
    library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['cblas', 'acml']
    library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['cblas', 'acml', 'cblas', 'acml']
    library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE
Owner

njsmith commented Dec 6, 2012

The question isn't so much "is there some arcane way to do this now that is
possible to learn with difficulty", it's "does this work in the best
possible way". The fact that you couldn't figure it out suggests not, so
there's something to fix. We just need to do due diligence to avoid
breaking some other weird use case in the process, since the build system
stuff is so arcane...

On Thu, Dec 6, 2012 at 6:26 PM, Bradley M. Froehle <notifications@github.com

wrote:

After some discussion in the mailing list, I have discovered that a
site.cfg which explicitly disables ATLAS would suffice. However this
leads to the other big issue which is that there is no way to tell the
build system that you have CBLAS (and hence build numpy.core._dotblas).
The pull request did an end-run around that by letting the user specify the
blas_opt and lapack_opt arguments very explicitly and in doing so carefully
inform numpy about all of the capabilities of the system libraries.

site.cfg

[atlas]
atlas_libs =
library_dirs =

[blas]
blas_libs = cblas,acml
library_dirs = /opt/acml5.2.0/gfortan64_fma4/lib

[lapack]
blas_libs = cblas,acml
library_dirs = /opt/acml5.2.0/gfortan64_fma4/lib

import numpy
numpy.show_config()
blas_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
language = f77
lapack_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
language = f77
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['cblas', 'acml', 'cblas', 'acml']
library_dirs = ['/opt/acml5.2.0/gfortan64_fma4/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE


Reply to this email directly or view it on GitHubhttps://github.com/numpy/numpy/pull/2751#issuecomment-11097400.

Contributor

bfroehle commented Dec 6, 2012

Yes, I know. And it's full of WTF's for first-timers. For example, define_macros=[('NO_ATLAS_INFO',1)] really means something like "CBLAS is not available." I didn't see that one coming. I think it'd be a lot clearer if it was phrased in the affirmative as define_macros=[('HAVE_CBLAS', 1), ('HAVE_ATLAS_INFO', 1)].

There are a few issues here, some of which overlap with one another:

  1. site.cfg is pretty misleading. It says things like the [atlas] sections are deprecated, but setting them was key to getting things to work. It also says you can just toss your config in the [blas_opt] and [lapack_opt] sections, but those are pretty much totally ignored. Also I don't think you can specify define_macros in the site.cfg file. This might be necessary to tell numpy about cblas, etc.
  2. The build system has no way to detect cblas. This has come up before (e.g., in the context of the debian build) but always just shrugged off as something that the maintainer can just patch themselves. I think either this patch needs to be documented (in site.cfg, perhaps), or we need a flag in site.cfg, or the build system should just test for a cblas function.
  3. Other packages might rely on the format of the site.cfg file, or at least the output of the sysconfig functions... need to be careful not to break any of them.
Owner

rgommers commented Dec 9, 2012

My preference would be to improve the documentation in site.cfg. Adding a new section/flag (say cblas_available) in a backwards compatible way could perhaps be useful, but changing how the existing sections work will likely be a problem for users with existing site.cfg's.

Contributor

bfroehle commented Dec 12, 2012

I've tried to update the documentation in site.cfg.example in pull request gh-2809. Let's move the discussion there for the moment and circle back here after we decide a resolution on that sub-issue.

Contributor

alimuldal commented Jan 18, 2013

It would also be great if site.cfg could explain how to force numpy to recognize OpenBLAS - I spent a long frustrating day before I figured out that I needed to put openblas under the [atlas] section!

Owner

charris commented May 5, 2013

@bfroehle Can you update this? @funcrusher Could you supply more information on what you did?

Contributor

bfroehle commented May 5, 2013

I don't think this is quite the right approach... :/ Closing for now.

@bfroehle bfroehle closed this May 5, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment