configuring numpy 1.16.5 fails on FreeBSD with OpenBlas #15389

rsmith-nl · 2020-01-22T20:27:54Z

This issue can also be found in the FreeBSD bugzilla.
There it was suggested to open an issue here.

TL;DR

Problem: Compiling py37-numpy-1.16.5_3,1 with OpenBlas fails in numpy/distutils/system_info.py with an uncaught distutils.errors.LinkError exception.
Solution: also catch distutils.errors.LinkError on line 1742 of numpy/distutils/system_info.py.

This is the error (only the last part shown for clarity).

File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1740, in has_cblas
    extra_postargs=info.get('extra_link_args', []))
File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
    raise LinkError(msg)
distutils.errors.LinkError: Command "cc /tmp/tmpm3gnb0ev/tmp/tmpm3gnb0ev/source.o -L/usr/local/lib -lblas -o /tmp/tmpm3gnb0ev/a.out" failed with exit status 1
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/math/py-numpy
*** Error code 1

Stop.
make: stopped in /usr/ports/math/py-numpy

This is the relevant code from system_info.py:

    try:
        with open(src, 'wt') as f:
            f.write(s)

        try:
            # check we can compile (find headers)
            obj = c.compile([src], output_dir=tmpdir,
                            include_dirs=self.get_include_dirs())

            # check we can link (find library)
            # some systems have separate cblas and blas libs. First
            # check for cblas lib, and if not present check for blas lib.
            try:
                c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                  libraries=["cblas"],
                                  library_dirs=info['library_dirs'],
                                  extra_postargs=info.get('extra_link_args', []))
                res = "cblas"
            except distutils.ccompiler.LinkError:
                c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                  libraries=["blas"],
                                  library_dirs=info['library_dirs'],
                                  extra_postargs=info.get('extra_link_args', []))
                res = "blas"
        except distutils.ccompiler.CompileError:
            res = None
    finally:
        shutil.rmtree(tmpdir)
    return res

If linking with “cblas” fails, it tries again by linking with “blas”.
However, the last “except” only handles a compilation error but not a linking error.

When we change:

        except distutils.ccompiler.CompileError:

to

        except (distutils.ccompiler.CompileError, distutils.ccompiler.LinkError):

the problem is fixed.

Reproducing code example:

On a FreeBSD system with an up-to-date ports tree installed:

cd /usr/ports/math/py-numpy
make

Error message:

creating /tmp/tmpoquazeyp/tmp
creating /tmp/tmpoquazeyp/tmp/tmpoquazeyp
compile options: '-I/usr/include -I/usr/local/include -I/usr/local/include/suitesparse -c'
cc: /tmp/tmpoquazeyp/source.c
cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out
ld: error: unable to find library -lcblas
cc: error: linker command failed with exit code 1 (use -v to see invocation)
cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out
ld: error: undefined symbol: cblas_ddot
>>> referenced by source.c
>>>               /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o:(main)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 204, in link
    self.spawn(linker + ld_args)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 92, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 172, in CCompiler_spawn
    (cmd, s, msg))
distutils.errors.DistutilsExecError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1734, in has_cblas
    extra_postargs=info.get('extra_link_args', []))
File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
    raise LinkError(msg)
distutils.errors.LinkError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 204, in link
    self.spawn(linker + ld_args)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 92, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 172, in CCompiler_spawn
    (cmd, s, msg))
distutils.errors.DistutilsExecError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "setup.py", line 419, in <module>
    setup_package()
File "setup.py", line 411, in setup_package
    setup(**metadata)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/core.py", line 137, in setup
    config = configuration()
File "setup.py", line 164, in configuration
    config.add_subpackage('numpy')
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1036, in add_subpackage
    caller_level = 2)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1005, in get_subpackage
    caller_level = caller_level + 1)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 942, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
File "numpy/setup.py", line 10, in configuration
    config.add_subpackage('core')
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1036, in add_subpackage
    caller_level = 2)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1005, in get_subpackage
    caller_level = caller_level + 1)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 942, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
File "numpy/core/setup.py", line 760, in configuration
    blas_info = get_info('blas_opt', 0)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 448, in get_info
    return cl().get_info(notfound_action)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 641, in get_info
    self.calc_info()
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1661, in calc_info
    blas_info = get_info('blas')
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 448, in get_info
    return cl().get_info(notfound_action)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 641, in get_info
    self.calc_info()
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1697, in calc_info
    lib = self.has_cblas(info)
File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1740, in has_cblas
    extra_postargs=info.get('extra_link_args', []))
File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
    raise LinkError(msg)
distutils.errors.LinkError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/math/py-numpy
*** Error code 1

Stop.
make: stopped in /usr/ports/math/py-numpy

Numpy/Python version information:

import sys, numpy; print(numpy.version, sys.version)
1.16.5 3.7.6 (default, Dec 23 2019, 18:53:23)
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]

(the above versions were generated with a fixed build of numpy.)

The text was updated successfully, but these errors were encountered:

mattip · 2020-01-22T21:08:50Z

Seems reasonable. Would you like to open a PR? It should have the label "BLD: catch link error" or so.

rsmith-nl · 2020-01-23T17:22:08Z

Sure. Which branch whould I base the PR on? maintenance/1.16.x or master?

seberg · 2020-01-23T17:49:00Z

@rsmith-nl unless it only affects a release version, everything goes through master first and is then backported.

When system_info.py is trying to find cblas as libblas, there can be a different library called libblas that doesn't contain the referenced function cblas_ddot. This raises a distutils.errors.LinkError, which is currently not caught. This patch fixes numpy#15389. The system_info in master was reworked for 1.17, and doesn't have this issue.

rsmith-nl · 2020-01-23T20:15:20Z

The master branch's system_info.py was reworked and doesn't have this bug. So I based my PR on the 1.16 maintenance branch.

charris · 2020-01-23T20:37:24Z

@rsmith-nl Master will be Numpy 1.19, and we don't plan any more releases for 1.16 or 1.17. If the bug is still in 1.18, there will be a 1.18.2, so that is likely the first official release that could have the fix. For earlier releases I would suggest FreeBSD maintain their own patch. I might could be talked into another 1.16 if FreeBSD really wants to support the last Python2.7 release.

rsmith-nl · 2020-01-23T23:50:10Z

On Thu, Jan 23, 2020 at 12:37:25PM -0800, Charles Harris wrote: ***@***.*** Master will be Numpy 1.19, and we don't plan any more releases for 1.16 or 1.17. If the bug is still in 1.18, there will be a 1.18.2, so that is likely the first official release that could have the fix.

Since system_info.py was reworked in 1.17, only 1.16 has this issue.

For earlier releases I would suggest FreeBSD maintain their own patch.

It was just merged into the ports tree. :-)

I might could be talked into another 1.16 if FreeBSD really wants to support the last Python2.7 release.

As luck would have it, that issue was raised today on the FreeBSD bugzilla. In the FreeBSD ports tree, python 2.7 will expire at the end of this year. (It has been marked as deprecated already.) Since we support building numpy with three different blas libraries (atlas, netlib and openblas), this is not one of our easiest ports. Keeping numpy in the ports tree at 1.16 until python 2.7 expires is probably the easiest thing to do. We'll see how the discussion goes. Roland -- R.F.Smith http://rsmith.home.xs4all.nl/ [plain text email much appreciated] https://github.com/rsmith-nl/

charris · 2020-01-24T01:14:03Z

I went ahead and merged your patch, you can always build from the branch if there is not another release of 1.16. Maintenance of 1.16 is available if anyone is interested.

Closing this but feel free to initiate more discussion.

charris · 2020-01-24T01:26:06Z

Note that NumPy 1.19 will not support Python3.5.

rsmith-nl mentioned this issue Jan 23, 2020

BLD: catch link error. #15406

Merged

charris closed this as completed Jan 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

configuring numpy 1.16.5 fails on FreeBSD with OpenBlas #15389

configuring numpy 1.16.5 fails on FreeBSD with OpenBlas #15389

rsmith-nl commented Jan 22, 2020

mattip commented Jan 22, 2020

rsmith-nl commented Jan 23, 2020

seberg commented Jan 23, 2020

rsmith-nl commented Jan 23, 2020

charris commented Jan 23, 2020

rsmith-nl commented Jan 23, 2020 via email

charris commented Jan 24, 2020

charris commented Jan 24, 2020

configuring numpy 1.16.5 fails on FreeBSD with OpenBlas #15389

configuring numpy 1.16.5 fails on FreeBSD with OpenBlas #15389

Comments

rsmith-nl commented Jan 22, 2020

Reproducing code example:

Error message:

Numpy/Python version information:

mattip commented Jan 22, 2020

rsmith-nl commented Jan 23, 2020

seberg commented Jan 23, 2020

rsmith-nl commented Jan 23, 2020

charris commented Jan 23, 2020

rsmith-nl commented Jan 23, 2020 via email

charris commented Jan 24, 2020

charris commented Jan 24, 2020