Numpy update with OpenBLAS and hypothesis test depends #883
Conversation
|
Missing updated numpy-py.patch. |
nieder
added
waiting for reply
Doh, looking too hard only if I missed any dependency .infos... |
|
It's fine to force SHELL=/bin/sh, either through a patch or by adding it to the test command in TestScript. |
dhomeier
force-pushed
the
numpy-hypothesis-openblas
branch
2 times, most recently
from
5eb025f
to
ec18743
Compare
|
Build failure on 12.1. I'll try with 10.14.5 in a bit: NPY_BLAS_ORDER=openblas,accelerate NPY_LAPACK_ORDER=openblas,accelerate ATLAS=None /opt/sw/bin/python3.8 setup.py build
Running from numpy source directory.
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/bit_generator.pyx
Processing numpy/random/mtrand.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
INFO: blas_opt_info:
INFO: openblas_info:
INFO: customize UnixCCompiler
INFO: C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-nullability-completeness -Wno-expansion-to-defined -mtune=intel -I/opt/sw/include
creating /private/tmp/tmpoychteav/private
creating /private/tmp/tmpoychteav/private/tmp
creating /private/tmp/tmpoychteav/private/tmp/tmpoychteav
INFO: compile options: '-c'
INFO: gcc: /private/tmp/tmpoychteav/source.c
error: unknown target CPU 'intel'
note: valid target CPU values are: i386, i486, winchip-c6, winchip2, c3, i586, pentium, pentium-mmx, pentiumpro, i686, pentium2, pentium3, pentium3m, pentium-m, c3-2, yonah, pentium4, pentium4m, prescott, nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client, icelake-server, tigerlake, sapphirerapids, alderlake, knl, knm, lakemont, k6, k6-2, k6-3, athlon, athlon-tbird, athlon-xp, athlon-mp, athlon-4, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, znver3, x86-64, geode
Traceback (most recent call last):
File "setup.py", line 461, in <module>
setup_package()
File "setup.py", line 453, in setup_package
setup(**metadata)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/core.py", line 135, in setup
config = configuration()
File "setup.py", line 131, in configuration
config.add_subpackage('numpy')
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 1054, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 1020, in get_subpackage
config = self._get_configuration_from_setup_py(
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 962, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "numpy/setup.py", line 9, in configuration
config.add_subpackage('core')
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 1054, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 1020, in get_subpackage
config = self._get_configuration_from_setup_py(
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/misc_util.py", line 962, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "numpy/core/setup.py", line 813, in configuration
blas_info = get_info('blas_opt', 0)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 585, in get_info
return cl().get_info(notfound_action)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 845, in get_info
self.calc_info()
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2077, in calc_info
if self._calc_info(blas):
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2063, in _calc_info
return getattr(self, '_calc_info_{}'.format(name))()
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2000, in _calc_info_openblas
info = get_info('openblas')
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 585, in get_info
return cl().get_info(notfound_action)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 845, in get_info
self.calc_info()
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2283, in calc_info
info = self._calc_info()
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2271, in _calc_info
if not (skip_symbol_check or self.check_symbols(info)):
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/system_info.py", line 2350, in check_symbols
obj = c.compile([src], output_dir=tmpdir)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/ccompiler.py", line 89, in <lambda>
m = lambda self, *args, **kw: func(self, *args, **kw)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/ccompiler.py", line 365, in CCompiler_compile
single_compile(o)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/ccompiler.py", line 325, in single_compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/ccompiler.py", line 89, in <lambda>
m = lambda self, *args, **kw: func(self, *args, **kw)
File "/opt/sw/build.build/numpy-py38-1.22.2-1/numpy-1.22.2/numpy/distutils/unixccompiler.py", line 58, in UnixCCompiler__compile
raise CompileError(msg) from None
distutils.errors.CompileError: Command "gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-nullability-completeness -Wno-expansion-to-defined -mtune=intel -I/opt/sw/include -c /private/tmp/tmpoychteav/source.c -o /private/tmp/tmpoychteav/private/tmp/tmpoychteav/source.o -MMD -MF /private/tmp/tmpoychteav/private/tmp/tmpoychteav/source.o.d" failed with exit status 1
### execution of NPY_BLAS_ORDER=openblas,accelerate failed, exit code 1
### execution of /tmp/fink.33msp failed, exit code 1 |
|
numpy-py38/39/310-1.22.2 passed all tests on 10.14.5, with |
|
At least on macOS12 (clang 13), mtune does not accept "intel". SetCFLAGS: -mtune=intel
+SetCC: gcc-fsf-%type_raw[gcc]
+SetCXX: g++-fsf-%type_raw[gcc]Just unsetting mtune leads to two failed tests: test_mem_policy.py::test_new_policy and test_ufunc.py::TestUfuncGenericLoops::test_unary_PyUFunc_O_O_method_full[reciprocal] |
|
The update-alternatives for hypothesis-pyXX owner got changed from |
|
Sorry, thought "generic name" always referred to the link name, but I obviously overlooked the nonstandard usage in the old version! |
Ah, I had in fact been wondering what to do with that if Fink some day runs on M1...
Building any recent numpy with gcc at least since gcc-9 has failed for me with AVX512-related errors like I thought I had seen some old regressions on related gcc/assembler bugs, but could not find anything right now; Alternatively for clang one could make the mtune-option system-dependent, or just use |
|
Maybe it's not as simple as incorrectly detecting AVX512; although and reports on optimisation (even with the clang build produces exactly the same, and successfully compiles all that avx512 custom code. In fact gcc11 also compiled and only failed on |
|
Still more interesting, or confusing, I am getting the same failure with gcc10, but not with 9.4 – optimisation reports looking nearly identical, but gcc9 lists one extra file... Xcode clang reports the same, and both are building in |
My macOS12 in on intel. To get a fink install working on 11.0 and up (intel), you need to checkout the dpkg1.16 branch from https://github.com/TheSin-/fink |
Thanks, that's not a place I would have found on my own. Seems to work overall, but somewhere along the upgrade to now dpkg 1.19.7 it appears the script versions have been moved to |
|
install-info should never be called at all anymore, it's a trigger now. see "Deprecate InfoDocs field (add a validator warning that it is deprecated, the triggers in install-info supersede it)" in fink/fink#66 |
|
Is that a regression against fink-0.45.99.git-20220228.1334 then? |
dhomeier
force-pushed
the
numpy-hypothesis-openblas
branch
from
7fb05f3
to
4ea016f
Compare
I confirmed that gcc11 works on Monterey, but keep failing with the AVX512 errors on Mojave. |
|
I didn't notice that you force pushed changes. I tested the current state of the branch and all the hypothesis built (but requires itself to be installed to also pass tests). numpy-py37-py310 built and passed tests on 10.14.5. py36 failed as described in numpy/numpy#10515, but adding On 12.5.1, numpy-py38-310 built but failed these two tests for all variants: |
See numpy/numpy#10515 fink-bld has a minimal environment and TestFromTxt.test_utf8_file_nodtype_unicode requires more than plain ASCII.
|
Since hypothesis-py is working, I've cherry picked it's commits to master. This only leaves the two failing numpy tests on 12.5.1. If we think those are not necessary we can disable them and push through numpy and finish this. |
|
Tested, numpy-py 1.22.4 (last of the 1.22 series), and it failed the same two tests on 12.5. Version 1.23.3 needs new package typing_extensions and a bump to the most recent cython. |
|
I got typing-extensions-py working on my 12.5 machine (needs several new packages, will make a PR), but numpy-py-1.23.3 fails the same tests: |
|
I've pushed the new typing_extensions package; it depends on your |
|
I'm getting 404 on the test_typing_extensions.py file in TestDepends. Are there needed changes to it that are not part of the 4.4.0 tarball? |
|
Also, getting this error in TestScript if I ignore the TestSource: |
|
And, I'm getting this failure then: See python/typing_extensions#61 I used Debian's patch to get the missing module that's in upstream's github but not in the pypi tarball. |
|
Thanks! That test file was still left in my Fink sources from a pre-4.3.0 release, which had no tests included at all. |
Co-authored-by: <nieder@users.noreply.github.com>
dhomeier
force-pushed
the
numpy-hypothesis-openblas
branch
from
632f39f
to
d519137
Compare
|
typing-extensions now works. Test pass on 10.14.5. |
|
Opened numpy/numpy#22680 for the two tests on 12.5.1. Not sure they're big errors, and it might be possible to ignore them, but we'll see what they say. |
|
1.24.1 now has 19 test failures on 10.14.6 (numpy-1.24.0 passes all; other macOS untested). I thought numpy/numpy#22954 might take care of most of them even though the bug is for arm64, but it didn't help. |
|
numpy-1.24.0 passes all tests on 10.14.6 and fails just the same two tests on 12.6 (no response upstream to the bug I filed). Should we just skip those two tests? |
|
numpy-1.25.0 is out. It only supports py3.9-3.11. 1 failure in test_mem_policy.py All the failures in test_umath.py end like this: So is it failing because a (compiler?) warning was thrown? There's no actual warning in the terminal output. |
The warning is kept (intercepted) in the Cannot test on 10.14 or any native x86_64 atm, but what do you get e.g. when running python3.9 -c 'import numpy as np; array = np.array([0.5, 0.5, 0.5, np.nan], dtype="f"); print(np.cos(array))'? |
|
Also reproducing the two failures from #883 (comment) on 13.4/x86_64, plus for |
|
This is on 10.14.6: Using 1.25.0: |
|
Using our new openblas 0.3.26, numpy-py310-1.25.0 still fails those 19 tests: |
Running the hypothesis tests with tox installed failed on
In addition the PyPI source does not contain any tests that could be run, and tox would likely try to pip-install its own dependencies even if got to run, apart from the circular dependency.
I've therefore switched to the github source and running pytest directly. This still involves a circular dependency with numpy, the presence of which is now checked instead.
Numpy can in fact now be built against Accelerate again (see notes), but OpenBLAS should still be preferable; also the latter is detected and selected automatically, if present. So I made that a [Build]Depends and the preferred
NPY_BLAS– otherwise a deterministic build against Accelerate would require deinstallation ofopenblasduring the build.The tests are also running on pytest; the nose-based
runtest.pyhas not been recommended in numpy for some time, and appears to be omitting a number of tests compared to pytest. Pytest also prints clearer results. Tested on 10.14 and in mostly the same configuration on 10.13.