Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NumPy 1.22 support #7754

Closed
5 tasks
gmarkall opened this issue Jan 18, 2022 · 24 comments · Fixed by #8027
Closed
5 tasks

NumPy 1.22 support #7754

gmarkall opened this issue Jan 18, 2022 · 24 comments · Fixed by #8027

Comments

@gmarkall
Copy link
Member

gmarkall commented Jan 18, 2022

For tracking issues related to NumPy 1.22 support.

Known issues so far:

@esc
Copy link
Member

esc commented Jan 18, 2022

I ran a quick check on OSX using the current Numba 0.56.0dev0 and Numpy 1.22.1 (via pip).

 $ conda list
# packages in environment at /Users/esc/miniconda3/envs/numba-dev:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
ca-certificates           2021.10.26           hecd8cb5_2    defaults
certifi                   2021.10.8        py39hecd8cb5_2    defaults
intel-openmp              2021.4.0          hecd8cb5_3538    defaults
libcxx                    12.0.0               h2f01273_0    defaults
libffi                    3.3                  hb1e8313_2    defaults
llvm-openmp               12.0.0               h0dcd299_1    defaults
llvmlite                  0.39.0dev0              py39_21    numba/label/dev
mkl                       2021.4.0           hecd8cb5_637    defaults
mkl-service               2.4.0            py39h9ed2024_0    defaults
mkl_fft                   1.3.1            py39h4ab4a9b_0    defaults
mkl_random                1.2.2            py39hb2f4e1b_0    defaults
ncurses                   6.3                  hca72f7f_2    defaults
numba                     0.56.0dev0      np1.16py3.9h2fabc88_gf427bd6_22    numba/label/dev
numpy                     1.22.1                   pypi_0    pypi
openssl                   1.1.1m               hca72f7f_0    defaults
pip                       21.2.4           py39hecd8cb5_0    defaults
python                    3.9.7                h88f2d9e_1    defaults
readline                  8.1.2                hca72f7f_1    defaults
setuptools                58.0.4           py39hecd8cb5_0    defaults
six                       1.16.0             pyhd3eb1b0_0    defaults
sqlite                    3.37.0               h707629a_0    defaults
tk                        8.6.11               h7bc2e8c_0    defaults
tzdata                    2021e                hda174b7_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
xz                        5.2.5                h1de35cc_0    defaults
zlib                      1.2.11               h4dc903c_4    defaults
$ cat NUMPY_LOG.txt | grep ERROR | wc -l
70
cat NUMPY_LOG.txt | grep FAIL  | wc -l
10
 💣 zsh» cat NUMPY_LOG.txt | grep ERROR
ERROR: test_floordiv_float (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_floordiv_int (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_implicit_broadcasting (numba.tests.npyufunc.test_ufunc.TestUFuncs)
ERROR: test_nested_call (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuilding)
ERROR: test_nested_call_explicit (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuilding)
ERROR: test_nested_call_explicit (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuildingJitDisabled)
ERROR: test_nested_call (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuildingJitDisabled)
ERROR: test_broadcasting (numba.tests.npyufunc.test_ufunc.TestUFuncs)
ERROR: test_basic_ufunc (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuildingJitDisabled)
ERROR: test_basic_ufunc (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuilding)
ERROR: test_power_float (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_remainder_float (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_remainder_int (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_truediv_float (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_scalar (numba.tests.npyufunc.test_dufunc.TestDUFunc)
ERROR: test_restricted (numba.tests.npyufunc.test_dufunc.TestDUFuncPickling)
ERROR: test_power_integer (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_truediv_integer (numba.tests.npyufunc.test_errors.TestFloatingPointExceptions)
ERROR: test_ufunc_forceobj (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuilding)
ERROR: test_ufunc_attributes (numba.tests.npyufunc.test_ufunc.TestUFuncs)
ERROR: test_ufunc_raise (numba.tests.npyufunc.test_errors.TestExceptions)
ERROR: test_ufunc_forceobj (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuildingJitDisabled)
ERROR: test_ufunc_props (numba.tests.npyufunc.test_dufunc.TestDUFunc)
ERROR: test_ufunc_raise_objmode (numba.tests.npyufunc.test_errors.TestExceptions)
ERROR: test_ufunc_struct (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuilding)
ERROR: test_ufunc_struct (numba.tests.npyufunc.test_ufuncbuilding.TestUfuncBuildingJitDisabled)
ERROR: test_vectorize_objmode (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_vectorize_no_args (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_unrestricted (numba.tests.npyufunc.test_dufunc.TestDUFuncPickling)
ERROR: test_vectorize_bool_return (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_vectorize (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_3 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeJitted)
ERROR: test_3 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeDecor)
ERROR: test_1 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeDecor)
ERROR: test_2 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeDecor)
ERROR: test_1 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeJitted)
ERROR: test_2 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeJitted)
ERROR: test_4 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeJitted)
ERROR: test_4 (numba.tests.npyufunc.test_vectorize_decor.TestCPUVectorizeDecor)
ERROR: test_target_cpu_nopython (numba.tests.npyufunc.test_vectorize_decor.TestVectorizeNopythonArg)
ERROR: test_target_cpu_nopython_no_sig (numba.tests.npyufunc.test_vectorize_decor.TestVectorizeNopythonArg)
ERROR: test_vectorize (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_vectorize_bool_return (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_vectorize_no_args (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_vectorize_output_kwarg (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_vectorize_objmode (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_vectorize_only_kws (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecor)
ERROR: test_vectorize_only_kws (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_vectorize_output_kwarg (numba.tests.npyufunc.test_ufuncbuilding.TestVectorizeDecorJitDisabled)
ERROR: test_ufunc_and_dufunc_calls (numba.tests.test_array_exprs.TestArrayExpressions)
ERROR: test_assignments (numba.tests.test_dataflow.TestDataFlow)
ERROR: test_assignments2 (numba.tests.test_dataflow.TestDataFlow)
ERROR: test_chained_compare_npm (numba.tests.test_dataflow.TestDataFlow)
ERROR: test_chained_compare (numba.tests.test_dataflow.TestDataFlow)
ERROR: test_vectorize (numba.tests.test_enums.TestIntEnum)
ERROR: test_vectorize (numba.tests.test_fastmath.TestFastMath)
ERROR: test_vectorize_explicit_signature (numba.tests.test_npdatetime.TestMiscCompiling)
ERROR: test_gitdiff (numba.tests.test_runtests.TestCase)
ERROR: test_type_inference (numba.tests.test_vectorization_type_inference.TestVectTypeInfer)
ERROR: test_issue_651 (numba.tests.test_ufuncs.TestUfuncIssues)
ERROR: test_issue_713 (numba.tests.test_ufuncs.TestUfuncIssues)
ERROR: test_direct_dufunc_cache (numba.tests.npyufunc.test_caching.TestDUfuncCacheTest)
ERROR: test_indirect_dufunc_cache (numba.tests.npyufunc.test_caching.TestDUfuncCacheTest)
ERROR: test_direct_ufunc_cache (numba.tests.npyufunc.test_caching.TestUfuncCacheTest)
ERROR: test_direct_ufunc_cache_objmode (numba.tests.npyufunc.test_caching.TestUfuncCacheTest)
ERROR: test_indirect_ufunc_cache (numba.tests.npyufunc.test_caching.TestUfuncCacheTest)
ERROR: test_threading_concurrent_mix_use_masks (numba.tests.test_parallel_backend.TestParallelBackend)
ERROR: test_threading_concurrent_mix_use (numba.tests.test_parallel_backend.TestParallelBackend)
ERROR: test_random_concurrent_mix_use (numba.tests.test_parallel_backend.TestParallelBackend)
ERROR: test_random_concurrent_mix_use_masks (numba.tests.test_parallel_backend.TestParallelBackend)
 cat NUMPY_LOG.txt | grep "FAIL"
FAIL: test_threading_concurrent_mix_use_masks_omp (numba.tests.test_parallel_backend.TestSpecificBackend)
FAILED (errors=1)
FAIL: test_threading_concurrent_mix_use_omp (numba.tests.test_parallel_backend.TestSpecificBackend)
FAILED (errors=1)
FAIL: test_random_concurrent_mix_use_omp (numba.tests.test_parallel_backend.TestSpecificBackend)
FAILED (errors=1)
FAIL: test_random_concurrent_mix_use_masks_omp (numba.tests.test_parallel_backend.TestSpecificBackend)
FAILED (errors=1)
FAIL: test_lock (numba.tests.test_ufuncs.TestUFuncCompilationThreadSafety)
FAILED (failures=5, errors=66, skipped=748, expected failures=13)

NUMPY_LOG.txt

@gmarkall
Copy link
Member Author

@esc Thanks for posting the log - I've created issues for each distinct type of error and added them to this issue. It's quite probable that some of them have a common root cause, but I've considered them all separately in case there are distinct causes.

@esc
Copy link
Member

esc commented Jan 18, 2022

@gmarkall thank you! I imagine we can proceed to work on this once the Numpy 1.22 packages become available via https://anaconda.org/anaconda/numpy/files -- so that we can integrate this into our build farm and commence automated testing.

@2sn
Copy link

2sn commented Jan 19, 2022

After installing numba it downgrades numpy and then I get this error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
---------------------------------------------------------------------------

Maybe in future versions, numba install could be adjusted to not break other installations?

Possibly this upgrade round is a special situations, but it does have undesired side effects.

@esc
Copy link
Member

esc commented Jan 19, 2022

After installing numba it downgrades numpy and then I get this error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
---------------------------------------------------------------------------

Maybe in future versions, numba install could be adjusted to not break other installations?

Possibly this upgrade round is a special situations, but it does have undesired side effects.

@2sn thank you for your report. Which Python version are you using and on which platform did this occur? This error message is reminiscent of the one reported (and fixed) at: #7747 can you check if that fixes it for you too since this may now be a duplicate.

@2sn
Copy link

2sn commented Jan 19, 2022

I use current release versions on Pypi, including Python 3.10.1.

What happens is as follows: FIrst Numpy 1.22.x was installed, then numba numba becomes available, it downgrades numpy to 1.21.5, which has a different API version, but some things were already compiled with the new version and cached. After de-installing numpy and numba, deleting cache, re-installation from Pypi, then it works. Just not out of the box.

@esc
Copy link
Member

esc commented Jan 19, 2022

I use current release versions on Pypi, including Python 3.10.1.

What happens is as follows: FIrst Numpy 1.22.x was installed, then numba numba becomes available, it downgrades numpy to 1.21.5, which has a different API version, but some things were already compiled with the new version and cached. After de-installing numpy and numba, deleting cache, re-installation from Pypi, then it works. Just not out of the box.

Thank you for your response @2sn . It sounds a bit weird. I am glad you managed to workaround the issue. If the issue returns or persists, please feel free to open a new issue with details, thank you!

@2sn
Copy link

2sn commented Jan 19, 2022

I posted it in the hope the solution may be useful to others. I have seen it on both, my linux install and on M1/MacOS 12.1.

For me it occured because I had already installed numpy 1.22.x before installing numba, which downgraded it, maybe still used the newer numpy version to compile numba.

@seibert
Copy link
Contributor

seibert commented Jan 19, 2022

Unfortunately, Numba doesn't have control over how package managers (in this case, pip) decide to use Numba's version constraint on NumPy. In this case, it sounds like pip downgraded NumPy without checking if the downgraded NumPy was still compatible with the other packages in your environment.

Hopefully we can get NumPy 1.22 support working for Numba 0.56.

@2sn
Copy link

2sn commented Jan 19, 2022

Whereas not sure, I think what must have happened is that it used the installed numpy to compile numba - which was 1.22 - before replacing it.

@esc esc mentioned this issue Jan 30, 2022
devmessias added a commit to devmessias/mlops_project that referenced this issue Feb 24, 2022
Numba dosen't provide support for the new 1.22 numpy versions.
Thus, some pkgs like `shap` canno't be used whitout a proper
downgrade. This downgrade should removed after numba team
close the issue : (numba/numba/issues/7754)[numba/numba#7754]
devmessias added a commit to devmessias/mlops_project that referenced this issue Feb 24, 2022
Numba does not support the new version of NumPy (1.22).
So some packages like `shap` can\'t be used without downgrading.

The downgrade should be removed after a new release of the Numba pkg
with the following issue addressed.

[numba/numba/issues/7754](numba/numba#7754)
@rgommers
Copy link

rgommers commented Mar 24, 2022

In this case, it sounds like pip downgraded NumPy without checking if the downgraded NumPy was still compatible with the other packages in your environment.

This is not actually 100% Pip's fault. The packages that were built against NumPy 1.22.0 should have an install_requires = {'numpy>=1.22'} constraint, but packages typically don't add those for from-source builds (even SciPy doesn't, it's on my todo list) because it's hard to do. Wheel build machinery has no functionality that lets one do this automatically. What is needed there is the equivalent of conda-forge's {{ pin_compatible('numpy') }} (see docs) but added automatically for pip install . or similar, rather than through a build recipe.

@2sn
Copy link

2sn commented Mar 24, 2022

Issue arise when there is competing requirements. Numba seems to typically lag behind 1-2 versions in NumPy, and maybe one version in Python support. It seems a lot of effort to keep current just by volunteers.

For me, I decided that this makes Numba not usable for my practical purposes. A nice tech demo, though. Well done!

@stonebig
Copy link
Contributor

Things should improve with the coming data-apis standard

@GuillemGSubies
Copy link

Any updates here? I would really appreciate having the compatibility with numpy 1.22

@aelzenaar
Copy link

Any updates here? I would really appreciate having the compatibility with numpy 1.22

Seems to work for me (running the dev versions of both numba and numpy)

@GuillemGSubies
Copy link

Any updates here? I would really appreciate having the compatibility with numpy 1.22

Seems to work for me (running the dev versions of both numba and numpy)

That's great! Are you planning a stable release soon?

@aelzenaar
Copy link

I'm not a developer of numba, sorry

@blortuga
Copy link

This issue really needs to be resolved, because we can not deploy numpy versions earlier than 1.22.3 given:

https://snyk.io/vuln/pip%3Anumpy

@rgommers
Copy link

This issue really needs to be resolved, because we can not deploy numpy versions earlier than 1.22.3 given:

I know you probably have no choice in the matter because you're bound by company policies, but just in case: those CVEs are all basically garbage. We (the NumPy team) are disputing and rejecting them where we can. You'd be fine ignoring them.

@blortuga
Copy link

Thank you, @rgommers.
CVE-2021-34141 is really the main one that's marked as being concerning. I'm allowed to justify these where the package vendor has a good case, and it's argued well here: numpy/numpy#18993 . So that helps me to get my job done. :D
(whether the security team accepts those arguments is another matter but the NIST downgrade on this one in particular is pretty compelling. Again; thank you.)

@etmyhome
Copy link

etmyhome commented May 3, 2022

any update?

@stuartarchibald
Copy link
Contributor

Locally I've got a NumPy 1.22 support patch that's passing all tests on x86_64 linux. The next release of Numba (0.56) will have support for NumPy 1.22.

@ruifeng96150
Copy link

ruifeng96150 commented May 5, 2022

any schedule to release 0.56?

@stuartarchibald
Copy link
Contributor

It is hoped that a release candidate (RC) for 0.56 will be produced in about two weeks' time (sometime around the middle of May 2022). There will then be the usual RC testing phase which is usually (order of) a few weeks, during which more RCs may need to be produced to fix any issues found and then these will also need to be tested. Release candidate and general release announcements etc will be made on Numba's discourse forum https://numba.discourse.group/, in the "Announcements" section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.