Support for NumPy 1.22

[stuartarchibald]

As title.

Fixes #7756
Fixes #7757
Fixes #7759
Fixes #7760
Fixes #7758

Closes #7754

[sklam]

LGTM! Guess we just need it go through the buildfarm

[sklam]

BFID numba_np122_yaml_1

[stuartarchibald]

I think there's a reasonable chance that this will need patching:

numba/numba/tests/test_array_reductions.py

Lines 688 to 693 in 0f5953d

	# Test with a NaT
	if numpy_version != (1, 21) and 'median' not in pyfunc.__name__:
	# There's problems with NaT handling in "median" on at least NumPy
	# 1.21.{3, 4}. See https://github.com/numpy/numpy/issues/20376
	arr[arr.size // 2] = 'NaT'
	self.assertPreciseEqual(cfunc(arr), pyfunc(arr))

[stuartarchibald]

I opened #8030 to get NumPy 1.22 running on more systems. A test failed on a skylake chip (similar to the observed failure pattern in the Numba build farm) with a small numerical error (couple of ULPs).

Having looked at the NumPy 1.21-1.22 change log and diff. I suspect this is the cause: numpy/numpy#19478. From what I understand from skimming the patch and discussion, on chips with AVX-512 support, NumPy now links to a vendored-in copy of the SVML library and uses the low precision variant of a number of transcendental and trigonometric functions in the ufunc loops. These variants have a maximum declared error of 4 ULP, which may well explain why testing functions such as:

numba/numba/tests/test_array_exprs.py

Lines 67 to 69 in 0f5953d

	def explicit_output(a, b, out):
	np.cos(a, out)
	return np.add(out, b, out)

are failing like:

FAIL: test_explicit_output (numba.tests.test_array_exprs.TestArrayExpressions)
Check that ufunc calls with explicit outputs are not rewritten.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<path>/numba/tests/test_array_exprs.py", line 387, in test_explicit_output
    ns = self._test_explicit_output_function(explicit_output)
  File "<path>/numba/tests/test_array_exprs.py", line 247, in _test_explicit_output_function
    self.assertPreciseEqual(expected, run_func(control_cfunc))
  File "<path>/numba/tests/support.py", line 387, in assertPreciseEqual
    self.fail("when comparing %s and %s: %s" % (first, second, failure_msg))
AssertionError: when comparing [2.         2.54030231 2.58385316 3.0100075  4.34635638 6.28366219
 7.96017029 8.75390225 8.85449997 9.08886974] and [2.         2.54030231 2.58385316 3.0100075  4.34635638 6.28366219
 7.96017029 8.75390225 8.85449997 9.08886974]: 8.753902254343306 != 8.753902254343304

As far as I can tell NumPy does not have the capacity from user code or otherwise to switch fastmath behaviours on and off cf. Numba, as a result I think the 4 ULP errors in these functions when running on micro-architectures with AVX-512 are going to be something that the test suite/Numba will have to work around.

[stuartarchibald]

After doing some trials with #8030 (comment). Have added 1ca5b90 to switch off NumPy's AVX512 SVML ufunc dispatch mechanics.

[sklam]

New changes looks good.

Needs to smoke it in the farm once public CI is done.

[stuartarchibald]

@sklam I had to add c5a3010. The failure of test_np_MachAr_deprecation_np122 on the farm where no warning was being produced was down to:

1. The warning being raised in the typing part of the @overload(np.MachAr) such that it should appear once the first time the function is compiled.
2. The @overload(np.MachAr) being used somewhere else in the test suite which did raise the warning, but also cached the result of compilation against the no-arg call type.
3. Attempting to later trigger the warning by calling a function using np.MachAr fails because the in memory cache replay as a result of 2. is used as there's always going to be a cache hit for no-arg function after the first encounter.

[sklam]

BFID numba_np122_yaml_6

[sklam]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[sklam]

c5a3010 make sense.

btw, to avoid this in the future, we should support warnings in nopython.

[stuartarchibald]

BFID numba_np122_yaml_6

This passed everywhere except linux aarch64 Python 3.8 NumPy 1.22. The failing test is: numba.tests.test_linalg.TestLinalgEigenSystems.test_linalg_eig which fails with 1/100 elements not matching, expected: rtol=5e-15, atol=1e-15, got: rtol=1.7e-12, atol=8e-15.

I think it's reasonably likely that this is something to do with the BLAS and/or LAPACK implementation in use opposed to Numba, as all Numba does for np.linalg support is effectively wire pointers through to these libraries. The specific check that's failing is here:

numba/numba/tests/test_linalg.py

Lines 974 to 980 in 4684b9c

	if np.iscomplexobj(v):
	np.testing.assert_allclose(
	lhs.imag,
	rhs.imag,
	rtol=resolution,
	atol=resolution
	)

I'd guess it's a complex64 input to that which is causing the problem as that data type has a history of being problematic in these tests on ARM chips.

[stuartarchibald]

Running numba_smoketest_cpu_yaml_88

Note: this is running against c5a3010

[stuartarchibald]

This branch at 812a1d7 has a test matrix distribution:

$ python -c 'from numba.tests.support import print_azure_matrix; print_azure_matrix()'
NumPy | Python | Count
-----------------------
 1.18 |  3.7   |   4
 1.18 |  3.8   |   1
 1.19 |  3.8   |   3
 1.19 |  3.9   |   1
 1.20 |  3.8   |   3
 1.20 |  3.9   |   1
 1.21 |  3.9   |   1
 1.21 |  3.10  |   1
 1.22 |  3.10  |   3
 1.22 |  3.8   |   1
 1.22 |  3.9   |   1

[stuartarchibald]

For reviewers:

• d6d5296 reverts a previous commit that added NumPy 1.22 via pip for testing purposes.
• 02856d8 adds a small utility function to parse the Azure config and produce a table of the matrix cf. those I've pasted above.
• 812a1d7 updates the Azure config to include NumPy 1.22 and more evenly distribute the Python/NumPy combinations.

[stuartarchibald]

Running numba_smoketest_cpu_yaml_88

Note: this is running against c5a3010

This passed.

[sklam]

LGTM

[drawks]

When can we expect a new release version to be cut that includes this change?

[esc]

When can we expect a new release version to be cut that includes this change?

we will attempt this today (Thu 19th May 2022)

[esc]

When can we expect a new release version to be cut that includes this change?

we will attempt this today (Thu 19th May 2022)

Unfortunately there were some issues backporting which took longer than expected. We now hope to have packages available by Monday 23 May 🤞

[SkandanC]

When can we expect a new release version to be cut that includes this change?

we will attempt this today (Thu 19th May 2022)

Unfortunately there were some issues backporting which took longer than expected. We now hope to have packages available by Monday 23 May 🤞

Hello, any chances the packages will be released today?

Thank you.

[esc]

When can we expect a new release version to be cut that includes this change?

we will attempt this today (Thu 19th May 2022)

Unfortunately there were some issues backporting which took longer than expected. We now hope to have packages available by Monday 23 May 🤞

Hello, any chances the packages will be released today?

Thank you.

please subscribe to: #8066