Upgrade numpy supported versions

[josephevans]

Based on vulnerabilities in the numpy library, we need to update the support versions of numpy.

I am current working on a PR to update numpy (which requires Python and other dependencies to be upgraded as well.)

Current Status:

The following tests are failing in CI and need code updates or tests updated to work with newer numpy versions:

• test_np_array_function_protocol
  • test_numpy_interoperability.test_np_array_function_protocol
  • test_operator_gpu.test_np_array_function_protocol
• Failing test with numpy upgrade - test_np_delete() #20880
  • test_numpy_op.test_np_delete
  • test_operator_gpu.test_np_delete
• Failing test with numpy upgrade - test_np_get_dtype() #20886
  • test_numpy_ndarray.test_np_get_dtype
  • test_operator_gpu.test_np_get_dtype
• test_operator_gpu.test_np_percentile
  • fails for midpoint test case, increasing tolerances passes test, see 13d9ae0
• test_np_quantile
  • fails for midpoint test case, increasing tolerances passes test, see 13d9ae0
  • test_numpy_op.test_np_quantile
  • test_operator_gpu.test_np_quantile

Details:

We currently pin down numpy versions in our setup.py script:

master: numpy<2.0.0,>1.16.0 (see https://github.com/apache/incubator-mxnet/blob/master/tools/pip/setup.py#L62)
v1.9.x: numpy<2.0.0,>1.16.0 (see https://github.com/apache/incubator-mxnet/blob/v1.9.x/tools/pip/setup.py#L63)

Our CI system, however, pins down specific versions:

master: numpy>=1.17,<1.20.0 (see https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/requirements#L22)
v1.9.x: numpy>1.16.0,<1.19.0 (see https://github.com/apache/incubator-mxnet/blob/v1.9.x/ci/docker/install/requirements#L29)

The are vulnerabilities reported in numpy versions currently used in our CI system.

https://nvd.nist.gov/vuln/detail/CVE-2021-41495
https://nvd.nist.gov/vuln/detail/CVE-2021-41496

When attempting to upgrade the version of numpy (and scipy), the following tests are failing:

centos-cpu pipeline failing tests (example: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-20872/3/pipeline):

• test_numpy_interoperability.test_np_array_function_protocol
• test_numpy_op.test_np_quantile
• test_numpy_interoperability.test_np_fallback_ops
• test_numpy_op.test_np_delete
• test_numpy_ndarray.test_np_get_dtype
• test_numpy_op.test_np_linspace

windows-cpu pipeline failing tests (example https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-cpu/detail/PR-20872/3/pipeline):

• test_numpy_interoperability.test_np_array_function_protocol
• test_numpy_interoperability.test_np_fallback_ops
• test_numpy_op.test_np_delete
• test_numpy_ndarray.test_np_get_dtype
• test_numpy_op.test_np_linspace

unix-cpu pipeline failing tests (example: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20872/3/pipeline/310)

• test_numpy_interoperability.test_np_array_function_protocol
• test_numpy_interoperability.test_np_fallback_ops
• test_numpy_op.test_np_delete
• test_numpy_ndarray.test_np_get_dtype
• test_numpy_op.test_np_linspace

These tests are failing, which are scipy-related, as discussed in #20389. Pinning down the scipy version makes these tests pass:

• test_numpy_op.test_np_rand
• test_numpy_op.test_np_randint
• test_random.test_normal_generator
• test_random.test_uniform_generator
• test_random.test_exponential_generator
• test_random.test_poisson_generator
• test_random.test_negative_binomial_generator
• test_random.test_multinomial_generator
• test_random.test_randint_generator
• test_operator_gpu.test_randint_generator
• test_operator_gpu.test_np_randint
• test_operator_gpu.test_multinomial_generator

[josephevans]

I have created a draft PR (#20872) to work on updating, will update the list of failing tests in this issue as more information is collected.

[huubvh95]

Is it also possible to update Graphviz in the PR? The required version for Graphviz is rather old.
See issue #20897

[szha]

It looks like the issue has been resolved in the linked PR by @DickJC123 for v1.x, v1.9.x, and master branch.