test_cond_cpu tests fail when running with numpy compiled against OpenBLAS 0.3.15

[casparvl]

🐛 Bug

The following tests fail in the PyTorch 1.10.0 test suite when using numpy compiled with OpenBLAS 0.3.15:

ERROR: test_cond_cpu_complex128 (__main__.TestLinalgCPU)
ERROR: test_cond_cpu_complex64 (__main__.TestLinalgCPU)
ERROR: test_cond_cpu_float32 (__main__.TestLinalgCPU)
ERROR: test_cond_cpu_float64 (__main__.TestLinalgCPU)

The failure looks like this:

======================================================================
ERROR: test_cond_cpu_complex128 (__main__.TestLinalgCPU)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/casparl/eb_tmp/eb-zicdqucj/tmp5xk49djn/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 368, in instantiated_test
    result = test(self, **param_kwargs)
  File "/tmp/casparl/eb_tmp/eb-zicdqucj/tmp5xk49djn/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 769, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/tmp/casparl/eb_tmp/eb-zicdqucj/tmp5xk49djn/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 769, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/tmp/casparl/eb_tmp/eb-zicdqucj/tmp5xk49djn/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 769, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/tmp/casparl/eb_build/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/test_linalg.py", line 1635, in test_cond
    run_test_case(a, p)
  File "/tmp/casparl/eb_build/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/test_linalg.py", line 1600, in run_test_case
    result_numpy = np.linalg.cond(input.cpu().numpy(), p)
  File "<__array_function__ internals>", line 5, in cond
  File "/tmp/sw_stack_gpu/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 1780, in cond
    r = norm(x, p, axis=(-2, -1)) * norm(invx, p, axis=(-2, -1))
  File "<__array_function__ internals>", line 5, in norm
  File "/tmp/sw_stack_gpu/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 2600, in norm
    ret = _multi_svd_norm(x, row_axis, col_axis, sum)
  File "/tmp/sw_stack_gpu/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 2354, in _multi_svd_norm
    result = op(svd(y, compute_uv=False), axis=-1)
  File "<__array_function__ internals>", line 5, in svd
  File "/tmp/sw_stack_gpu/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 1672, in svd
    s = gufunc(a, signature=signature, extobj=extobj)
  File "/tmp/sw_stack_gpu/software/SciPy-bundle/2021.05-foss-2021a/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 97, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

The reason is that OpenBLAS changed behavior as to how NaN's are propagated, see numpy/numpy#18914

In the particular case of these PyTorch tests, the input is

input=np.array([[1., 0., 0.], [0., 1., 0.], [0., 0., 0.]])

And the call being made when the test fails is:

np.linalg.cond(input,'nuc')

Note that it is only the nuc norm that fails, all the others seem to return with inf.

To Reproduce

Steps to reproduce the behavior:

1. Build PyTorch 1.10.0 from source with a numpy that is build against OpenBLAS 0.3.15
2. Run the run_test.py -i test_linalg.TestLinalgCPU tests

Instead of running the test case, one can also run the failing commands from the test case directly:

input=np.array([[1., 0., 0.], [0., 1., 0.], [0., 0., 0.]])
np.linalg.cond(input,'nuc')

Expected behavior

I expected the test to pass :)

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

• PyTorch Version (e.g., 1.0): 1.10.0
• OS (e.g., Linux): CentOS 8
• How you installed PyTorch (conda, pip, source): Source
• Build command you used (if compiling from source):

USE_CUPTI_SO=1 PYTORCH_BUILD_VERSION=1.10.0 PYTORCH_BUILD_NUMBER=1 MAX_JOBS=72 BLAS=Eigen WITH_BLAS=open USE_FFMPEG=1 BUILD_CUSTOM_PROTOBUF=0 USE_IBVERBS=1 USE_CUDA=1 CUDNN_LIB_DIR=/tmp/sw_stack_gpu/software/cuDNN/8.2.1.32-CUDA-11.3.1/lib64 CUDNN_INCLUDE_DIR=/tmp/sw_stack_gpu/software/cuDNN/8.2.1.32-CUDA-11.3.1/include USE_SYSTEM_NCCL=1 NCCL_INCLUDE_DIR=/tmp/sw_stack_gpu/software/NCCL/2.10.3-GCCcore-10.3.0-CUDA-11.3.1/include USE_METAL=0   /tmp/sw_stack_gpu/software/Python/3.9.5-GCCcore-10.3.0/bin/python setup.py build

• Python version: 3.9.5
• CUDA/cuDNN version: CUDA: 11.3.1 / cuDNN: 8.2.1.32
• GPU models and configuration: N/A
• Any other relevant information: numpy 1.20.3, OpenBLAS 0.3.15

Additional context

The PyTorch tests use the np.linalg functions as a reference, and check if the equivalent torch.linalg calls produce the same result. In this case, the np.linalg function fails because of changed OpenBLAS behavior. This failure is LAPACK implementation-dependent.

There is still discussion in numpy/numpy#18914 as to how to deal with the changed OpenBLAS behavior. It seems likely that we should conclude that behavior of np.linalg.cond is undefined for inputs that contain a NaN or that are non-invertible / singular. This will at least be the behavior of all past versions of numpy that are compiled with OpenBLAS 0.3.15. Thus, using test cases in a test suite that rely on singular/non-inveritble inputs for np.linalg.cond as a reference is probably not a great idea.

The test that fails is the one that uses singular input on purpose

pytorch/test/test_linalg.py

Line 1637 in cd51d2a

# test for singular input

In my view, this test should simply be skipped.

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @lezcano

[mruberry]

Thanks for reporting this issue, @casparvl!