failing numpy tests on AMD Rome with SciPy-bundle 2021.05 using intel/2021a

[boegel]

I'm seeing 2 failing tests for numpy when installing SciPy-bundle-2021.05-intel-2021a.eb on our AMD Rome system (apparently we only got test reports from Intel systems in #12964).

• FAILED linalg/tests/test_linalg.py::TestNorm_NonSystematic::test_longdouble_norm:

  AssertionError: 
  Arrays are not almost equal to 6 decimals
    ACTUAL: array([2.+1.j, 1.+2.j], dtype=complex64)
    DESIRED: array([nan+nanj, nan+nanj], dtype=complex64)
  

• FAILED linalg/tests/test_linalg.py::TestSolve::test_sq_cases:

  AssertionError:
  Arrays are not almost equal to 10 decimals
    ACTUAL: 2.3333333333333
    DESIRED: 2.33333334
  

(full details below)

================================================================================================= FAILURES =================================================================================================
_________________________________________________________________________________________ TestSolve.test_sq_cases __________________________________________________________________________________________

actual = array([2.+1.j, 1.+2.j], dtype=complex64), desired = array([nan+nanj, nan+nanj], dtype=complex64), decimal = 6, err_msg = '', verbose = True

    def assert_almost_equal(actual,desired,decimal=7,err_msg='',verbose=True):
        """
        Raises an AssertionError if two items are not equal up to desired
        precision.
    
        .. note:: It is recommended to use one of `assert_allclose`,
                  `assert_array_almost_equal_nulp` or `assert_array_max_ulp`
                  instead of this function for more consistent floating point
                  comparisons.
    
        The test verifies that the elements of ``actual`` and ``desired`` satisfy.
    
            ``abs(desired-actual) < 1.5 * 10**(-decimal)``
    
        That is a looser test than originally documented, but agrees with what the
        actual implementation in `assert_array_almost_equal` did up to rounding
        vagaries. An exception is raised at conflicting values. For ndarrays this
        delegates to assert_array_almost_equal
    
        Parameters
        ----------
        actual : array_like
            The object to check.
        desired : array_like
            The expected object.
        decimal : int, optional
            Desired precision, default is 7.
        err_msg : str, optional
            The error message to be printed in case of failure.
        verbose : bool, optional
            If True, the conflicting values are appended to the error message.
    
        Raises
        ------
        AssertionError
          If actual and desired are not equal up to specified precision.
    
        See Also
        --------
        assert_allclose: Compare two array_like objects for equality with desired
                         relative and/or absolute precision.
        assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal
    
        Examples
        --------
        >>> import numpy.testing as npt
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334)
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334, decimal=10)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 10 decimals
         ACTUAL: 2.3333333333333
         DESIRED: 2.33333334
    
        >>> npt.assert_almost_equal(np.array([1.0,2.3333333333333]),
        ...                         np.array([1.0,2.33333334]), decimal=9)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 9 decimals
        <BLANKLINE>
        Mismatched elements: 1 / 2 (50%)
        Max absolute difference: 6.66669964e-09
        Max relative difference: 2.85715698e-09
         x: array([1.         , 2.333333333])
         y: array([1.        , 2.33333334])
    
        """
        __tracebackhide__ = True  # Hide traceback for py.test
        from numpy.core import ndarray
        from numpy.lib import iscomplexobj, real, imag
    
        # Handle complex numbers: separate into real/imag to handle
        # nan/inf/negative zero correctly
        # XXX: catch ValueError for subclasses of ndarray where iscomplex fail
        try:
            usecomplex = iscomplexobj(actual) or iscomplexobj(desired)
        except ValueError:
            usecomplex = False
    
        def _build_err_msg():
            header = ('Arrays are not almost equal to %d decimals' % decimal)
            return build_err_msg([actual, desired], err_msg, verbose=verbose,
                                 header=header)
    
        if usecomplex:
            if iscomplexobj(actual):
                actualr = real(actual)
                actuali = imag(actual)
            else:
                actualr = actual
                actuali = 0
            if iscomplexobj(desired):
                desiredr = real(desired)
                desiredi = imag(desired)
            else:
                desiredr = desired
                desiredi = 0
            try:
>               assert_almost_equal(actualr, desiredr, decimal=decimal)

__tracebackhide__ = True
_build_err_msg = <function assert_almost_equal.<locals>._build_err_msg at 0x14e024a1be50>
actual     = array([2.+1.j, 1.+2.j], dtype=complex64)
actuali    = array([1., 2.], dtype=float32)
actualr    = array([2., 1.], dtype=float32)
decimal    = 6
desired    = array([nan+nanj, nan+nanj], dtype=complex64)
desiredi   = array([nan, nan], dtype=float32)
desiredr   = array([nan, nan], dtype=float32)
err_msg    = ''
imag       = <function imag at 0x14e04740f1f0>
iscomplexobj = <function iscomplexobj at 0x14e04740f5e0>
ndarray    = <class 'numpy.ndarray'>
real       = <function real at 0x14e04740f040>
usecomplex = True
verbose    = True

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:574: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

actual = array([2., 1.], dtype=float32), desired = array([nan, nan], dtype=float32), decimal = 6, err_msg = '', verbose = True

    def assert_almost_equal(actual,desired,decimal=7,err_msg='',verbose=True):
        """
        Raises an AssertionError if two items are not equal up to desired
        precision.
    
        .. note:: It is recommended to use one of `assert_allclose`,
                  `assert_array_almost_equal_nulp` or `assert_array_max_ulp`
                  instead of this function for more consistent floating point
                  comparisons.
    
        The test verifies that the elements of ``actual`` and ``desired`` satisfy.
    
            ``abs(desired-actual) < 1.5 * 10**(-decimal)``
    
        That is a looser test than originally documented, but agrees with what the
        actual implementation in `assert_array_almost_equal` did up to rounding
        vagaries. An exception is raised at conflicting values. For ndarrays this
        delegates to assert_array_almost_equal
    
        Parameters
        ----------
        actual : array_like
            The object to check.
        desired : array_like
            The expected object.
        decimal : int, optional
            Desired precision, default is 7.
        err_msg : str, optional
            The error message to be printed in case of failure.
        verbose : bool, optional
            If True, the conflicting values are appended to the error message.
    
        Raises
        ------
        AssertionError
          If actual and desired are not equal up to specified precision.
    
        See Also
        --------
        assert_allclose: Compare two array_like objects for equality with desired
                         relative and/or absolute precision.
        assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal
    
        Examples
        --------
        >>> import numpy.testing as npt
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334)
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334, decimal=10)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 10 decimals
         ACTUAL: 2.3333333333333
         DESIRED: 2.33333334
    
        >>> npt.assert_almost_equal(np.array([1.0,2.3333333333333]),
        ...                         np.array([1.0,2.33333334]), decimal=9)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 9 decimals
        <BLANKLINE>
        Mismatched elements: 1 / 2 (50%)
        Max absolute difference: 6.66669964e-09
        Max relative difference: 2.85715698e-09
         x: array([1.         , 2.333333333])
         y: array([1.        , 2.33333334])
    
        """
        __tracebackhide__ = True  # Hide traceback for py.test
        from numpy.core import ndarray
        from numpy.lib import iscomplexobj, real, imag
    
        # Handle complex numbers: separate into real/imag to handle
        # nan/inf/negative zero correctly
        # XXX: catch ValueError for subclasses of ndarray where iscomplex fail
        try:
            usecomplex = iscomplexobj(actual) or iscomplexobj(desired)
        except ValueError:
            usecomplex = False
    
        def _build_err_msg():
            header = ('Arrays are not almost equal to %d decimals' % decimal)
            return build_err_msg([actual, desired], err_msg, verbose=verbose,
                                 header=header)
    
        if usecomplex:
            if iscomplexobj(actual):
                actualr = real(actual)
                actuali = imag(actual)
            else:
                actualr = actual
                actuali = 0
            if iscomplexobj(desired):
                desiredr = real(desired)
                desiredi = imag(desired)
            else:
                desiredr = desired
                desiredi = 0
            try:
                assert_almost_equal(actualr, desiredr, decimal=decimal)
                assert_almost_equal(actuali, desiredi, decimal=decimal)
            except AssertionError:
                raise AssertionError(_build_err_msg())
    
        if isinstance(actual, (ndarray, tuple, list)) \
                or isinstance(desired, (ndarray, tuple, list)):
>           return assert_array_almost_equal(actual, desired, decimal, err_msg)

__tracebackhide__ = True
_build_err_msg = <function assert_almost_equal.<locals>._build_err_msg at 0x14e024ab68b0>
actual     = array([2., 1.], dtype=float32)
decimal    = 6
desired    = array([nan, nan], dtype=float32)
err_msg    = ''
imag       = <function imag at 0x14e04740f1f0>
iscomplexobj = <function iscomplexobj at 0x14e04740f5e0>
ndarray    = <class 'numpy.ndarray'>
real       = <function real at 0x14e04740f040>
usecomplex = False
verbose    = True

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:581: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([2., 1.], dtype=float32), y = array([nan, nan], dtype=float32), decimal = 6, err_msg = '', verbose = True

    def assert_array_almost_equal(x, y, decimal=6, err_msg='', verbose=True):
        """
        Raises an AssertionError if two objects are not equal up to desired
        precision.
    
        .. note:: It is recommended to use one of `assert_allclose`,
                  `assert_array_almost_equal_nulp` or `assert_array_max_ulp`
                  instead of this function for more consistent floating point
                  comparisons.
    
        The test verifies identical shapes and that the elements of ``actual`` and
        ``desired`` satisfy.
    
            ``abs(desired-actual) < 1.5 * 10**(-decimal)``
    
        That is a looser test than originally documented, but agrees with what the
        actual implementation did up to rounding vagaries. An exception is raised
        at shape mismatch or conflicting values. In contrast to the standard usage
        in numpy, NaNs are compared like numbers, no assertion is raised if both
        objects have NaNs in the same positions.
    
        Parameters
        ----------
        x : array_like
            The actual object to check.
        y : array_like
            The desired, expected object.
        decimal : int, optional
            Desired precision, default is 6.
        err_msg : str, optional
          The error message to be printed in case of failure.
        verbose : bool, optional
            If True, the conflicting values are appended to the error message.
    
        Raises
        ------
        AssertionError
            If actual and desired are not equal up to specified precision.
    
        See Also
        --------
        assert_allclose: Compare two array_like objects for equality with desired
                         relative and/or absolute precision.
        assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal
    
        Examples
        --------
        the first assert does not raise an exception
    
        >>> np.testing.assert_array_almost_equal([1.0,2.333,np.nan],
        ...                                      [1.0,2.333,np.nan])
    
        >>> np.testing.assert_array_almost_equal([1.0,2.33333,np.nan],
        ...                                      [1.0,2.33339,np.nan], decimal=5)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 5 decimals
        <BLANKLINE>
        Mismatched elements: 1 / 3 (33.3%)
        Max absolute difference: 6.e-05
        Max relative difference: 2.57136612e-05
         x: array([1.     , 2.33333,     nan])
         y: array([1.     , 2.33339,     nan])
    
        >>> np.testing.assert_array_almost_equal([1.0,2.33333,np.nan],
        ...                                      [1.0,2.33333, 5], decimal=5)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 5 decimals
        <BLANKLINE>
        x and y nan location mismatch:
         x: array([1.     , 2.33333,     nan])
         y: array([1.     , 2.33333, 5.     ])
    
        """
        __tracebackhide__ = True  # Hide traceback for py.test
        from numpy.core import number, float_, result_type, array
        from numpy.core.numerictypes import issubdtype
        from numpy.core.fromnumeric import any as npany
    
        def compare(x, y):
            try:
                if npany(gisinf(x)) or npany( gisinf(y)):
                    xinfid = gisinf(x)
                    yinfid = gisinf(y)
                    if not (xinfid == yinfid).all():
                        return False
                    # if one item, x and y is +- inf
                    if x.size == y.size == 1:
                        return x == y
                    x = x[~xinfid]
                    y = y[~yinfid]
            except (TypeError, NotImplementedError):
                pass
    
            # make sure y is an inexact type to avoid abs(MIN_INT); will cause
            # casting of x later.
            dtype = result_type(y, 1.)
            y = array(y, dtype=dtype, copy=False, subok=True)
            z = abs(x - y)
    
            if not issubdtype(z.dtype, number):
                z = z.astype(float_)  # handle object arrays
    
            return z < 1.5 * 10.0**(-decimal)
    
>       assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
                 header=('Arrays are not almost equal to %d decimals' % decimal),
                 precision=decimal)

__tracebackhide__ = True
array      = <built-in function array>
compare    = <function assert_array_almost_equal.<locals>.compare at 0x14e024ab6dc0>
decimal    = 6
err_msg    = ''
float_     = <class 'numpy.float64'>
issubdtype = <function issubdtype at 0x14e0475b74c0>
npany      = <function any at 0x14e047573790>
number     = <class 'numpy.number'>
result_type = <function result_type at 0x14e04d5da9d0>
verbose    = True
x          = array([2., 1.], dtype=float32)
y          = array([nan, nan], dtype=float32)

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:1044: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

comparison = <function assert_array_almost_equal.<locals>.compare at 0x14e024ab6dc0>, x = array([2., 1.], dtype=float32), y = array([nan, nan], dtype=float32), err_msg = '', verbose = True
header = 'Arrays are not almost equal to 6 decimals', precision = 6, equal_nan = True, equal_inf = True

    def assert_array_compare(comparison, x, y, err_msg='', verbose=True, header='',
                             precision=6, equal_nan=True, equal_inf=True):
        __tracebackhide__ = True  # Hide traceback for py.test
        from numpy.core import array, array2string, isnan, inf, bool_, errstate, all, max, object_
    
        x = array(x, copy=False, subok=True)
        y = array(y, copy=False, subok=True)
    
        # original array for output formatting
        ox, oy = x, y
    
        def isnumber(x):
            return x.dtype.char in '?bhilqpBHILQPefdgFDG'
    
        def istime(x):
            return x.dtype.char in "Mm"
    
        def func_assert_same_pos(x, y, func=isnan, hasval='nan'):
            """Handling nan/inf.
    
            Combine results of running func on x and y, checking that they are True
            at the same locations.
    
            """
            __tracebackhide__ = True  # Hide traceback for py.test
    
            x_id = func(x)
            y_id = func(y)
            # We include work-arounds here to handle three types of slightly
            # pathological ndarray subclasses:
            # (1) all() on `masked` array scalars can return masked arrays, so we
            #     use != True
            # (2) __eq__ on some ndarray subclasses returns Python booleans
            #     instead of element-wise comparisons, so we cast to bool_() and
            #     use isinstance(..., bool) checks
            # (3) subclasses with bare-bones __array_function__ implementations may
            #     not implement np.all(), so favor using the .all() method
            # We are not committed to supporting such subclasses, but it's nice to
            # support them if possible.
            if bool_(x_id == y_id).all() != True:
                msg = build_err_msg([x, y],
                                    err_msg + '\nx and y %s location mismatch:'
                                    % (hasval), verbose=verbose, header=header,
                                    names=('x', 'y'), precision=precision)
                raise AssertionError(msg)
            # If there is a scalar, then here we know the array has the same
            # flag as it everywhere, so we should return the scalar flag.
            if isinstance(x_id, bool) or x_id.ndim == 0:
                return bool_(x_id)
            elif isinstance(y_id, bool) or y_id.ndim == 0:
                return bool_(y_id)
            else:
                return y_id
    
        try:
            cond = (x.shape == () or y.shape == ()) or x.shape == y.shape
            if not cond:
                msg = build_err_msg([x, y],
                                    err_msg
                                    + f'\n(shapes {x.shape}, {y.shape} mismatch)',
                                    verbose=verbose, header=header,
                                    names=('x', 'y'), precision=precision)
                raise AssertionError(msg)
    
            flagged = bool_(False)
            if isnumber(x) and isnumber(y):
                if equal_nan:
>                   flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')

__tracebackhide__ = True
all        = <function all at 0x14e047573940>
array      = <built-in function array>
array2string = <function array2string at 0x14e047592ee0>
bool_      = <class 'numpy.bool_'>
comparison = <function assert_array_almost_equal.<locals>.compare at 0x14e024ab6dc0>
cond       = True
equal_inf  = True
equal_nan  = True
err_msg    = ''
errstate   = <class 'numpy.errstate'>
flagged    = False
func_assert_same_pos = <function assert_array_compare.<locals>.func_assert_same_pos at 0x14e024ab6e50>
header     = 'Arrays are not almost equal to 6 decimals'
inf        = inf
isnan      = <ufunc 'isnan'>
isnumber   = <function assert_array_compare.<locals>.isnumber at 0x14e024ab6f70>
istime     = <function assert_array_compare.<locals>.istime at 0x14e024ab6ee0>
max        = <function amax at 0x14e047573e50>
object_    = <class 'numpy.object_'>
ox         = array([2., 1.], dtype=float32)
oy         = array([nan, nan], dtype=float32)
precision  = 6
verbose    = True
x          = array([2., 1.], dtype=float32)
y          = array([nan, nan], dtype=float32)

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:766: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([2., 1.], dtype=float32), y = array([nan, nan], dtype=float32), func = <ufunc 'isnan'>, hasval = 'nan'

    def func_assert_same_pos(x, y, func=isnan, hasval='nan'):
        """Handling nan/inf.
    
        Combine results of running func on x and y, checking that they are True
        at the same locations.
    
        """
        __tracebackhide__ = True  # Hide traceback for py.test
    
        x_id = func(x)
        y_id = func(y)
        # We include work-arounds here to handle three types of slightly
        # pathological ndarray subclasses:
        # (1) all() on `masked` array scalars can return masked arrays, so we
        #     use != True
        # (2) __eq__ on some ndarray subclasses returns Python booleans
        #     instead of element-wise comparisons, so we cast to bool_() and
        #     use isinstance(..., bool) checks
        # (3) subclasses with bare-bones __array_function__ implementations may
        #     not implement np.all(), so favor using the .all() method
        # We are not committed to supporting such subclasses, but it's nice to
        # support them if possible.
        if bool_(x_id == y_id).all() != True:
            msg = build_err_msg([x, y],
                                err_msg + '\nx and y %s location mismatch:'
                                % (hasval), verbose=verbose, header=header,
                                names=('x', 'y'), precision=precision)
>           raise AssertionError(msg)
E           AssertionError: 
E           Arrays are not almost equal to 6 decimals
E           
E           x and y nan location mismatch:
E            x: array([2., 1.], dtype=float32)
E            y: array([nan, nan], dtype=float32)

__tracebackhide__ = True
bool_      = <class 'numpy.bool_'>
err_msg    = ''
func       = <ufunc 'isnan'>
hasval     = 'nan'
header     = 'Arrays are not almost equal to 6 decimals'
msg        = '\nArrays are not almost equal to 6 decimals\n\nx and y nan location mismatch:\n x: array([2., 1.], dtype=float32)\n y: array([nan, nan], dtype=float32)'
precision  = 6
verbose    = True
x          = array([2., 1.], dtype=float32)
x_id       = array([False, False])
y          = array([nan, nan], dtype=float32)
y_id       = array([ True,  True])

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:743: AssertionError

During handling of the above exception, another exception occurred:

self = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>, require = {'square'}, exclude = {'generalized', 'size-0'}

    def check_cases(self, require=set(), exclude=set()):
        """
        Run func on each of the cases with all of the tags in require, and none
        of the tags in exclude
        """
        for case in self.TEST_CASES:
            # filter by require and exclude
            if case.tags & require != require:
                continue
            if case.tags & exclude:
                continue
    
            try:
>               case.check(self.do)

case       = <LinalgCase: csingle>
exclude    = {'generalized', 'size-0'}
msg        = 'In test case: <LinalgCase: csingle>\n\nTraceback (most recent call last):\n  File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/p... decimals\n ACTUAL: array([2.+1.j, 1.+2.j], dtype=complex64)\n DESIRED: array([nan+nanj, nan+nanj], dtype=complex64)\n'
require    = {'square'}
self       = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:350: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <LinalgCase: csingle>, do = <bound method SolveCases.do of <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>>

    def check(self, do):
        """
        Run the function `do` on this test case, expanding arguments
        """
>       do(self.a, self.b, tags=self.tags)

do         = <bound method SolveCases.do of <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>>
self       = <LinalgCase: csingle>

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:85: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>, a = array([[1.+2.j, 2.+3.j],
       [3.+4.j, 4.+5.j]], dtype=complex64), b = array([2.+1.j, 1.+2.j], dtype=complex64)
tags = frozenset({'square'})

    def do(self, a, b, tags):
        x = linalg.solve(a, b)
>       assert_almost_equal(b, dot_generalized(a, x))

a          = array([[1.+2.j, 2.+3.j],
       [3.+4.j, 4.+5.j]], dtype=complex64)
b          = array([2.+1.j, 1.+2.j], dtype=complex64)
self       = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>
tags       = frozenset({'square'})
x          = array([nan+nanj, nan+nanj], dtype=complex64)

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = array([2.+1.j, 1.+2.j], dtype=complex64), b = array([nan+nanj, nan+nanj], dtype=complex64), single_decimal = 6, double_decimal = 12, kw = {}, decimal = 6

    def assert_almost_equal(a, b, single_decimal=6, double_decimal=12, **kw):
        if asarray(a).dtype.type in (single, csingle):
            decimal = single_decimal
        else:
            decimal = double_decimal
>       old_assert_almost_equal(a, b, decimal=decimal, **kw)

a          = array([2.+1.j, 1.+2.j], dtype=complex64)
b          = array([nan+nanj, nan+nanj], dtype=complex64)
decimal    = 6
double_decimal = 12
kw         = {}
single_decimal = 6

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

actual = array([2.+1.j, 1.+2.j], dtype=complex64), desired = array([nan+nanj, nan+nanj], dtype=complex64), decimal = 6, err_msg = '', verbose = True

    def assert_almost_equal(actual,desired,decimal=7,err_msg='',verbose=True):
        """
        Raises an AssertionError if two items are not equal up to desired
        precision.
    
        .. note:: It is recommended to use one of `assert_allclose`,
                  `assert_array_almost_equal_nulp` or `assert_array_max_ulp`
                  instead of this function for more consistent floating point
                  comparisons.
    
        The test verifies that the elements of ``actual`` and ``desired`` satisfy.
    
            ``abs(desired-actual) < 1.5 * 10**(-decimal)``
    
        That is a looser test than originally documented, but agrees with what the
        actual implementation in `assert_array_almost_equal` did up to rounding
        vagaries. An exception is raised at conflicting values. For ndarrays this
        delegates to assert_array_almost_equal
    
        Parameters
        ----------
        actual : array_like
            The object to check.
        desired : array_like
            The expected object.
        decimal : int, optional
            Desired precision, default is 7.
        err_msg : str, optional
            The error message to be printed in case of failure.
        verbose : bool, optional
            If True, the conflicting values are appended to the error message.
    
        Raises
        ------
        AssertionError
          If actual and desired are not equal up to specified precision.
    
        See Also
        --------
        assert_allclose: Compare two array_like objects for equality with desired
                         relative and/or absolute precision.
        assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal
    
        Examples
        --------
        >>> import numpy.testing as npt
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334)
        >>> npt.assert_almost_equal(2.3333333333333, 2.33333334, decimal=10)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 10 decimals
         ACTUAL: 2.3333333333333
         DESIRED: 2.33333334
    
        >>> npt.assert_almost_equal(np.array([1.0,2.3333333333333]),
        ...                         np.array([1.0,2.33333334]), decimal=9)
        Traceback (most recent call last):
            ...
        AssertionError:
        Arrays are not almost equal to 9 decimals
        <BLANKLINE>
        Mismatched elements: 1 / 2 (50%)
        Max absolute difference: 6.66669964e-09
        Max relative difference: 2.85715698e-09
         x: array([1.         , 2.333333333])
         y: array([1.        , 2.33333334])
    
        """
        __tracebackhide__ = True  # Hide traceback for py.test
        from numpy.core import ndarray
        from numpy.lib import iscomplexobj, real, imag
    
        # Handle complex numbers: separate into real/imag to handle
        # nan/inf/negative zero correctly
        # XXX: catch ValueError for subclasses of ndarray where iscomplex fail
        try:
            usecomplex = iscomplexobj(actual) or iscomplexobj(desired)
        except ValueError:
            usecomplex = False
    
        def _build_err_msg():
            header = ('Arrays are not almost equal to %d decimals' % decimal)
            return build_err_msg([actual, desired], err_msg, verbose=verbose,
                                 header=header)
    
        if usecomplex:
            if iscomplexobj(actual):
                actualr = real(actual)
                actuali = imag(actual)
            else:
                actualr = actual
                actuali = 0
            if iscomplexobj(desired):
                desiredr = real(desired)
                desiredi = imag(desired)
            else:
                desiredr = desired
                desiredi = 0
            try:
                assert_almost_equal(actualr, desiredr, decimal=decimal)
                assert_almost_equal(actuali, desiredi, decimal=decimal)
            except AssertionError:
>               raise AssertionError(_build_err_msg())
E               AssertionError: 
E               Arrays are not almost equal to 6 decimals
E                ACTUAL: array([2.+1.j, 1.+2.j], dtype=complex64)
E                DESIRED: array([nan+nanj, nan+nanj], dtype=complex64)

__tracebackhide__ = True
_build_err_msg = <function assert_almost_equal.<locals>._build_err_msg at 0x14e024a1be50>
actual     = array([2.+1.j, 1.+2.j], dtype=complex64)
actuali    = array([1., 2.], dtype=float32)
actualr    = array([2., 1.], dtype=float32)
decimal    = 6
desired    = array([nan+nanj, nan+nanj], dtype=complex64)
desiredi   = array([nan, nan], dtype=float32)
desiredr   = array([nan, nan], dtype=float32)
err_msg    = ''
imag       = <function imag at 0x14e04740f1f0>
iscomplexobj = <function iscomplexobj at 0x14e04740f5e0>
ndarray    = <class 'numpy.ndarray'>
real       = <function real at 0x14e04740f040>
usecomplex = True
verbose    = True

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:577: AssertionError

During handling of the above exception, another exception occurred:

self = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>

    def test_sq_cases(self):
>       self.check_cases(require={'square'},
                         exclude={'generalized', 'size-0'})

self       = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:360: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>, require = {'square'}, exclude = {'generalized', 'size-0'}

    def check_cases(self, require=set(), exclude=set()):
        """
        Run func on each of the cases with all of the tags in require, and none
        of the tags in exclude
        """
        for case in self.TEST_CASES:
            # filter by require and exclude
            if case.tags & require != require:
                continue
            if case.tags & exclude:
                continue
    
            try:
                case.check(self.do)
            except Exception:
                msg = f'In test case: {case!r}\n\n'
                msg += traceback.format_exc()
>               raise AssertionError(msg)
E               AssertionError: In test case: <LinalgCase: csingle>
E               
E               Traceback (most recent call last):
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 574, in assert_almost_equal
E                   assert_almost_equal(actualr, desiredr, decimal=decimal)
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 581, in assert_almost_equal
E                   return assert_array_almost_equal(actual, desired, decimal, err_msg)
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1044, in assert_array_almost_equal
E                   assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 766, in assert_array_compare
E                   flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 743, in func_assert_same_pos
E                   raise AssertionError(msg)
E               AssertionError: 
E               Arrays are not almost equal to 6 decimals
E               
E               x and y nan location mismatch:
E                x: array([2., 1.], dtype=float32)
E                y: array([nan, nan], dtype=float32)
E               
E               During handling of the above exception, another exception occurred:
E               
E               Traceback (most recent call last):
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py", line 350, in check_cases
E                   case.check(self.do)
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py", line 85, in check
E                   do(self.a, self.b, tags=self.tags)
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py", line 460, in do
E                   assert_almost_equal(b, dot_generalized(a, x))
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py", line 41, in assert_almost_equal
E                   old_assert_almost_equal(a, b, decimal=decimal, **kw)
E                 File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 577, in assert_almost_equal
E                   raise AssertionError(_build_err_msg())
E               AssertionError: 
E               Arrays are not almost equal to 6 decimals
E                ACTUAL: array([2.+1.j, 1.+2.j], dtype=complex64)
E                DESIRED: array([nan+nanj, nan+nanj], dtype=complex64)

case       = <LinalgCase: csingle>
exclude    = {'generalized', 'size-0'}
msg        = 'In test case: <LinalgCase: csingle>\n\nTraceback (most recent call last):\n  File "/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/p... decimals\n ACTUAL: array([2.+1.j, 1.+2.j], dtype=complex64)\n DESIRED: array([nan+nanj, nan+nanj], dtype=complex64)\n'
require    = {'square'}
self       = <numpy.linalg.tests.test_linalg.TestSolve object at 0x14dfaa9e8820>

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:354: AssertionError
_______________________________________________________________________________ TestNorm_NonSystematic.test_longdouble_norm ________________________________________________________________________________

self = <numpy.linalg.tests.test_linalg.TestNorm_NonSystematic object at 0x14dfa9b0ce50>

    def test_longdouble_norm(self):
        # Non-regression test: p-norm of longdouble would previously raise
        # UnboundLocalError.
        x = np.arange(10, dtype=np.longdouble)
>       old_assert_almost_equal(norm(x, ord=3), 12.65, decimal=2)
E       AssertionError: 
E       Arrays are not almost equal to 2 decimals
E        ACTUAL: nan
E        DESIRED: 12.65

self       = <numpy.linalg.tests.test_linalg.TestNorm_NonSystematic object at 0x14dfa9b0ce50>
x          = array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float128)

/tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/tests/test_linalg.py:1513: AssertionError
============================================================================================= warnings summary =============================================================================================
../../../../../../../user/gent/400/vsc40023/eb_arcaninescratch/RHEL8/zen2-ib/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/nose/importer.py:12
  /user/gent/400/vsc40023/eb_arcaninescratch/RHEL8/zen2-ib/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    from imp import find_module, load_module, acquire_lock, release_lock

lib/tests/test_twodim_base.py::test_tril_triu_ndim2
  /tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/testing/_private/utils.py:770: RuntimeWarning: invalid value encountered in equal
    func=lambda xy: xy == +inf,

linalg/tests/test_linalg.py::TestNorm_NonSystematic::test_longdouble_norm
  /tmp/eb-n0x7is3x/tmpl6b3urdj/lib/python3.9/site-packages/numpy/linalg/linalg.py:2568: RuntimeWarning: invalid value encountered in reduce
    ret = add.reduce(absx, axis=axis, keepdims=keepdims)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================= short test summary info ==========================================================================================
FAILED linalg/tests/test_linalg.py::TestSolve::test_sq_cases - AssertionError: In test case: <LinalgCase: csingle>
FAILED linalg/tests/test_linalg.py::TestNorm_NonSystematic::test_longdouble_norm - AssertionError: 
==================================================== 2 failed, 12562 passed, 456 skipped, 1208 deselected, 22 xfailed, 3 warnings in 235.28s (0:03:55) =====================================================

I tried a couple of different compiler options, tests always fail in the same way:

• -xHost (via default optarch) + toolchainopts = {'pic': True, 'lowopt': True, 'strict': True}
  • => -O1 -xHost -fp-speculation=strict -fp-model strict -fPIC
• eb --optarch="Intel:march=core-avx2" + toolchainopts = {'pic': True, 'lowopt': True, 'strict': True}
  • => -O1 -march=core-avx2 -fp-speculation=strict -fp-model strict -fPIC
• eb --optarch="Intel:mavx2 -fma" + toolchainopts = {'pic': True, 'lowopt': True, 'strict': True}
  • => -O1 -mavx2 -fma -fp-speculation=strict -fp-model strict -fPIC
• using eb --optarch="Intel:march=core-avx2" + toolchainopts = {'pic': True}
  • => -O2 -march=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC

[boegel]

It's worth noting that this isn't a problem on the same system with SciPy-bundle/2021.10 using intel/2021b, see test report in #14255

[boegel]

@SebastianAchilles Can you check if you're seeing this too on AMD Rome?

[SebastianAchilles]

@SebastianAchilles Can you check if you're seeing this too on AMD Rome?

Yes, I can see the same 2 failing tests for numpy on AMD Rome:
https://gist.github.com/SebastianAchilles/b627f19b9c8248ec63a54b6dbe8dfcaf

[Flamefire]

I see the same issue for SciPy-bundle-2020.11-intel-2020b.eb on AMD Rome