Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: Cause of test_linsolve.py test failure #18721

Closed
alugowski opened this issue Jun 22, 2023 · 2 comments
Closed

TST: Cause of test_linsolve.py test failure #18721

alugowski opened this issue Jun 22, 2023 · 2 comments
Labels
CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure upstream bug Items related to bugs in upstream projects
Milestone

Comments

@alugowski
Copy link
Contributor

Recent CI runs fail with the following error in the "Prerelease deps and 64-bit BLAS" workflow. I believe I've tracked it down.

FAILED scipy/sparse/linalg/_dsolve/tests/test_linsolve.py::TestSplu::test_threads_parallel - AssertionError: 
Items are not equal:
 ACTUAL: 14
 DESIRED: 20

That test runs the same basic SuperLU tests concurrently in 20 threads and expects them all to pass.

Cause

Removing the exception catchall yields this:

E   Traceback (most recent call last):
E     File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
E       self.run()
E     File "/usr/lib/python3.10/threading.py", line 953, in run
E       self._target(*self._args, **self._kwargs)
E     File "/scipy/build-install/lib/python3/dist-packages/scipy/sparse/linalg/_dsolve/tests/test_linsolve.py", line 709, in worker
E       self._internal_test_splu_smoketest()
E     File "/scipy/build-install/lib/python3/dist-packages/scipy/sparse/linalg/_dsolve/tests/test_linsolve.py", line 496, in _internal_test_splu_smoketest
E       self._smoketest(splu, check, np.float64)
E     File "/scipy/build-install/lib/python3/dist-packages/scipy/sparse/linalg/_dsolve/tests/test_linsolve.py", line 460, in _smoketest
E       rng = random.RandomState(1234)
E     File "numpy/random/mtrand.pyx", line 184, in numpy.random.mtrand.RandomState.__init__
E     File "_mt19937.pyx", line 130, in numpy.random._mt19937.MT19937.__init__
E     File "/usr/lib/python3.10/contextlib.py", line 78, in inner
E       with self._recreate_cm():
E     File "/usr/local/lib/python3.10/dist-packages/numpy/core/_ufunc_config.py", line 406, in __enter__
E       raise TypeError("Cannot enter `np.errstate` twice.")
E   TypeError: Cannot enter `np.errstate` twice.

(and the same error on another line)

The image builds using numpy nightly, which is 2.0.0.dev0.

I believe the error is this one: numpy/numpy#24013

Solutions

Wait for the fix in numpy nightly.

If a green CI is desired sooner, I can confirm that protecting both random.RandomState calls with a lock resolves the issue. Happy to make a PR to this effect.

class TestSplu:
    rng_mutex = threading.Lock()

    ...

        with self.rng_mutex:
            rng = random.RandomState(1234)
@steppi
Copy link
Contributor

steppi commented Jun 23, 2023

Thanks for tracking this down @alugowski! I'm +1 for just waiting for the fix in the numpy nightly.

@tupui tupui added CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure upstream bug Items related to bugs in upstream projects labels Jun 27, 2023
@tupui tupui added this to the 1.12.0 milestone Jun 27, 2023
@alugowski alugowski changed the title MAINT: Cause of test_linsolve.py test failure TST: Cause of test_linsolve.py test failure Jun 28, 2023
@rgommers
Copy link
Member

New nightlies were uploaded ~12 hours ago. This test passed on the most recent PR, gh-18790. So I'll close this. Thanks @alugowski!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure upstream bug Items related to bugs in upstream projects
Projects
None yet
Development

No branches or pull requests

4 participants