Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests fail with StopTest (OVERRUN) when generating a random integer (strategies.randoms) #3999

Open
khardix opened this issue May 20, 2024 · 2 comments
Labels
bug something is clearly wrong here

Comments

@khardix
Copy link

khardix commented May 20, 2024

I originally thought this is the same issue as #3874, but even after upgrading to hypothesis-6.102.4, my tests sometimes fail as described in $title.

Run log from the last failure:

nox > python -m pytest --exitfirst
==================================================== test session starts ====================================================
platform linux -- Python 3.12.3, pytest-8.2.1, pluggy-1.5.0
rootdir: /home/jstanek/personal/project/shoji
configfile: pyproject.toml
plugins: hypothesis-6.102.4, anyio-4.3.0, trio-0.8.0
collected 12 items

tests/test_management.py ........                                                                                     [ 66%]
tests/test_testing_helper.py ..                                                                                       [ 83%]
tests/util/test_io.py F

========================================================= FAILURES ==========================================================
__________________________________________ test_reader_receive_line_reads_all_data __________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: Optional[TResult] = func()
  |                                 ^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/runner.py", line 241, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
  |     return outcome.get_result()
  |            ^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
  |     yield
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
  |     yield
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/logging.py", line 833, in _runtest_for
  |     yield
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/capture.py", line 878, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/runner.py", line 173, in pytest_runtest_call
  |     item.runtest()
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/python.py", line 1632, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/_pytest/python.py", line 162, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jstanek/personal/project/shoji/tests/util/test_io.py", line 71, in test_reader_receive_line_reads_all_data
  |     @example(data=b"\r\n", rng=Random(0))
  |                    ^^^
  |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/core.py", line 1638, in wrapped_test
  |     raise the_error_hypothesis_found
  | BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pytest_trio/plugin.py", line 195, in _fixture_manager
    |     yield nursery_fixture
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/pytest_trio/plugin.py", line 250, in run
    |     await self._func(**resolved_kwargs)
    |   File "/home/jstanek/personal/project/shoji/tests/util/test_io.py", line 77, in test_reader_receive_line_reads_all_data
    |     recv_lines = [line async for line in reader]
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/src/shoji/util/io.py", line 88, in __anext__
    |     return await self.receive_line()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/src/shoji/util/io.py", line 66, in receive_line
    |     if chunk := await self.stream.receive_some(max_len):
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/tests/util/test_io.py", line 36, in receive_some
    |     return next(self.source_iterator, b"")
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/src/shoji/testing/helper.py", line 45, in random_chunks
    |     end = start + rng.randint(min(min_size, remains), remains)
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<string>", line 5, in randint
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/strategies/_internal/random.py", line 132, in implementation
    |     result = self._hypothesis_do_random(name, kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/strategies/_internal/random.py", line 273, in _hypothesis_do_random
    |     result = self.__data.draw_integer(kwargs["a"], kwargs["b"])
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2093, in draw_integer
    |     value = self.provider.draw_integer(
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1461, in draw_integer
    |     return self._draw_bounded_integer(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1769, in _draw_bounded_integer
    |     probe = self._cd.draw_bits(
    |             ^^^^^^^^^^^^^^^^^^^
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2574, in draw_bits
    |     self.__check_capacity(n_bytes)
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2616, in __check_capacity
    |     self.mark_overrun()
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2641, in mark_overrun
    |     self.conclude_test(Status.OVERRUN)
    |   File "/home/jstanek/personal/project/shoji/.nox/test-3-12/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2628, in conclude_test
    |     raise StopTest(self.testcounter)
    | hypothesis.errors.StopTest: 1366
    +------------------------------------
-------------------------------------------------------- Hypothesis ---------------------------------------------------------
You can add @seed(282959970759348657142277025956285860727) to this test or run pytest with --hypothesis-seed=282959970759348657142277025956285860727 to reproduce this failure.
================================================== short test summary info ==================================================
FAILED tests/util/test_io.py::test_reader_receive_line_reads_all_data - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 1 failed, 10 passed in 2.58s ================================================
nox > Command python -m pytest --exitfirst failed with exit code 1
@tybug
Copy link
Member

tybug commented May 20, 2024

reproduces with:

@given(st.randoms())
def f(rng):
    rng.randint(0, 1 << 25_000)

for i in range(10):
    print(i)
    f()

this is potentially not ir-related, I haven't tried to bisect yet.

@tybug tybug added the bug something is clearly wrong here label May 20, 2024
@Zac-HD
Copy link
Member

Zac-HD commented May 20, 2024

elif method == "randint":
result = self.__data.draw_integer(kwargs["a"], kwargs["b"])

I think we get the StopTest here and somehow fail to turn it into an Overrun error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug something is clearly wrong here
Projects
None yet
Development

No branches or pull requests

3 participants