Incompatible with pytest-mypy-plugins (3.10 version)

[sobolevn]

Python Version

3.8.11

Package Version

3.10

Description

I am trying to update pytest-randomly from 3.8 to 3.10. But my test suite now fails:

 ============================= test session starts ==============================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=2912292964
rootdir: /home/runner/work/returns/returns, configfile: setup.cfg
plugins: randomly-3.10.0, subtests-0.5.0, xdist-2.3.0, hypothesis-6.14.6, returns-0.16.0, mypy-plugins-1.7.0, anyio-3.3.0, forked-1.3.0
collected 830 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 637, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 212, in pytest_collection_modifyitems
INTERNALERROR>     _shuffle_by_class(list(group), seed),
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 239, in _shuffle_by_class
INTERNALERROR>     klass_items.sort()
INTERNALERROR> TypeError: '<' not supported between instances of 'YamlTestItem' and 'YamlTestItem'

Link: https://github.com/dry-python/returns/pull/1021/checks?check_run_id=3322840792

I guess this happens because we use pytest-mypy-plugins where we define our tests in yml files, example: https://github.com/dry-python/returns/blob/master/typesafety/test_functions/test_tap.yml

Steps to reproduce:

1. Clone https://github.com/dry-python/returns
2. Run poetry install
3. Run poetry run pip install 'pytest-randomly==3.10.0'
4. Run poetry run pytest typesafety -p no:cov -o addopts="" --mypy-ini-file=setup.cfg : https://github.com/dry-python/returns/blob/master/.github/workflows/test.yml#L67

[adamchainz]

Fix released in 3.10.1.

The issue was that your test items have the same test ID's, and the original sort would progres from the equal keys to comparing the items themselves. You might want to adjust the loaded test id's from your plugin.

[sobolevn]

Thanks a lot for the quick fix!