test: use `xfail_strict` for pytest config #2328

sobolevn · 2023-09-21T08:45:06Z

One more check that I forgot in #2327

Looks like we have several xfail tests:

» ag xfail
tests/unit/test_file_system.py
27:@pytest.mark.xfail(sys.platform == "win32", reason="permissions equivalent missing on windows")
53:@pytest.mark.xfail(sys.platform == "win32", reason="Suspected fsspec issue")
85:@pytest.mark.xfail(sys.platform == "win32", reason="permissions equivalent missing on windows")

tests/unit/test_template/test_builtin_functions.py
18:@pytest.mark.xfail(sys.platform == "win32", reason="For some reason this is flaky on windows")
78:@pytest.mark.xfail(sys.platform == "win32", reason="For some reason this is flaky on windows")
183:@pytest.mark.xfail(sys.platform == "win32", reason="For some reason this is flaky on windows")
247:@pytest.mark.xfail(sys.platform == "win32", reason="For some reason this is flaky on windows")

tests/unit/test_stores.py
140:        pytest.xfail("bug in FileStore and MemoryStore")

Without xfail_strict test can pass and pytest won't report unexpected pass as an xfail error. With this setting it will.

Docs: https://docs.pytest.org/en/7.1.x/how-to/skipping.html#strict-parameter

sobolevn · 2023-09-21T08:50:50Z

I am not safe merging this without proper Windows CI, since a lot of xfails are Windows-related. Can I / someone else trigger it manually somehow?

peterschutt · 2023-09-21T10:23:41Z

I am not safe merging this without proper Windows CI, since a lot of xfails are Windows-related. Can I / someone else trigger it manually somehow?

What about temporarily making a commit that removes if: github.event_name == 'push' from the platform compat job, and reverting it before we merge? There might be a neater way to do it, but I don't know what it is.

sobolevn · 2023-09-21T10:59:44Z

Done:

I will now revert .github/ folder changes.

Later we can add more correct way to handle flaky tests, I propose to add https://github.com/box/flaky or https://pypi.org/project/pytest-repeat/ to do the job properly.