[RFC] unittests: allow running with pytest/unittest directly

[lazka]

Currently the unittests are not runnable with pytest or unittest without going through the run_unittests.py wrapper.

This has that downside that the common "pytest ..." fails and integration with things like VSCode fails too.

To work around that we set everything that is needed to run the tests in __init__.py and run_unittests is only one more variant to invoke them by providing different defaults and settings.

To make sure that pytest/unittest discover and run_unittests don't diverge implement an automatic test discovery in run_unittests to avoid hardcoding the tests to run there. There shouldn't be any functional changes.

All tests can be run the following way with this:

• pytest unittests (new)
• python3 -m unittest discover unittests '*tests.py' . (new)
• ./run_unittests.py

Single tests can be run the following way with this:

• pytest -k TAPParserTests (new)
• python3 -m unittest unittests/taptests.py (new)
• ./run_unittests.py TAPParserTests

[lazka]

A similar thing could even be done for project tests, in theory, if you dynamically create them and wrap them in a test class:

def create_test_case(suffix):
    class DynamicTestCase(unittest.TestCase):
        def test_example(self):
            assert 1 == 1

    DynamicTestCase.__name__ = DynamicTestCase.__name__ + suffix
    globals()[DynamicTestCase.__name__] = DynamicTestCase

create_test_case("ProjectOne")
create_test_case("ProjectTwo")

Which would make it easy to discover tests, run single tests, or even parts of some project in case it is split into different phases.

Configuration could be done via env vars, maybe even a dotenv file of some sorts if that makes things easier.

[dcbaker]

I'm confused, I always run with just pytest [args] and never use the run_unittests.py wrapper.

edit: We have the configuration in setup.cfg to make that work by default, and I was able to configure vscode to work by selecting pytest, and then "use existing configuration in setup.cfg"

[lazka]

I can't reproduce. Calling pytest gives me 320 failed, 107 passed, 103 skipped, 2 warnings in 12.45s. All failing because MESON_UNIT_TEST_BACKEND isn't set: KeyError: 'MESON_UNIT_TEST_BACKEND'

am I missing something?

[dcbaker]

Ah, yes. looking it at I did put that environment variable in my nix shell configuration. I guess that means all we need to do is actually ensure that the MESON_UNITTEST_BACKEND is set, or to get it with .get() so that it doesn't need to be set, and then everything should work.

[lazka]

Yes, plus the other things I added to __init__.py, in theory.

I've added some CLI examples for how to run the tests in the original post.

[dcbaker]

Yeah, I don't need any of that. MESON_UNIT_TEST_BACKEND=ninja pytest -k test_bindgen_drops is sufficient. Having the environment variable set for vscode is enough to get pytest working as well in the built in unit test runner as well. Is the code you're adding to __init__.py only used when launching the unittest runner directly?

[eli-schwartz]

I don't really see a point in having an init() function here? Any particular reason for doing it this way?

My guess would be that it's imitating the use of main() in run_unittests.py but there's a comparatively specific reason why we do that there, and it isn't applicable to __init__.py.

[lazka]

It's just my instinct of not calling non-pure functions in the global scope at import time, and limiting it to one place. No other reason.

With pytest it could be done in conftest.py, but that's not a thing with unittest.

[eli-schwartz]

This is python, not haskell, so I think we are fine on the non-pure functions thing.