bug-1885442: Fail platform specific system tests when unexpectedly executed

[relud]

No description provided.

[willkg]

r+wc. This works really nicely. Thank you!

[willkg]

Thank you for fixing this!

[willkg]

Because we added this pytest.ini, it takes precedent over the configuration in the setup.cfg in the root so we need to add the pytest options here.

Suggested change

	addopts = -m 'not gcp'
	+addopts = -m 'not gcp' -rsxX --tb=native --showlocals

arg	meaning
--showlocals	show local variables in tracebacks
--tb=native	Python standard library formatting
-r	show summary info s skipped, x failed, X passed

I think I have pytest configured this way across all the services. I'm game for revisiting what args we want to use. The thing I really wanted here was the list of skipped tests in the test summary.

[relud]

ah, i didn't know setup.cfg was being picked up. should i just move the stuff i added to pytest.ini there?

[willkg]

The stuff you added only applies to the systemtests, so it needs to be in some pytest configuration file in the systemtests directory.

Relatedly, pytest can pick up configuration from a million places and I think I configured all our services inconsistently for whatever reasons which is confusing. Maybe we should standardize on putting pytest configuration only in pytest.ini files?

[relud]

The stuff you added only applies to the systemtests, so it needs to be in some pytest configuration file in the systemtests directory.

after some thought I disagree, I think this should go in setup.cfg (or whatever pytest config file applies to the whole repo in the future) for a few reasons:

• I only added pytest.ini because i thought systemtest was the test root and that setup.cfg wasn't impacting systemtest. I would actually like to preserve the existing test root because it means that both systemtest and regular tests can be run with a single invocation of pytest by overriding the default testpaths, which I previously did not know and would very much like to retain so that I can easily run systemtests from vscode with the debugger enabled.
• I only configured markers in pytest.ini, and markers always only apply to a subset of tests, even in this case it only applies to a specific file: systemtest/test_post_crash.py.
  • if the other tests ever did test something that depended on the environment at startup like in socorro, it could be appropriate to use this same marker there. I even worded the doc string to cover cases like this on purpose.
  • I could be convinced to go the other direction and put the config in code in either conftest.py or systemtest/test_post_crash.py, effectively restricting it from being picked up when systemtest isn't in testpaths.
• I don't like duplicating configuration in a subdir, feels like asking for it to get out of date with our standards.

Maybe we should standardize on putting pytest configuration only in pytest.ini files?

i think we should standardize on pyproject.toml, it seems to be the current idiomatic place.

[relud]

so it needs to be in some pytest configuration file in the systemtest directory.

also in case i misunderstood what you meant by needs above, in a technical sense it functions fine to put it in setup.cfg, that's not a problem here, I checked.

[willkg]

Ok, I think I understand what's going on.

If we're going to have all the configuration in the root of the repo and the systemtests don't need any additional systemtest-specific configuration, then moving it all to pyproject.toml seems like a good idea. 👍

[relud]

moved pytest config to pyproject.toml, and filed https://bugzilla.mozilla.org/show_bug.cgi?id=1886536 to clean up whats left in setup.cfg and maybe setup.py