Add pytest slow test filter

[IAlibay]

Pretty much doing what's in the pytest docs + some fun actions trickery

[codecov]

Codecov Report

Patch coverage: 93.33% and project coverage change: -1.26 ⚠️

Comparison is base (52dba26) 91.82% compared to head (8835a9d) 90.56%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #333      +/-   ##
==========================================
- Coverage   91.82%   90.56%   -1.26%     
==========================================
  Files          92       92              
  Lines        5712     5723      +11     
==========================================
- Hits         5245     5183      -62     
- Misses        467      540      +73     

Impacted Files	Coverage Δ
openfe/protocols/openmm_rfe/equil_rfe_methods.py	89.81% <ø> (-0.19%)	⬇️
openfe/tests/conftest.py	98.85% <92.30%> (-1.15%)	⬇️
...tests/protocols/test_openmm_equil_rfe_protocols.py	86.39% <100.00%> (-13.61%)	⬇️

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[mikemhenry]

I love this

[mikemhenry]

If this works, I am going to steal it since this pattern comes up all the time

[IAlibay]

I'll admit it's not fully original - may have been inspired by github forums answers 😅

That being said, if you like this, you'll love this: https://github.com/MDAnalysis/MDAKits/blob/main/.github/workflows/gh-ci-cron.yaml#L44

[IAlibay]

so adding this as a slow test does make CI go down to ~ couple of minutes (at least on Linux), however it does mean not having recorded coverage for the "protein" bits of the Protocol. It's ~ 1%, so I don't know if we care that much.

[richardjgowers]

pretty sure we can mark individual parameters as slow here, so you could keep repex as always on, then test the other two when slow tests are enabled.

[IAlibay]

tbh the other two cases (sams and independent) don't cover anything that's not covered in the vacuum and solvent tests, so I'd advocate for removing them and keeping repex.

My main question is whether we want to take the 4 mins bullet for this one repex test (which will give us coverage for protein paths).

[richardjgowers]

depends on your viewpoint of coverage. internally we know these are doing the same thing which is your point, but externally it looks like two different codepaths so we probably ought to test both. I'd probably wrap all three in a slow call?

[IAlibay]

This is ugly but the only way we can appropriately call pytest at the root level directory and have additional options picked up (otherwise we'd have to call pytest openfe.tests and pyest openfecli.tests separately).

Interestingly, this is the recommended approach by ptyest: https://docs.pytest.org/en/latest/how-to/writing_plugins.html#plugin-discovery-order-at-tool-startup :/ ... doesn't make it any prettier though

[IAlibay]

@mikemhenry does bring up the good point that this will probably not get packaged if it's in root, which would break the idea of doing this anyways. Shall we move it back to openfe.tests?

[mikemhenry]

I vote we move back to openfe.tests so they get packaged. Then we can tell users to run pytest --pyargs openfecli.tests --pyargs openfe.tests -v -n auto and that will test everything. Then if we add more burn in tests that we want people to run, they can add the --runslow arg

[mikemhenry]

And then on our main CI we just need to add openfe/tests to the pytest invocation (if we are not using the pyargs method)

[IAlibay]

pytest --pyargs invocation won't work with the flag it seems (this is a known issue), my suggestion is that we either a) turn the flag into --fast so that by default we already run all the tests, or b) switch to using env variables (boring, but at least it'll likely work?)

[richardjgowers]

can stuff like this get a comment? it's not clear what's going on here to me

[IAlibay]

done

[IAlibay]

Ok - it's now got both the option of using a flag --runslow, for the lazy folks like me that don't like always having to set an env var on/off, and an env var (OFE_SLOW_TESTS) for when using pytest --pyargs invocations, or calling tests purely in the root directory (i.e. pytest . at the top level directory).