Use skip markers in tests #4188
Conversation
.github/workflows/unit_tests.yml
Outdated
| @@ -212,7 +212,7 @@ jobs: | |||
| # run test suite | |||
| python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log | |||
| # try and make sure output of running tests is clean (no printed messages/warnings) | |||
| IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.[56]|from cryptography.*default_backend|CryptographyDeprecationWarning: Python 2|from cryptography.utils import int_from_bytes|Blowfish|GC3Pie not available, skipping test" | |||
| IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.[56]|from cryptography.*default_backend|CryptographyDeprecationWarning: Python 2|from cryptography.utils import int_from_bytes|Blowfish|GC3Pie not available, skipping test|\.s+" | |||
There was a problem hiding this comment.
Adding \s+ to IGNORE_PATTERNS defeats the whole purpose of it?
There was a problem hiding this comment.
Pythons unittest will print a literal dot for each test executed and a literal s for each test skipped.
I did add \.s+ not \s+ to the patterns, i.e. I want to ignore a dot followed by one or more "s" as otherwise those skipped tests would make CI fail. See my last paragraph in the PR description about a discussion starter on this.
There was a problem hiding this comment.
Oh, right, I totally misinterpreting this, sorry... Makes sense
@Flamefire How about only allowing So we should conditionally extend |
Flamefire
force-pushed
the
skip-markers
branch
from
d5fb240
to
fa05f50
Compare
I guess you meant to insert a NOT somewhere as no tests should be skipped on push to develop.
Makes sense, however we should be more selective here. I hence added a dedicated decorator for that which doesn't produce output for PRs (i.e. silently succeeds) and make use of |
Flamefire
force-pushed
the
skip-markers
branch
5 times, most recently
from
f76b45c
to
65320fb
Compare
Flamefire
force-pushed
the
skip-markers
branch
from
65320fb
to
c7f4ee9
Compare
Flamefire
changed the title
Flamefire
force-pushed
the
skip-markers
branch
7 times, most recently
from
05c14fe
to
2a38d1e
Compare
Flamefire
force-pushed
the
skip-markers
branch
2 times, most recently
from
4040414
to
149f7ec
Compare
The pip-installable pysvn is incomplete and the full one is not easy to install.
Simply return if `$TEST_EASYBUILD_MODULES_TOOL` is not set to Lmod. This avoids any "skip" output on CI
For PRs silently skip the tests when no token is available else use a skip. On GHA force enable the tests when the source repo is 'easybuilds' so a token is expected to be available.
As that is used a lot use that for better describing the purpose.
GC3Pie on Python 3.11 does not work: gc3pie/gc3pie#674 So avoid the skip mark for that.
From #3790
We can use the more conventional
@unittest.skipIforself.skipTestmarkers to skip tests instead of C&P of conditions which might be wrong (some were)An approach taken from IIRC PyTorch is to define own decorators
requiresFooas a wrapper around a specificskipIfwhich is reusable and easier to understand.Only issue I can see is with the change to CI to ignore all skipped tests as now that I look at it previously we were able to selectively ignore some skipped tests.
So maybe we should not allow any skipped tests on CI which usually means installing additional Python packages. And for the tests requiring a github token we then need to resort back to the old approach but maybe we can still use a decorator here