Difficulty running tests without coverage

[jaraco]

Is there an existing issue for this?

• I have searched the existing issues (open and closed), and could not find an existing issue

What keywords did you use to search existing issues?

coverage

Please describe why your using this option

I'm running some tests locally and want to dive deep on a particular issue. I've found I can pass -k "Metadata-Version" to pytest to select out the test I want. If, however, I pass this option through tox, the tests fail even when they pass, because the downselecting of tests fails the test coverage:

 twine main @ tox -- -k 'Metadata-Version'
.pkg: _optional_hooks> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: get_requires_for_build_editable> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
.pkg: build_sdist> python '/Users/jaraco/Library/Application Support/pipx/venvs/tox/lib/python3.12/site-packages/pyproject_api/_backend.py' True setuptools.build_meta
py: install_package> python -I -m pip install --force-reinstall --no-deps /Users/jaraco/code/pypa/twine/.tox/.tmp/package/14/twine-5.1.1.dev10+g2c4e4a1.tar.gz
py: commands[0]> python -m coverage run -m pytest -k Metadata-Version
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/code/pypa/twine
configfile: pytest.ini
plugins: socket-0.7.0
collected 202 items / 201 deselected / 1 selected                                                                                                

tests/test_package.py .                                                                                                                    [100%]

================================================================ warnings summary ================================================================
twine/__init__.py:40
  /Users/jaraco/code/pypa/twine/twine/__init__.py:40: DeprecationWarning: Implicit None on return values is deprecated and will raise KeyErrors.
    __uri__ = metadata["home-page"]

twine/__init__.py:42
  /Users/jaraco/code/pypa/twine/twine/__init__.py:42: DeprecationWarning: Implicit None on return values is deprecated and will raise KeyErrors.
    __author__ = metadata["author"]

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================= 1 passed, 201 deselected, 2 warnings in 0.20s ==================================================
py: commands[1]> python -m coverage html
Wrote HTML report to htmlcov/index.html
py: commands[2]> python -m coverage report --skip-covered --show-missing --fail-under 97
Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
tests/conftest.py               34     15     12      0    63%   48-51, 56-60, 66-80, 85
tests/helpers.py                16      6      2      0    67%   41-47
tests/test_auth.py             127     95     40      0    24%   14, 18-26, 30-38, 42-44, 48-50, 54-55, 59-60, 66-89, 100, 105, 111, 117-118, 124-125, 131-142, 152-163, 172-175, 181-190, 203-212, 223-230, 237-245, 268-274
tests/test_check.py             65     43      8      0    36%   26, 29-30, 33-34, 37-38, 42-43, 58-73, 78-97, 112-131, 141-165, 177-202, 214-241, 246-272, 276-281
tests/test_cli.py               12      6      4      0    38%   22-27, 31-32
tests/test_commands.py          23     14      2      0    36%   10-19, 23-25, 29-34, 38-39, 43-51
tests/test_main.py              28     17      2      0    37%   28, 32-39, 46-66, 73-80
tests/test_package.py          175    138     36      0    25%   27-43, 47-63, 67-74, 78-85, 89-100, 104-115, 119-131, 139-149, 163-171, 176-177, 226-322, 334-337, 342-349, 355-362, 370-380, 442-447, 451-452
tests/test_register.py          55     41     24      0    23%   14, 26-41, 46-65, 70-79, 86-102, 107-136
tests/test_repository.py       109     79     36      0    26%   27, 36-41, 46-51, 56-68, 73-76, 81-84, 89, 93-98, 103-108, 113-120, 125-132, 137-150, 155-168, 173-184, 193-223, 228-263, 310-321, 326-334, 346-354
tests/test_settings.py          82     58     16      0    33%   27-28, 36-54, 59-81, 89-93, 102-107, 112-113, 119-120, 125-130, 135-137, 142-146, 152-154, 157-158, 161-166, 169-170
tests/test_upload.py           204    157     62      0    24%   34, 47, 57-59, 64-78, 86-103, 111-116, 121-148, 161-176, 181-193, 200-213, 219-232, 242-266, 277-280, 290-306, 316-337, 343-352, 364-376, 416-441, 447-458, 467-482, 566, 583, 591, 600-616, 620-649, 657-663, 675-686
tests/test_utils.py             87     52     38      1    46%   28-39, 50-58, 73-84, 94-101, 105, 121-132, 136-150, 177-181, 198, 215-219, 224-229, 235-239, 243-246, 260->exit, 269-270, 283-293, 300-304, 319-334, 343-347
tests/test_wheel.py             41     23     12      0    38%   35-36, 40, 44-45, 49-62, 67-69, 74-78, 83-88, 93-103
twine/__main__.py               22     13      2      0    38%   30-50
twine/auth.py                   73     43     26      0    46%   18-19, 24-25, 29, 34-38, 48, 57, 60-71, 74-81, 84-89, 92-100, 103, 109, 117
twine/cli.py                    29     15      2      0    45%   33-53, 73-81, 85, 91-123
twine/commands/__init__.py      21     14     11      0    22%   30-36, 40-54
twine/commands/check.py         76     56     22      0    20%   54-61, 64, 72-74, 81-108, 130-164, 176-194
twine/commands/register.py      25     16      4      0    31%   43-63, 72-87
twine/commands/upload.py        95     73     39      0    16%   51-61, 86-110, 133-148, 173-273, 282-298
twine/exceptions.py             28      4      6      0    82%   35-42, 69
twine/package.py               156    104     53      3    28%   60, 72-90, 96->95, 99-100, 106, 128-138, 145-203, 206-216, 221-225, 228-235, 239-252, 273-289, 292-293, 296-298, 301-302, 305-307, 310-311, 314-316, 320-324, 328
twine/repository.py            118     82     46      0    26%   48-66, 70-77, 81-87, 90, 94-101, 104-105, 108-109, 112-127, 130-179, 184-200, 207-232, 235-242, 250
twine/settings.py               82     54     18      0    40%   111-128, 135, 139-140, 144-145, 149, 154-156, 161-274, 285-288, 293-299, 304, 313-314, 322-327, 333-341
twine/utils.py                 138     98     49      0    22%   57-100, 115-118, 125-133, 145-160, 172-181, 185-188, 193-200, 210-237, 268-297, 317-321, 330, 337-339, 348, 353-354
twine/wheel.py                  52     25     18      0    44%   47-51, 56-57, 60-87
twine/wininst.py                38     26     16      0    26%   15-17, 21-25, 28-57
------------------------------------------------------------------------
TOTAL                         2022   1367    606      4    31%

2 files skipped due to complete coverage.
Coverage failure: total of 31 is less than fail-under=97
py: exit 2 (0.14 seconds) /Users/jaraco/code/pypa/twine> python -m coverage report --skip-covered --show-missing --fail-under 97 pid=57729
  py: FAIL code 2 (4.10=setup[2.45]+cmd[1.20,0.31,0.14] seconds)
  evaluation failed :( (4.14 seconds)

In the skeleton approach, which uses pytest-cov to enact coverage and pytest-enabler to provide default parameters for a plugin when enabled, it becomes possible to easily disable coverage testing by disabling the plugin: tox -- -p no:cov -k 'Metadata-Version'.

Without this approach, I'm unable to use tox to run the tests without also invoking coverage (and getting the spurious failure). I have to instead run the tests manually.

I guess that's not horrible, but coming from a framework that makes running through tox easier, I thought I'd share this degraded experience.

Anything else you'd like to mention?

No response