How to write plugin without lowering coverage reported by pytest-cov?

[andy-maier]

I have written a pytest fixture that is parametrized with data from a config file. In a first attempt, I simply read the config file at the global module level where the fixture is defined, and used the result to parametrize the fixture. That all worked fine, but had the disadvantage that the config file is loaded at module import time, plus the control of which file was used and what was used from the file was via environment variables.

I then switched to writing a pytest plugin that implements the pytest_generate_tests() hook which loads the config file and parametrizes the fixture. That successfully deferred the loading of the config file until pytest runs. Also, the additional benefit of a plugin is that the control can now be done nicely via command line options.

The plugin approach also works fine, but it caused the test coverage to drop dramatically. Here is the resulting dropped coverage:

---------- coverage: platform darwin, python 3.8.7-final-0 -----------
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
client_end2end_tester/__init__.py              7      7     0%
client_end2end_tester/_exceptions.py           9      9     0%
client_end2end_tester/_srvdef.py              27     16    41%
client_end2end_tester/_srvdef_fixture.py      14      8    43%
client_end2end_tester/_srvdeffile.py          76     18    76%
client_end2end_tester/_version.py              2      2     0%
client_end2end_tester/plugin.py               34     20    41%
--------------------------------------------------------------
TOTAL                                        169     80    53%

The coverage of the same code, just without registering the plugin as an entry point, and without running the sample test function that uses the fixture, is:

---------- coverage: platform darwin, python 3.8.7-final-0 -----------
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
client_end2end_tester/__init__.py              7      0   100%
client_end2end_tester/_exceptions.py           9      0   100%
client_end2end_tester/_srvdef.py              27      0   100%
client_end2end_tester/_srvdef_fixture.py      14      6    57%
client_end2end_tester/_srvdeffile.py          76      0   100%
client_end2end_tester/_version.py              2      0   100%
client_end2end_tester/plugin.py               34     34     0%
--------------------------------------------------------------
TOTAL                                        169     40    76%

Note that two less tests are run here, and the coverage is way up.

My suspicion is that the pytest-cov plugin that is used to measure/report the coverage is somehow impacted by my new plugin.

My implementation of the pytest_generate_tests() hook is here: https://github.com/andy-maier/client_end2end_tester/blob/master/client_end2end_tester/plugin.py#L72.

My question is: What in my plugin causes the reported coverage to drop so dramatically, and how can I fix that.

One more tidbit: I tried to wrapper the pytest_generate_tests() hook with @pytest.hookimpl(hookwrapper=True) but that resulted in:

../../virtualenvs/cet38/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
E   TypeError: 'NoneType' object is not an iterator

Pytest info:

• platform darwin -- Python 3.8.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
• plugins: cov-2.11.1, client-end2end-tester-0.5.0.dev1

[nicoddemus]

(I'm short on time so I only read the issue in passing)

When testing plugins, I avoid pytest-cov, because it also being a plugin, it sometimes misses coverage of the pytest internals (which is expected).

In those cases I run coverage -m pytest ... instead (pytest itself does this).

[andy-maier]

Thanks for the quick answer. I'll see whether I can go in that direction.

[andy-maier]

I tried it and it works. The command line I am now using is:

coverage run --source=client_end2end_tester --rcfile=.coveragerc -m pytest --color=yes -W default -W ignore::PendingDeprecationWarning -W ignore::ResourceWarning  tests/unittest -s

and the coverage is now:

coverage report --rcfile=.coveragerc
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
client_end2end_tester/__init__.py              7      0   100%
client_end2end_tester/_exceptions.py           9      0   100%
client_end2end_tester/_srvdef.py              27      0   100%
client_end2end_tester/_srvdef_fixture.py      10      0   100%
client_end2end_tester/_srvdeffile.py          76      0   100%
client_end2end_tester/_version.py              2      0   100%
client_end2end_tester/plugin.py               38      7    82%
--------------------------------------------------------------
TOTAL                                        169      7    96%

Again, thanks much and consider this one answered! :-)

[nicoddemus]

Glad it worked! Could you please mark my answer with "Mark as answer"? Thanks!