Coverage plugin broken with coverage.py 5.x

[jacklovell]

Coverage reporting using the Cython plugin does not work for the 5.x series of coverage.py. For example, attempting to run coverage tests on raysect produces errors:

python setup.py build_ext -j4 --inplace --line-profile
coverage run -m unittest discover

Output:

...................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 435 tests in 2.122s

OK
Can't add file tracer data for unmeasured file '/home/jlovell/FUN/cherab/raysect/raysect/core/math/_vec3.pxd'

Attempting to produce HTML output also results in an error:

coverage html

Output:

Couldn't parse '/home/jlovell/FUN/cherab/raysect/raysect/core/acceleration/accelerator.pyx' as Python source: 'invalid syntax' at line 32

Using coverage.py 4.5 on the same set of build files results in no errors reported about pyx files, and the HTML output is produced successfully.

[scoder]

Thanks for the report. Could be some change in the plugin API. Any investigation welcome.

[nocarryr]

I ran into a similar issue, but with different exceptions being raised...

Output from this build shows an exception raised when trying to add a FileTracer for a .pxd

No issues after constraining coverage.py to 4.5.x and it seems there are a LOT of changes:
https://coverage.readthedocs.io/en/coverage-5.1/whatsnew5x.html

Log output from py.test

 Traceback (most recent call last):
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/main.py", line 191, in wrap_session
     session.exitstatus = doit(config, session) or 0
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/main.py", line 247, in _main
     config.hook.pytest_runtestloop(session=session)
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
     return self._hookexec(self, self.get_hookimpls(), kwargs)
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
     return self._inner_hookexec(hook, methods, kwargs)
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pluggy/manager.py", line 87, in 
     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
     gen.send(outcome)
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pytest_cov/plugin.py", line 254, in pytest_runtestloop
     self.cov_controller.finish()
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pytest_cov/engine.py", line 197, in finish
     self.cov.stop()
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/coverage/control.py", line 649, in save
     data = self.get_data()
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/coverage/control.py", line 703, in get_data
     if self._collector and self._collector.flush_data():
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/coverage/collector.py", line 426, in flush_data
     self.covdata.add_file_tracers(self.mapped_file_dict(self.file_tracers))
   File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/coverage/sqldata.py", line 516, in add_file_tracers
     "Can't add file tracer data for unmeasured file '%s'" % (filename,)
 coverage.misc.CoverageException: Can't add file tracer data for unmeasured file '/home/travis/build/nocarryr/cython-sounddevice/cysounddevice/devices.pxd'

[webknjaz]

I'm curious if it's related to #3636

[nocarryr]

@webknjaz since neither of the projects linked to above use src/ layout, I'd think not. That issue though and your PR are something I'm sure I'll run up against in other projects so I'll have to keep an eye on it, so thank you for pointing it out

[jbrockmendel]

Getting this on pandas too. Traceback follows:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/_pytest/main.py", line 247, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pytest_cov/plugin.py", line 254, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/pytest_cov/engine.py", line 197, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/coverage/control.py", line 642, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/coverage/control.py", line 696, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/coverage/collector.py", line 426, in flush_data
INTERNALERROR>     self.covdata.add_file_tracers(self.mapped_file_dict(self.file_tracers))
INTERNALERROR>   File "~/.local/lib/python3.6/site-packages/coverage/sqldata.py", line 516, in add_file_tracers
INTERNALERROR>     "Can't add file tracer data for unmeasured file '%s'" % (filename,)
INTERNALERROR> coverage.misc.CoverageException: Can't add file tracer data for unmeasured file 'pandas/_libs/tslibs/offsets.pxd'

Same exception (on the same file) if I run only a subset of the tests. There's nothing all that special about offsets.pxd:

from numpy cimport int64_t

cpdef to_offset(object obj)
cdef bint is_offset_object(object obj)
cdef bint is_tick_object(object obj)

cdef class BaseOffset:
    cdef readonly:
        int64_t n
        bint normalize
        dict _cache

Since the traceback says this file is unmeasured, I tried adding

cdef inline foo():
    return "bar"

and that appeared to solve the issue (with a run on a small subset of tests at least). So this appears to be related to this issue reported on the mailing list (i didnt find a GH issue) where function/class definition lines are not counted as being covered.

[jbrockmendel]

Update: adding the inline function in offsets.pxd just pushed the problem into another declaration-only .pxd file

[scoder]

1. We need to add a test case for this.
2. There's probably some kind of special-casing that we need to do for .pxd files in the plugin. Or maybe always process all Cython files, even without a corresponding .c file. Or something along those lines.

[cjdsellers]

I can report this issue remains with Cython 3.0a7 and coverage 5.5.

[cjdsellers]

Just tested with Cython 3.0a9 and coverage 6.0. The issue remains.

Something I didn't notice last time I visited the issue was this workaround linked by @jbrockmendel above.

Our codebase has over 100 .pxd files with readonly members (no public), so will hold off on that for now.

[Timtam]

still encountering this issue with Cython 3.0.0a10 and coverage 6.3.1. any progress/ideas here?

[scoder]

According to my last comment, this needs a) a test and b) more investigation following that.

Unsurprisingly, someone has to do something in order to resolve this issue.

[Timtam]

obviously, I don't have enough experience with coverage and/or Cython's internals unfortunately, but if you need any kind of help with debugging, let me know and i'll get back to you asap.

[scoder]

Debugging is really what is needed here. At some point, either coverage or the plugin in Cython.Coverage take a wrong decision about the file, the function, or the coverage data. Once that point is found, there's probably a way to fix it.

[Timtam]

OK, I took some time to look into this issue from the Cython point of view by getting into Cython.Coverage and think that I got a pretty good feeling about what this plugin does/should do. I started to debug a little further and thats what I found out so far, maybe it will help someone else to track the problem down later. Let's consider this a progress report on my findings up to this point.

As far as my debugging goes, the file_tracer method of Cython.Coverage.Plugin never returns an actual file tracer object to coverage.py right now, at least not in my specific constellation. Furthermore, it only gets called with .py files from site packages like pytest and such, but never with a single file from the package that I told it to actually measure. I literally don't know where coverage gets the idea of a .pxd file from, because Cython.Coverage doesn't seem to get in contact with any of these, or I didn't find the function yet where this is supposed to happen. I'll probably see if I can further debug something from the coverage side of things, but the situation as it stands right now leaves me rather clueless. Coverage.py however thinks that .pxd files are associated to Cython.Coverage.Plugin, so that seems to be fine so far.

[scoder]

So, coverage.py doesn't know about .pyx or .pxd files. It just sees the tracing data that comes in, and the executed files that get reported. The Cython tracer should report the source files when an extension module is executed, and in order to do that, it reads the .c file that Cython has generated, and parses the original Cython source files and line numbers from it (they are stored there in C comments). These original source files are then reported to coverage.py. That should include any executable code generated from .pxd files.

At least, that's how it used to work when this code was written, for coverage.py < 5.

[Timtam]

Yeah, that's what I thought was supposed to happen. As far as my debugging goes however, the Cython Tracer never gets to work. All it gets to do is analyze alot of py files (mainly site packages) and skipping them, for obvious reasons. The issue might also be related to my context, but I can't find the reason why coverage doesn't ask Cython to actually trace my project files as it's supposed to do. I however had to delay my investigation due to other projects taking priority right now. Will hopefully come back to it ASAP. I also opened an issue over at the coverage.py issue tracker, maybe they can tell me specifically what they've changed back in coverage 5.0 when the plugin first stopped working.

[matusvalo]

I was able to find root cause of this issue. I was able to fix this issue in Coverage code but I am not 100% sure if it is the right place - see nedbat/coveragepy#972 (comment) and PR nedbat/coveragepy#1347

[nedbat]

I've merged nedbat/coveragepy#1347, fyi.

[nedbat]

Because of nedbat/coveragepy#1538, I might have to rever @matusvalo's fix. If anyone could look for another solution, I would really appreciate it.

[nedbat]

If someone could just provide a very very explicit set of instructions for reproducing the problem, that would also be a big help. Don't assume any knowledge of Cython (I have none). What steps produce the issue?

[da-woods]

@nedbat

Create the following files:

cy_mod.pxd

cdef class C:
    cdef readonly:
        bint normalize

cy_mod.pyx

# cython: linetrace=True

cdef class C:
    def __init__(self):
        self.normalize = True

main.py

import cy_mod
print(cy_mod.C().normalize)

.coveragerc

[run]
plugins = Cython.Coverage

Now create a virtualenv like below:

python3 -m venv env
source env/bin/activate
pip install cython==3.0.0a11 coverage==6.4.0

I've used coverage 6.4.0 as the last release before @matusvalo's fix. However, you may want to use the current working version.

Compile the Cython module:

CFLAGS="-DCYTHON_TRACE=1" cythonize -if cy_mod.pyx

Run coverage on main:

python3 -m coverage run main.py

I get Can't add file tracer data for unmeasured file '<full path>/cy_mod.pxd'

This is all on Linux (may affect what commands you have to run slightly). It also assumes that gcc is installed.

[nedbat]

Thanks, this was very helpful! I confirm that 6.4.0 shows the error, 6.4.3 does not, 7.0.5 does not, and my fix in the works for a performance problem from the 6.4.3 fix also does not show the error.

[nedbat]

Can this issue be closed now? coverage 7.1.0 works without this error.

[jakirkham]

Can this issue be closed now? coverage 7.1.0 works without this error.

^ @scoder ?

[da-woods]

I think this can be closed. Thanks @nedbat