Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Something wrong about branch coverage of class statements #62

Closed
nedbat opened this issue Apr 23, 2010 · 8 comments
Closed

Something wrong about branch coverage of class statements #62

nedbat opened this issue Apr 23, 2010 · 8 comments
Labels
bug Something isn't working

Comments

@nedbat
Copy link
Owner

nedbat commented Apr 23, 2010

Originally reported by Jean-Paul Calderone (Bitbucket: exarkun, GitHub: exarkun)


Consider the attached module. Using "coverage run --branch" on it and then "coverage html" produces a coverage report which indicates two partially covered lines.

I would expect that all three methods in the example would be treated the same (ie, either that all three would be marked partially covered, or all three would be marked completely covered).


@nedbat
Copy link
Owner Author

nedbat commented Apr 25, 2010

Branch coverage gets a bit tricky. I can explain why the coverage report looks the way it does. When analyzing code paths, coverage.py builds a list of "arcs": pairs of line numbers that are possible transitions during execution.

In your code:

class X:
	def foo(self):
		"hello"

	def bar(self):
		"world"

	def baz(self):
		"..."

these are the arcs:

[(-1, 1), (-1, 2), (-1, 5), (-1, 8), (1, -1), (1, 2), (2, -1), (2, 5), (5, -1), (5, 8), (8, -1)]

In these arcs, a -1 is used specially: as the first element in the pair, it means execution can enter at the line number given as the second element. If a -1 is the second element in the pair, it means execution can leave a context at the line number given as the first element.

So for example, (-1, 5) means that when executing the bar function, execution enters at line 5, and the (5, -1) arc means that execution also leaves at line 5.

When actually executing the code, the lines executed are 1, then 2, then 5, then 8, so the arcs actually encountered during execution are (-1, 1), (1, -1), (1, 2), (2, 5), (5, 8), (8, -1). The possible arcs not encountered were (-1, 2), (-1, 5), (-1, 8), (2, -1), (5, -1). Lines missing branches are those named in the first elements of the missing arcs, so there are missing branches on 2 and 5.

Look at the possible arcs for line 2: (2, -1) and (2, 5). The first is what would happen if you executed foo(). The second is what happens when you define foo. The problem is that for line 8, both execution and definition result in an arc of (8, -1). Execution runs from line 8 to the exit of the function, definition runs from line 8 to the exit of the class. There are really two possibilities, but they are both represented by -1, so the set of arcs doesn't include all the information it could.

When the program is run, we get an arc of (8, -1), so there are no arc possibilities for line 8 that are missing from the execution data, so 8 is not shown as missing a branch.

Now you can help me: is this just a hypothetical example, or do you really have code like this? Function defined with only docstrings are unusual.

@nedbat
Copy link
Owner Author

nedbat commented Apr 25, 2010

Original comment by Jean-Paul Calderone (Bitbucket: exarkun, GitHub: exarkun)


The real code that triggers this issue is doing interface definitions based on zope.interface. For example, http://buildbot.twistedmatrix.com/builds/twisted-coverage.py-r28849/twisted_internet_interfaces.html. I don't think this is the only case that can trigger this problem, though. Methods with only docstrings are rare outside of interface definitions, but certainly not impossible.

@nedbat
Copy link
Owner Author

nedbat commented Apr 25, 2010

Thanks for the real code example. I can see a way forward to make all the methods be marked as missing a branch, but I'm not sure that's what you want.

I would guess that you don't want coverage to draw your attention to these classes at all, since you know they are not executed, and are not going to be adding tests that will execute them.

You can keep coverage from mentioning them by excluding these classes from the coverage measurement. The best way would be to create a coverage.ini file, and use a regex to exclude any class derived from Interface:

# coverage.ini to control how coverage.py works
[report]
exclude_lines =
    # pragma: no cover
    class \w+\(Interface\):
    raise AssertionError

(I included "pragma: no cover" since I assume you still want that, and "raise AssertionError" since those statements, if they exist, are never meant to be executed, though you may have tests that do exercise those lines.)

If you use this coverage.ini file, then your interface classes will never be highlighted as missing execution.

@nedbat
Copy link
Owner Author

nedbat commented Apr 28, 2010

OK, I've changed coverage.py to record the multiple exits separately. Fixed in <<changeset cb4e94096b20 (bb)>>.

@nedbat
Copy link
Owner Author

nedbat commented Jun 25, 2013

Original comment by Tom Prince (Bitbucket: tomprince, GitHub: tomprince)


Excluding class \w+\(Interface\): will catch many but not all of the instances of this, since we have quite a few case of one interface inheriting from another (instead of from Interface directly.

Having looked at the coverage code, I don't see any obvious way to catch these instances, though.

Is there perhaps some way we could find them by inspecting the modules, and then telling coverage to ignore them?

@nedbat
Copy link
Owner Author

nedbat commented Jun 27, 2013

Tom: I marked this as fixed three years ago. Are you still seeing problems? If so, can you provide a code example that demonstrates the problem?

@nedbat
Copy link
Owner Author

nedbat commented Dec 8, 2014

Original comment by Tres Seaver (Bitbucket: tseaver, GitHub: tseaver)


I just stumbled across this myself today, while trying to assure that 'zope.interface' itself has 100% branch coverage. E.g.:

$ git clone https://github.com/zopefoundation/zope.interface.git
$ tox -e coverage # doesn't do branches
$ .tox/coverage/bin/nosetests --with-coverage --cover-branches

All the "branches" that get marked are empty "methods" of interfaces (defined using the 'class' statement, but with a different metaclass than 'type'): their suites consist purely of docstrings.

(Using 'coverage-4.0a1-py27').

@nedbat
Copy link
Owner Author

nedbat commented Mar 4, 2016

Original comment by Tristan Seligmann (Bitbucket: mithrandi, GitHub: mithrandi)


I'm still seeing this issue; the interface methods are marked "partially covered" since the body is never executed (and essentially can never be executed, there's no way to call it).

@nedbat nedbat closed this as completed Mar 4, 2016
@nedbat nedbat added major bug Something isn't working labels Jun 23, 2018
agronholm pushed a commit to agronholm/coveragepy that referenced this issue Aug 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant