Show who tests what

[nedbat]

Originally reported by andrea crotti (Bitbucket: andrea_crotti, GitHub: Unknown)

---

I was just using the awesome HTML report to see my test coverage and I had the following thought.

Wouldn't it be nice to be able to see easily what parts of the test suites are actually testing my code?

I guess that these information is collected while doing the annotation, right?

In this way we could actually see if the tests are actually good very easily, which is specially important when working with other people code.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/170

[nedbat]

This is a very interesting idea, one that figleaf pioneered with "sections". Right now we don't collect this information. The trace function would have to be modified to walk up the stack to identify "the test", then that information would have to be stored somehow. Then the reporting would have to be changed to somehow display the information.

That's three significant problems, but only three! Do you have ideas how to do them?

[nedbat]

Original comment by andrea crotti (Bitbucket: andrea_crotti, GitHub: Unknown)

---

Well I need to dive more into the internals to suggest something that
makes sense, however I see that the stats struct is this:

struct {
    unsigned int calls;
    unsigned int lines;
    unsigned int returns;
    unsigned int exceptions;
    unsigned int others;
    unsigned int new_files;
    unsigned int missed_returns;
    unsigned int stack_reallocs;
    unsigned int errors;
} stats;

which probably doesn't help much my idea, because I think we would need
to associate every line to a list of lines that are testing it.

So for example

silly_module.py
def silly_func():
foobar()

silly_test.py:
assert silly_func()

silly_test2.py:
assert silly_func()

I should have that

silly_func:0 = [silly_test.py:0, silly_test2:0]
silly_func:1 = [silly_test.py:0, silly_test2:0]

I'm afraid that it would be a awful lot of information to store if the
project gets really big though.

For the reporting I imagine just to add a clickable button near every
line that opens up a page that collects the different tests that run
that line, with some context around.

That should probably be the easier part, even if I'm not really a good
web-developer at the moment..

[nedbat]

I don't think we need to collect all the lines that test product lines, we need to collect the tests that test product lines, which reduces the data collection a bit, but it will still be a challenge.

[nedbat]

Original comment by andrea crotti (Bitbucket: andrea_crotti, GitHub: Unknown)

---

For the tests you mean the code object of the test function?

In that case I agree, because it should keep track of original file / line where it's defined if I remember correctly.

Anyway another possible usecase which of this feature is checking if unit-tests are really unit tests.

If I see for example that module a.py is tested by test_a.py but also
from test_z.py which has almost nothing to do with a.py, then there is
something wrong, and a system to visualize it would be nice..

[nedbat]

Issue #185 was marked as a duplicate of this issue.

[nedbat]

Original comment by Kevin Qiu (Bitbucket: kevinjqiu, GitHub: kevinjqiu)

---

I have a prototype that does just as the OP described: https://github.com/kevinjqiu/nostrils

It's currently as a nosetest plugin but I'd love to see coverage.py do this.

[nedbat]

Issue #311 was marked as a duplicate of this issue.

[nedbat]

Original comment by Thomas Güttler (Bitbucket: thomas-guettler, GitHub: Unknown)

---

I guess you need this data structure to implement this:

I use django ORM since it is what I know best, but SQLAlchemy might be a better solution
for this. Storage to sqlite would be enough.

#!python

class Line(models.Model):
    file_name=models.CharField()
    line_number=models.IntegerField()

class StackFrame(models.Model):
    executed_line=models.ForeignKey(Line)
    lines_of_stack=models.ManyToManyField(Line)

This structure needs to be filled with every line that gets executed by coverage.

A HTML Report could be created by this data.

I guess this is really slow... but who cares? For me it would be enough to run
this once very week in a cron/batch job.

[nedbat]

Original comment by Alexander Walters (Bitbucket: sdamon, GitHub: sdamon)

---

using an orm in the trace function... I would think would turn a 30 second test suit into a 45 minute to an hour test suit.

[nedbat]

Original comment by andrea crotti (Bitbucket: andrea_crotti, GitHub: Unknown)

---

Yes that would be probably too slow agreed..
Anyway there is a project that might solve the problem in a different way:

https://pypi.python.org/pypi/nose-knows/0.1.1

[nedbat]

Original comment by Alexander Walters (Bitbucket: sdamon, GitHub: sdamon)

---

Be very careful using that with coverage. ...In fact, it may be impossible, depending on how nice nose-knows and coverage are about setting their trace functions

[nedbat]

Original comment by Thomas Güttler (Bitbucket: thomas-guettler, GitHub: Unknown)

---

About the ORM: Linus Torvald said sometimes: good programmers care about data structures. That's why I would implement this first.
And the db structure in this case is quite easy. The implementation could use pure sqlite without ORM.

Yes, the execution time would increase a lot. But I don't think an alternative to sqlite would be much faster.

And: This is not intended to be run every time. We can optimize later.

[nedbat]

If you are going to record which line was tested by each test, what will you do as the code shifts around due to insertion and deletion of lines?

[nedbat]

Original comment by xcombelle (Bitbucket: xcombelle, GitHub: xcombelle)

---

Instead of inspecting the trace at each call to the trace function, I thought of something which could be faster:

at the start of a test record on which test we are, and during the call of trace check which is the current test at the end of the test forget the current test

[nedbat]

Original comment by Florian Bruhin (Bitbucket: The-Compiler, GitHub: The-Compiler)

---

@xcombelle that would work with extensible test frameworks (like pytest and nose), but how are you going to do this with e.g. unittest?