Possible incorrect line/path coverage? #254

Open
neich opened this Issue Feb 9, 2016 · 11 comments

Projects

None yet

4 participants

@neich
neich commented Feb 9, 2016

Hi,

I'm using jmockit line/path coverage to grade my students projects on unit testing, and I'm getting what looks like inconsistent results. Specifically for exit points inside if inside loops. I have stripped down a project to a minimum to show the error.

The repo can be found here:

https://github.com/neich/jmockit_testcase

The example is very simple. A class Portfolio contains an array of Stock. There is a method hasStock() in Portfolio to find Stock elements. I want to test that a Stock is found when the array contains 3 elements. There are 3 cases: the item is found in first, second or third place.

If the item is found in first place, the results are as expected:

screenshot from 2016-02-09 20 44 53

If the item is found in second place, things start to get weird. First, the if is reported to be executed 3 times, when it should be 2. But the problem is that even when the return true is reported to be executed 1 time, path A is not !

screenshot from 2016-02-09 20 44 07

And if the item is found in third place, line coverage reports that the if is executed 5 times when it should be 3, and still path A is not covered when return true is executed 1 time:

screenshot from 2016-02-09 20 47 40

These results look like inconsistent to me. Am I missing something?

Thanks in advance

@rliesenfeld rliesenfeld added the bug label Feb 10, 2016
@rliesenfeld rliesenfeld self-assigned this Feb 10, 2016
@rliesenfeld
Member

The execution count for line 33 is wrong in the second and third tests.

Thanks for reporting the bug!

@neich
neich commented Feb 10, 2016

And also, path A should show as covered, right?

@rliesenfeld
Member

Right.

@neich
neich commented Apr 27, 2016

Any news on this ? If you could give me some insight about where could be the problem, maybe I could try to find the bug and fix it.

@printingin3d

I have a similar problem with path coverage. Any update on the fix?

@rliesenfeld
Member

No updates yet.

@Siarhei-Yarkavy

Faced the same issue.
Just +1 if possible to address it. Thanks

@neich
neich commented Sep 28, 2016

I've dug into the code and now I understand the problem. I wouldn't call it a bug but it is a case that is not dealt with. It is related to not dealing with GOTO instructions that go backwards. It generates an incorrect graph, and I cannot find and easy solution ... I've tried a couple, and it solves the above problem, but then some other coverage tests fail.

My question for @rliesenfeld and others, would it be ok if I transform the current path coverage into prime path coverage as described in the book Introduction to Software Testing ?

@rliesenfeld
Member

"would it be ok if I transform the current path coverage into prime path coverage"?
I don't really understand what would be the implications of that, but no, probably not.

I haven't made an attempt at solving this issue yet. So, for now I can't make any decision regarding the best way to proceed.

@neich
neich commented Sep 28, 2016

Ok. Since I need a reliable path coverage tool for my course (it has already started), I'm going to start the work in my fork and use it with my students. I'll let you know when it is finished and then you can decide.

Maybe it could be included as another coverage method instead of replacing the actual one.

@neich
neich commented Nov 9, 2016

I have a first implementation of prime path coverage at:

https://github.com/neich/jmockit1

It is still very preliminary, probably buggy, but usable. I have built a github maven repository so it can be easily used just configuring the pom.xml file. Instructions in the README. Comments are welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment