I've reworked the chronology report #26

Closed
wants to merge 8 commits into
from

Conversation

Projects
None yet
2 participants
@earthling

Recently we were experiencing intermittent and seemingly random failures in our rather long running integration test suite. It turned out to be a race condition in our application caused by concurrently executing tests. I would not have cracked that nut were it not for the chronology report in the testng reports. Pointlessly colorful though it may be, it was still better than nothing - which is what I found when I went looking for the chronology report in reportng. Originally, I was just going to re-enable the report, but then I recalled the progress monitor for incredibuild (a distributed C++ compiler I once had the fortune of using). It shows compilation progress across the cluster: http://www.xoreax.com/xge_xoreax_grid_engine.htm.

I thought it would be cool to do the same for concurrent test executions. This chronology report shows each test thread as a horizontal bar with tests executed by that thread represented in proportion to their start time and duration. This makes it easy to see which tests are executing concurrently. It also makes it easy to see any tests which take much longer than others. It also shows 'idle' time when threads are not executing tests (which showed me, at least, the parallel configuration 'tests' in testng did not quite work as I'd expected). Here is a screen shot of the reportng unit test suite's chronology report: Chronology.

Any feedback or suggestions for additional changes would be very much appreciated. Thank you!

@dwdyer

This comment has been minimized.

Show comment Hide comment
@dwdyer

dwdyer Mar 8, 2012

Owner

Thanks for this, it looks impressive.

I never got around to looking into a fix/workaround for the issues that made me disable the chronology in the first place (http://blog.uncommons.org/2010/05/21/reportng-1-1-1-the-less-embarrassingly-bad-version/). Did you figure out these issues?

Unfortunately I don't remember the exact details and the commit message from when I disabled it was not one of my finest (e856f98).

Owner

dwdyer commented Mar 8, 2012

Thanks for this, it looks impressive.

I never got around to looking into a fix/workaround for the issues that made me disable the chronology in the first place (http://blog.uncommons.org/2010/05/21/reportng-1-1-1-the-less-embarrassingly-bad-version/). Did you figure out these issues?

Unfortunately I don't remember the exact details and the commit message from when I disabled it was not one of my finest (e856f98).

earthling added some commits Mar 9, 2012

format stack traces better in the failure tooltip
filter 'failed configurations' and count test results more accurately
convert scrollwheel events into horizontal scroll
@earthling

This comment has been minimized.

Show comment Hide comment
@earthling

earthling Mar 9, 2012

I did have some trouble running the original chronology report - I
think it was because the 'IInvokedMethod' interface is meant to be
'internal' to TestNG. I switched to using the 'ITestResult' interfaces
and things went much smoother - but we don't make much use of
'AfterXXX' methods either. So I set some up in my tests and nothing
unusual happened. Then I had them throw some exceptions and I think I
see what you mean about surprising TestNG API's. This new chronology
report will only show 'failed' configurations (and will not show all
the 'skipped' configurations). I've made a few improvements since my
initial pull request, but I'm terrible with git :/ - I think what I
want is on git hub now.

On Thu, Mar 8, 2012 at 2:50 PM, Dan Dyer
reply@reply.github.com
wrote:

Thanks for this, it looks impressive.

I never got around to looking into a fix/workaround for the issues that made me disable the chronology in the first place (http://blog.uncommons.org/2010/05/21/reportng-1-1-1-the-less-embarrassingly-bad-version/).  Did you figure out these issues?

Unfortunately I don't remember the exact details and the commit message from when I disabled it was not one of my finest (e856f98).


Reply to this email directly or view it on GitHub:
#26 (comment)

I did have some trouble running the original chronology report - I
think it was because the 'IInvokedMethod' interface is meant to be
'internal' to TestNG. I switched to using the 'ITestResult' interfaces
and things went much smoother - but we don't make much use of
'AfterXXX' methods either. So I set some up in my tests and nothing
unusual happened. Then I had them throw some exceptions and I think I
see what you mean about surprising TestNG API's. This new chronology
report will only show 'failed' configurations (and will not show all
the 'skipped' configurations). I've made a few improvements since my
initial pull request, but I'm terrible with git :/ - I think what I
want is on git hub now.

On Thu, Mar 8, 2012 at 2:50 PM, Dan Dyer
reply@reply.github.com
wrote:

Thanks for this, it looks impressive.

I never got around to looking into a fix/workaround for the issues that made me disable the chronology in the first place (http://blog.uncommons.org/2010/05/21/reportng-1-1-1-the-less-embarrassingly-bad-version/).  Did you figure out these issues?

Unfortunately I don't remember the exact details and the commit message from when I disabled it was not one of my finest (e856f98).


Reply to this email directly or view it on GitHub:
#26 (comment)

@dwdyer dwdyer closed this Dec 3, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment