Reports are created but no summary (NullPointerException in Utils.groovy)

[mfuerstenau]

I can run tests and get reposrts for each test but not a summary.

my .properties-file is

com.athaydes.spockframework.report.IReportCreator=com.athaydes.spockframework.report.template.TemplateReportCreator

# Set properties of the report creator
com.athaydes.spockframework.report.template.TemplateReportCreator.specTemplateFile=/templates/spec-template.jira
com.athaydes.spockframework.report.template.TemplateReportCreator.reportFileExtension=jira
com.athaydes.spockframework.report.template.TemplateReportCreator.summaryTemplateFile=/templates/summary-template.jira
com.athaydes.spockframework.report.template.TemplateReportCreator.summaryFileName=summary.jira

# Output directory (where the spock reports will be created) - relative to working directory
com.athaydes.spockframework.report.outputDir=build/spock-reports

# If set to true, hides blocks which do not have any description
com.athaydes.spockframework.report.hideEmptyBlocks=false

the paths are correct and no error is thrown or logged. I run test with or without single tests. Is a summary supposed to be always created. I do have a file aggregated_report.json created inside my spock-reports directory though. I use version 1.2.4.

[rdmueller]

try to add a try/catch block to your summary and detail template in order to output errors which might been thrown by the templates. At least, that's what helped me in the past.

[mfuerstenau]

Thanks for the suggestion. I will report any progress.

[mfuerstenau]

I may have found something, there is a NullPointerException thrown within Utils.groovy even if I use an almost empty template:

java.lang.NullPointerException: Cannot get property 'failures' on null object
    at com.athaydes.spockframework.report.util.Utils$_aggregateStats_closure2.doCall(Utils.groovy:61)
    at com.athaydes.spockframework.report.util.Utils.aggregateStats(Utils.groovy:59)
    at com.athaydes.spockframework.report.util.Utils$aggregateStats$8.call(Unknown Source)
    at com.athaydes.spockframework.report.template.TemplateReportAggregator.summary(TemplateReportAggregator.groovy:35)
    at com.athaydes.spockframework.report.template.TemplateReportAggregator.this$2$summary(TemplateReportAggregator.groovy)
    at com.athaydes.spockframework.report.template.TemplateReportAggregator.writeOut(TemplateReportAggregator.groovy:46)
    at com.athaydes.spockframework.report.template.TemplateReportAggregator$writeOut$0.call(Unknown Source)
    at com.athaydes.spockframework.report.template.TemplateReportCreator.done(TemplateReportCreator.groovy:36)
    at com.athaydes.spockframework.report.IReportCreator$done$0.call(Unknown Source)
    at com.athaydes.spockframework.report.SpockReportExtension.stop(SpockReportExtension.groovy:48)
    at org.spockframework.runtime.GlobalExtensionRegistry.stopGlobalExtensions(GlobalExtensionRegistry.java:111)
    at org.spockframework.runtime.RunContext.stop(RunContext.java:67)
    at org.spockframework.runtime.RunContext.access$000(RunContext.java:30)
    at org.spockframework.runtime.RunContext$2.run(RunContext.java:159)

Update: This seems to be a bug. The commit is from release 1.2.10 and the error won't occur with 1.2.9.

[mfuerstenau]

@renatoathaydes This NullPointerException may be a bug in the latest release.

[renatoathaydes]

My test project does not cause this issue. Can you make sure to clean the build directory (so there's no hidden files in the directory where the reports are created) and try to run your tests again?

Are you trying to run an empty Specification?

[renatoathaydes]

BTW to enable logging by spock-reports (and see more clearly errors like this), follow the instructions on README (just add these to your Gradle file):

dependencies {
    ...
    testCompile group: 'org.slf4j', name: 'slf4j-simple', version: '1.7.13'
}
test {
    // set spock-reports logging level
    systemProperty 'org.slf4j.simpleLogger.defaultLogLevel', 'debug'
}

[mfuerstenau]

I get the error only if I use my own summary template via .properties-file and when I also get it if use the .md-summary template (summary-template.md) the spock reports repository.

For stacktrace I add the prefix:

<%
try {
%>

and suffix:

<%
} catch (Exception ex) {
   org.codehaus.groovy.runtime.StackTraceUtils.sanitize(ex).printStackTrace()
}
%>

HtmlReportCreator works fine, I see the problem either in the templates but more certain in the TemplateReportCreator.

The suggestion witzh slf4j did not print the error only silently failed.

[rdmueller]

Can you please try to find out what exactly causes this exception?
My summary template just runs fine: https://github.com/rdmueller/etka15/blob/master/src/test/resources/spockreporttemplate/summary-template.ad

What is different in yours?

[renatoathaydes]

Yep, my template also runs fine, I can't reproduce this.

[renatoathaydes]

The suggestion witzh slf4j did not print the error only silently failed.

Is that because you are catching the Exception in your template, maybe?

[mfuerstenau]

Incorrect, ignore! Yes, the error is cause by utils.aggregateStats( data ), but the template is from the repo.

def stats = utils.aggregateStats( data )

must be

def stats = data

because data is stats. I just don't know if the data map is only a part of what should be handed to the template or if that was something from the past and it is intended to work on the former data.stats directly as it is now data.

The stats are already extracted when SpecData is added (from TemplateReportAggregator.groovy:17):

    void addData( SpecData specData ) {
        log.debug( "Adding data to report ${specData.info.description.className}" )
        aggregatedData[ specData.info.description.className ] = Utils.stats( specData )
    }

TLDR data seems to be already aggregated stats, therefore it'll fail.

Also: TemplateReportCreatorSpec has a problem with locales as it compares text with floating point numbers (% in the in the footer) which in English have a dot as separator and in German have a comma, therefore the test fails me.

Also: Am I correct to assume that there is a spec for testing the report but not the summary? That's probably why this one slipped through.

[mfuerstenau]

Okay, seems to be incorrect what I assumed about the data being stats. The stats seem to be simply missing in my tests., There is no data.stats, therefore utils.aggregateStats( data ) fails.
Update: I am pretty sure the problem lies within the TemplateReportAggregator.add (...) which extracts stats from SpepData, and the Util.aggregateStats (...) which assumes that stats are still a submap stats which is wrong. therefore def stats = json.stats results in null and the causes the NullPointerExcedption.

[rdmueller]

Do you have tests which are executed? Or is the data maybe empty because you currently just give the template a try and have no tests executed?

[mfuerstenau]

No, tests execute normally. I made a pull request with a fix. But maybe @renatoathaydes will want to refactor this alltogether.

[mfuerstenau]

@rdmueller The problem can be reproduced by making using TemplateReportCreator with spock-reports-1.2.10 and use the .md-Templates provided in the release. The summary will fail silently, while single reports are created successfully.

[renatoathaydes]

@mfuerstenau Thank you for the PR! I have confirmed the bug and, related, another bug regarding the logger which does not work for that particular class (and maybe others, but at least some classes are logging fine!! I am investigating it).

I will take your PR and then try to fix the loggers so they show errors like this in the future.

Thanks!

[renatoathaydes]

Please notice the commit where I fixed the default summary template to use the new aggregated data structure. Will have to update all current templates as this is a breaking change... sorry about that.

[renatoathaydes]

@rdmueller @mfuerstenau if you guys could help by testing master before I make a new release, I would really appreciate... everything seems to be working on my side.