Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Junit XML formatter #100

Closed
wants to merge 1 commit into from

8 participants

Jeremy Stanton Julien Biezemans Matt Wynne Matt Travi James Birmingham Oleg Daphne Maddox Sam Saccone
Jeremy Stanton

This is a primitive/initial implementation of a JUnit XML format result output. It works reasonably well as is. It still needs a couple more refactors to get the variable handling in the formatter nicer and to get the error/fail messages in line with the original cucumber implementation.

Enjoy!

Julien Biezemans
Owner

Thank you for your nice work Jeremy. Now we need some feature and specs to test this properly. And I was wondering: is there an XML schema for JUnit? It could be helpful regarding tests.

Jeremy Stanton
Julien Biezemans
Owner

It'd be nice to have something similar to the JSON formatter feature.

The XSD could be used in a Then step on every single scenario to ensure the output is valid. Yeah XML can be cool. E.g.:

  Scenario: output JSON for a feature with no scenarios
    Given a file named "features/a.feature" with:
      """
      Feature: some feature
      """
    When I run `cucumber.js -f junit`
    Then it produces a valid JUnit report # apply JUnit XSD to output
    And the resulting XML matches "//some-xpath-selector" # check for XPath matches

WDYT?

Jeremy Stanton
Matt Wynne
Owner
Jeremy Stanton
Julien Biezemans
Owner

@mattwynne Thank you for the hint. It makes complete sense. Writing (and particularly maintaining) formatter features is a daunting task with little added value. The JSON formatter feature is not something you want to read for fun and it is telling us something must be wrong. All the implementation details are surfacing there.

I opened a related issue.

Matt Travi

Would love to see this get merged in. This would enable test results to be consumed by continuous integration servers much more easily.

James Birmingham

6 months on and what is the status?
Need junit output for us to consider using cucumber.js with jenkins.

Is this good enough as is?

Jeremy Stanton
Julien Biezemans
Owner

Still no tests provided. I consider closing this.

Jeremy Stanton
Julien Biezemans
Owner

This could be extracted to a separate repo now we've got registerListener(). I'm actually considering keeping only pretty and progress formatters built-in and moving all other formatters to cucumber-js-blah-formatter or something like that.

Thoughts?

Jeremy Stanton
Oleg

I think that would be a a good design desicion. Maybe you could also keep the JSON formatter as well and provide an option to write the JSON report into a file. Then any other formatter library could take that as input to generate any report format with existing template language in a post processing step.

Another benefit of this approach would be is that developers do not need to understand and implement any listener interface to implement their reports.

What do you think?

Julien Biezemans
Owner

@olegomon good idea. Not sure the JSON formatter has to be in the main repo for that though. Do you think it'd be a pain for 3rd party formatter implementors?

Julien Biezemans jbpros added the undecided label
Oleg

@jbpros it depends on the documentation of course ;) If you provide some API to plug in the formatters then it should be stable. All implemented formatters will have to change if the API changes. If a JSON structure will be used as the exchange data for the 3rd party formatters it will be more stable since they will not have to rely on some API.

Also, I think there are two kinds of reporters. The one is the progress reporter for the consoles which prints the progress on the stdout along with the execution of the tests. The others like XML, HTML, JSON etc. are more suitable for CI servers where you want to review them comfortably later.

I think for the progress formatters you need some kind of event driven API like you did with the listeners in cucumberjs. But for the other reporters you will probably need a simpler approach, where you expect some end result and transform that one into your desired format. I didn't quite understand how the current JSON formatter works though.

I am highly interested in HTML reports for cucumberjs. Is there a sample code how to plug in some custom formatter and use the current listener API from an external module? Maybe I could try it out and make up my mind regarding this discussion :)

See also at this grunt plugin. They spawn the cucumberjs process and pipe the JSON data into a file. Then they use simple templates to create the HTML reports. https://www.npmjs.org/package/grunt-cucumberjs

Julien Biezemans
Owner

It all makes sense, thank you @jeremydstanton.

You can see the HTML formatter in use in the example directory.

Daphne Maddox

@jeremydstanton thank you! It would be fantastic if this could be merged. I also think it would be really valuable to break reporters out of the repo so that reporter customization does not require cucumber-js changes -- while I totally understand the need for the code to have tests in order to be merged in cucumber-js, I'd also really appreciate the ability to select a reporter that doesn't have full test coverage and "live dangerously" in order to fit cucumber-js tests into the rest of a larger development shop's needs for standardization in this area.

Sam Saccone
Collaborator

+1

Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((26 lines not shown))
+ self.handleStepResultEvent = function handleStepResult(event, callback) {
+ console.log('handleStepResultEvent');
+ callback();
+ };
+
+ self.handleUndefinedStepResult = function handleUndefinedStepResult(stepResult) {
+ console.log('handleUndefinedStepResult');
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((33 lines not shown))
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((38 lines not shown))
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((46 lines not shown))
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
+ testSuites = builder.create('testsuites');
+ testSuitesStartTime = self.getTime();
+ testSuitesTotalTestCount = testSuitesErrorTestCount = testSuitesFailureTestCount = 0;
+
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((68 lines not shown))
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
+ testSuite = testSuites.element('testsuite');
+ testSuiteStartTime = self.getTime();
+ testSuiteTotalTestCount = testSuiteErrorTestCount = testSuiteFailureTestCount = 0;
+ feature = event.getPayloadItem('feature');
+
+ callback();
+ };
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((58 lines not shown))
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
+ // update totals
+ testSuites.att('time', self.getTimeDiff(testSuitesStartTime, self.getTime()));
+ testSuites.att('tests', testSuitesTotalTestCount);
+ testSuites.att('errors', testSuitesErrorTestCount);
+ testSuites.att('failures', testSuitesFailureTestCount);
+
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((81 lines not shown))
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
+ var feature = event.getPayloadItem('feature');
+ //console.log(feature.getName());
+ testSuite.att('name', feature.getName());
+ testSuite.att('time', self.getTimeDiff(testSuiteStartTime, self.getTime()));
+ testSuite.att('tests', testSuiteTotalTestCount);
+ testSuite.att('errors', testSuiteErrorTestCount);
+ testSuite.att('failures', testSuiteFailureTestCount);
+
+ callback();
+ };
+
+ self.logXML = function logXML(){
+ //console.log('logXML');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Saccone
Collaborator

After thinking about this more, this PR again is invalidated by #215

Sam Saccone samccone added the invalid label
Sam Saccone samccone removed this from the 0.5 major features milestone
Daphne Maddox

Yes. Very exciting. When do you think you think #215 will be merged?

Sam Saccone
Collaborator

hopefully very soon :+1:

Daphne Maddox

Cool. I can use it when I have a commit sha for it in this repo (for licensing reasons), so if there's a branch our there that can be used to pre-merge the stuff that you have cooking for v0.5.0 I hope you'll share! :) Tx.

Sam Saccone
Collaborator

Closing per #215

Sam Saccone samccone closed this
Sam Saccone
Collaborator

This PR will be easy to use on your side via #215

Let me know if you have any questions.

Vilmantas Juras vilmysj referenced this pull request from a commit in vilmysj/cucumber-js
Vilmantas Juras vilmysj add junit formatter.
junit formatter from cucumber#100
f9119d0
Ben Godfrey afternoon referenced this pull request
Open

JUnit XML Reporting #63

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 24, 2012
This page is out of date. Refresh to see the latest.
4 lib/cucumber/cli/configuration.js
View
@@ -9,6 +9,9 @@ var Configuration = function(argv) {
var formatter;
var format = argumentParser.getFormat();
switch(format) {
+ case Configuration.JUNIT_FORMAT_NAME:
+ formatter = Cucumber.Listener.JunitFormatter();
+ break;
case Configuration.JSON_FORMAT_NAME:
formatter = Cucumber.Listener.JsonFormatter();
break;
@@ -70,6 +73,7 @@ var Configuration = function(argv) {
};
return self;
};
+Configuration.JUNIT_FORMAT_NAME = "junit";
Configuration.JSON_FORMAT_NAME = "json";
Configuration.PRETTY_FORMAT_NAME = "pretty";
Configuration.PROGRESS_FORMAT_NAME = "progress";
1  lib/cucumber/listener.js
View
@@ -36,6 +36,7 @@ Listener.EVENT_HANDLER_NAME_SUFFIX = 'Event';
Listener.Formatter = require('./listener/formatter');
Listener.PrettyFormatter = require('./listener/pretty_formatter');
Listener.ProgressFormatter = require('./listener/progress_formatter');
+Listener.JunitFormatter = require('./listener/junit_formatter');
Listener.JsonFormatter = require('./listener/json_formatter');
Listener.StatsJournal = require('./listener/stats_journal');
Listener.SummaryFormatter = require('./listener/summary_formatter');
155 lib/cucumber/listener/junit_formatter.js
View
@@ -0,0 +1,155 @@
+var JunitFormatter = function (options) {
+ var Cucumber = require('../../cucumber');
+ var builder = require('xmlbuilder');
+
+ //var failedScenarioLogBuffer = "";
+ //var undefinedStepLogBuffer = "";
+ //var failedStepResults = Cucumber.Type.Collection();
+ var testSuitesStartTime,testSuiteStartTime, testCaseStartTime;
+ var testSuites, testSuite, testCase;
+ var testSuitesTotalTestCount, testSuiteTotalTestCount;
+ var testSuitesErrorTestCount, testSuiteErrorTestCount;
+ var testSuitesFailureTestCount, testSuiteFailureTestCount;
+ var feature;
+
+ var statsJournal = Cucumber.Listener.StatsJournal();
+
+ var self = Cucumber.Listener.Formatter(options);
+
+ var parentHear = self.hear;
+ self.hear = function hear(event, callback) {
+ statsJournal.hear(event, function () {
+ parentHear(event, callback);
+ });
+ };
+/*
+ self.handleStepResultEvent = function handleStepResult(event, callback) {
+ console.log('handleStepResultEvent');
+ callback();
+ };
+
+ self.handleUndefinedStepResult = function handleUndefinedStepResult(stepResult) {
+ console.log('handleUndefinedStepResult');
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testSuites = builder.create('testsuites');
+ testSuitesStartTime = self.getTime();
+ testSuitesTotalTestCount = testSuitesErrorTestCount = testSuitesFailureTestCount = 0;
+
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ // update totals
+ testSuites.att('time', self.getTimeDiff(testSuitesStartTime, self.getTime()));
+ testSuites.att('tests', testSuitesTotalTestCount);
+ testSuites.att('errors', testSuitesErrorTestCount);
+ testSuites.att('failures', testSuitesFailureTestCount);
+
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testSuite = testSuites.element('testsuite');
+ testSuiteStartTime = self.getTime();
+ testSuiteTotalTestCount = testSuiteErrorTestCount = testSuiteFailureTestCount = 0;
+ feature = event.getPayloadItem('feature');
+
+ callback();
+ };
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ var feature = event.getPayloadItem('feature');
+ //console.log(feature.getName());
+ testSuite.att('name', feature.getName());
+ testSuite.att('time', self.getTimeDiff(testSuiteStartTime, self.getTime()));
+ testSuite.att('tests', testSuiteTotalTestCount);
+ testSuite.att('errors', testSuiteErrorTestCount);
+ testSuite.att('failures', testSuiteFailureTestCount);
+
+ callback();
+ };
+
+ self.logXML = function logXML(){
+ //console.log('logXML');
Sam Saccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ // log results
+ self.log(testSuites.end({ pretty: true}));
+ };
+
+ self.buildTestCase = function(event, callback){
+ scenario = event.getPayloadItem('scenario');
+
+ var testCase = testSuite.element('testcase');
+
+ // set name
+ testCase.att('name', scenario.getName());
+ // set classname/feature
+ testCase.att('classname', feature.getName());
+ // set time
+ testCase.att('time', self.getTimeDiff(testCaseStartTime, self.getTime()));
+
+ // increment the total counters
+ testSuitesTotalTestCount ++;
+ testSuiteTotalTestCount ++;
+
+ var name = scenario.getName();
+ var uri = scenario.getUri();
+ var line = scenario.getLine();
+
+ //console.log(scenario);
+ var text = uri + ":" + line + " # Scenario: " + name + "";
+ // handle errors
+ if(statsJournal.isCurrentScenarioPending() || statsJournal.isCurrentScenarioUndefined()){
+
+ // increment the error counters
+ testSuitesErrorTestCount ++;
+ testSuiteErrorTestCount ++;
+
+ // create the error element
+ testCase.element('error', {message:"Scenario: "+name+" ERROR"}, text);
+
+ // handle failures
+ } else if (statsJournal.isCurrentScenarioFailing()) {
+
+ // increment the failure counters
+ testSuitesFailureTestCount ++;
+ testSuiteFailureTestCount ++;
+
+ // create the failure element
+ testCase.element('failure', {message:"Scenario: "+name+" FAILED"}, text);
+ }
+ callback();
+
+ };
+ self.getTime = function getTime(){
+ return (new Date).getTime();
+ };
+ self.getTimeDiff = function getTimeDiff(before, after){
+ return (after - before)/1000;
+ };
+
+ return self;
+};
+module.exports = JunitFormatter;
3  package.json
View
@@ -54,7 +54,8 @@
"mkdirp": "0.3.3",
"cucumber-html": "0.2.2",
"walkdir": "0.0.4",
- "coffee-script": "1.4.0"
+ "coffee-script": "1.4.0",
+ "xmlbuilder":"0.4.2"
},
"scripts": {
"test": "./bin/cucumber.js && jasmine-node spec"
Something went wrong with that request. Please try again.