Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Junit XML formatter #100

Closed
wants to merge 1 commit into from

8 participants

@jeremydstanton

This is a primitive/initial implementation of a JUnit XML format result output. It works reasonably well as is. It still needs a couple more refactors to get the variable handling in the formatter nicer and to get the error/fail messages in line with the original cucumber implementation.

Enjoy!

@jbpros
Owner

Thank you for your nice work Jeremy. Now we need some feature and specs to test this properly. And I was wondering: is there an XML schema for JUnit? It could be helpful regarding tests.

@jeremydstanton
@jbpros
Owner

It'd be nice to have something similar to the JSON formatter feature.

The XSD could be used in a Then step on every single scenario to ensure the output is valid. Yeah XML can be cool. E.g.:

  Scenario: output JSON for a feature with no scenarios
    Given a file named "features/a.feature" with:
      """
      Feature: some feature
      """
    When I run `cucumber.js -f junit`
    Then it produces a valid JUnit report # apply JUnit XSD to output
    And the resulting XML matches "//some-xpath-selector" # check for XPath matches

WDYT?

@jeremydstanton
@mattwynne
Owner
@jeremydstanton
@jbpros
Owner

@mattwynne Thank you for the hint. It makes complete sense. Writing (and particularly maintaining) formatter features is a daunting task with little added value. The JSON formatter feature is not something you want to read for fun and it is telling us something must be wrong. All the implementation details are surfacing there.

I opened a related issue.

@travi

Would love to see this get merged in. This would enable test results to be consumed by continuous integration servers much more easily.

@Jimflip

6 months on and what is the status?
Need junit output for us to consider using cucumber.js with jenkins.

Is this good enough as is?

@jeremydstanton
@jbpros
Owner

Still no tests provided. I consider closing this.

@jeremydstanton
@jbpros
Owner

This could be extracted to a separate repo now we've got registerListener(). I'm actually considering keeping only pretty and progress formatters built-in and moving all other formatters to cucumber-js-blah-formatter or something like that.

Thoughts?

@jeremydstanton
@olegomon

I think that would be a a good design desicion. Maybe you could also keep the JSON formatter as well and provide an option to write the JSON report into a file. Then any other formatter library could take that as input to generate any report format with existing template language in a post processing step.

Another benefit of this approach would be is that developers do not need to understand and implement any listener interface to implement their reports.

What do you think?

@jbpros
Owner

@olegomon good idea. Not sure the JSON formatter has to be in the main repo for that though. Do you think it'd be a pain for 3rd party formatter implementors?

@jbpros jbpros added the undecided label
@olegomon

@jbpros it depends on the documentation of course ;) If you provide some API to plug in the formatters then it should be stable. All implemented formatters will have to change if the API changes. If a JSON structure will be used as the exchange data for the 3rd party formatters it will be more stable since they will not have to rely on some API.

Also, I think there are two kinds of reporters. The one is the progress reporter for the consoles which prints the progress on the stdout along with the execution of the tests. The others like XML, HTML, JSON etc. are more suitable for CI servers where you want to review them comfortably later.

I think for the progress formatters you need some kind of event driven API like you did with the listeners in cucumberjs. But for the other reporters you will probably need a simpler approach, where you expect some end result and transform that one into your desired format. I didn't quite understand how the current JSON formatter works though.

I am highly interested in HTML reports for cucumberjs. Is there a sample code how to plug in some custom formatter and use the current listener API from an external module? Maybe I could try it out and make up my mind regarding this discussion :)

See also at this grunt plugin. They spawn the cucumberjs process and pipe the JSON data into a file. Then they use simple templates to create the HTML reports. https://www.npmjs.org/package/grunt-cucumberjs

@jbpros
Owner

It all makes sense, thank you @jeremydstanton.

You can see the HTML formatter in use in the example directory.

@laurelnaiad

@jeremydstanton thank you! It would be fantastic if this could be merged. I also think it would be really valuable to break reporters out of the repo so that reporter customization does not require cucumber-js changes -- while I totally understand the need for the code to have tests in order to be merged in cucumber-js, I'd also really appreciate the ability to select a reporter that doesn't have full test coverage and "live dangerously" in order to fit cucumber-js tests into the rest of a larger development shop's needs for standardization in this area.

@samccone
Collaborator

+1

@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((26 lines not shown))
+ self.handleStepResultEvent = function handleStepResult(event, callback) {
+ console.log('handleStepResultEvent');
+ callback();
+ };
+
+ self.handleUndefinedStepResult = function handleUndefinedStepResult(stepResult) {
+ console.log('handleUndefinedStepResult');
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((33 lines not shown))
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((38 lines not shown))
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((46 lines not shown))
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
+ testSuites = builder.create('testsuites');
+ testSuitesStartTime = self.getTime();
+ testSuitesTotalTestCount = testSuitesErrorTestCount = testSuitesFailureTestCount = 0;
+
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((68 lines not shown))
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
+ testSuite = testSuites.element('testsuite');
+ testSuiteStartTime = self.getTime();
+ testSuiteTotalTestCount = testSuiteErrorTestCount = testSuiteFailureTestCount = 0;
+ feature = event.getPayloadItem('feature');
+
+ callback();
+ };
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((58 lines not shown))
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
+ // update totals
+ testSuites.att('time', self.getTimeDiff(testSuitesStartTime, self.getTime()));
+ testSuites.att('tests', testSuitesTotalTestCount);
+ testSuites.att('errors', testSuitesErrorTestCount);
+ testSuites.att('failures', testSuitesFailureTestCount);
+
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone samccone commented on the diff
lib/cucumber/listener/junit_formatter.js
((81 lines not shown))
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
+ var feature = event.getPayloadItem('feature');
+ //console.log(feature.getName());
+ testSuite.att('name', feature.getName());
+ testSuite.att('time', self.getTimeDiff(testSuiteStartTime, self.getTime()));
+ testSuite.att('tests', testSuiteTotalTestCount);
+ testSuite.att('errors', testSuiteErrorTestCount);
+ testSuite.att('failures', testSuiteFailureTestCount);
+
+ callback();
+ };
+
+ self.logXML = function logXML(){
+ //console.log('logXML');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@samccone
Collaborator

After thinking about this more, this PR again is invalidated by #215

@samccone samccone added the invalid label
@samccone samccone removed this from the 0.5 major features milestone
@laurelnaiad

Yes. Very exciting. When do you think you think #215 will be merged?

@samccone
Collaborator

hopefully very soon :+1:

@laurelnaiad

Cool. I can use it when I have a commit sha for it in this repo (for licensing reasons), so if there's a branch our there that can be used to pre-merge the stuff that you have cooking for v0.5.0 I hope you'll share! :) Tx.

@samccone
Collaborator

Closing per #215

@samccone samccone closed this
@samccone
Collaborator

This PR will be easy to use on your side via #215

Let me know if you have any questions.

@vilmysj vilmysj referenced this pull request from a commit in vilmysj/cucumber-js
@vilmysj vilmysj add junit formatter.
junit formatter from cucumber#100
f9119d0
@afternoon afternoon referenced this pull request
Open

JUnit XML Reporting #63

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 24, 2012
This page is out of date. Refresh to see the latest.
View
4 lib/cucumber/cli/configuration.js
@@ -9,6 +9,9 @@ var Configuration = function(argv) {
var formatter;
var format = argumentParser.getFormat();
switch(format) {
+ case Configuration.JUNIT_FORMAT_NAME:
+ formatter = Cucumber.Listener.JunitFormatter();
+ break;
case Configuration.JSON_FORMAT_NAME:
formatter = Cucumber.Listener.JsonFormatter();
break;
@@ -70,6 +73,7 @@ var Configuration = function(argv) {
};
return self;
};
+Configuration.JUNIT_FORMAT_NAME = "junit";
Configuration.JSON_FORMAT_NAME = "json";
Configuration.PRETTY_FORMAT_NAME = "pretty";
Configuration.PROGRESS_FORMAT_NAME = "progress";
View
1  lib/cucumber/listener.js
@@ -36,6 +36,7 @@ Listener.EVENT_HANDLER_NAME_SUFFIX = 'Event';
Listener.Formatter = require('./listener/formatter');
Listener.PrettyFormatter = require('./listener/pretty_formatter');
Listener.ProgressFormatter = require('./listener/progress_formatter');
+Listener.JunitFormatter = require('./listener/junit_formatter');
Listener.JsonFormatter = require('./listener/json_formatter');
Listener.StatsJournal = require('./listener/stats_journal');
Listener.SummaryFormatter = require('./listener/summary_formatter');
View
155 lib/cucumber/listener/junit_formatter.js
@@ -0,0 +1,155 @@
+var JunitFormatter = function (options) {
+ var Cucumber = require('../../cucumber');
+ var builder = require('xmlbuilder');
+
+ //var failedScenarioLogBuffer = "";
+ //var undefinedStepLogBuffer = "";
+ //var failedStepResults = Cucumber.Type.Collection();
+ var testSuitesStartTime,testSuiteStartTime, testCaseStartTime;
+ var testSuites, testSuite, testCase;
+ var testSuitesTotalTestCount, testSuiteTotalTestCount;
+ var testSuitesErrorTestCount, testSuiteErrorTestCount;
+ var testSuitesFailureTestCount, testSuiteFailureTestCount;
+ var feature;
+
+ var statsJournal = Cucumber.Listener.StatsJournal();
+
+ var self = Cucumber.Listener.Formatter(options);
+
+ var parentHear = self.hear;
+ self.hear = function hear(event, callback) {
+ statsJournal.hear(event, function () {
+ parentHear(event, callback);
+ });
+ };
+/*
+ self.handleStepResultEvent = function handleStepResult(event, callback) {
+ console.log('handleStepResultEvent');
+ callback();
+ };
+
+ self.handleUndefinedStepResult = function handleUndefinedStepResult(stepResult) {
+ console.log('handleUndefinedStepResult');
+ };
+
+ self.handleFailedStepResult = function handleFailedStepResult(stepResult) {
+ console.log('handleFailedStepResult');
+ };
+*/
+
+ self.handleBeforeScenarioEvent = function handleBeforeScenarioEvent(event, callback) {
+ //console.log('handleBeforeScenarioEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testCaseStartTime = self.getTime();
+ callback();
+ };
+
+
+ self.handleAfterScenarioEvent = function handleAfterScenarioEvent(event, callback) {
+ //console.log('handleAfterScenarioEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ self.buildTestCase(event, callback);
+ };
+
+ self.handleBeforeFeaturesEvent = function handleBeforeFeaturesEvent(event, callback) {
+ //console.log('handleBeforeFeaturesEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testSuites = builder.create('testsuites');
+ testSuitesStartTime = self.getTime();
+ testSuitesTotalTestCount = testSuitesErrorTestCount = testSuitesFailureTestCount = 0;
+
+ callback();
+ };
+ self.handleAfterFeaturesEvent = function handleAfterFeaturesEvent(event, callback) {
+ //console.log('handleAfterFeaturesEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ // update totals
+ testSuites.att('time', self.getTimeDiff(testSuitesStartTime, self.getTime()));
+ testSuites.att('tests', testSuitesTotalTestCount);
+ testSuites.att('errors', testSuitesErrorTestCount);
+ testSuites.att('failures', testSuitesFailureTestCount);
+
+ // log result
+ self.logXML();
+ callback();
+ };
+ self.handleBeforeFeatureEvent = function handleBeforeFeatureEvent(event, callback) {
+ //console.log('handleBeforeFeatureEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ testSuite = testSuites.element('testsuite');
+ testSuiteStartTime = self.getTime();
+ testSuiteTotalTestCount = testSuiteErrorTestCount = testSuiteFailureTestCount = 0;
+ feature = event.getPayloadItem('feature');
+
+ callback();
+ };
+
+ self.handleAfterFeatureEvent = function handleAfterFeatureEvent(event, callback) {
+ //console.log('handleAfterFeatureEvent');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ var feature = event.getPayloadItem('feature');
+ //console.log(feature.getName());
+ testSuite.att('name', feature.getName());
+ testSuite.att('time', self.getTimeDiff(testSuiteStartTime, self.getTime()));
+ testSuite.att('tests', testSuiteTotalTestCount);
+ testSuite.att('errors', testSuiteErrorTestCount);
+ testSuite.att('failures', testSuiteFailureTestCount);
+
+ callback();
+ };
+
+ self.logXML = function logXML(){
+ //console.log('logXML');
@samccone Collaborator
samccone added a note

drop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ // log results
+ self.log(testSuites.end({ pretty: true}));
+ };
+
+ self.buildTestCase = function(event, callback){
+ scenario = event.getPayloadItem('scenario');
+
+ var testCase = testSuite.element('testcase');
+
+ // set name
+ testCase.att('name', scenario.getName());
+ // set classname/feature
+ testCase.att('classname', feature.getName());
+ // set time
+ testCase.att('time', self.getTimeDiff(testCaseStartTime, self.getTime()));
+
+ // increment the total counters
+ testSuitesTotalTestCount ++;
+ testSuiteTotalTestCount ++;
+
+ var name = scenario.getName();
+ var uri = scenario.getUri();
+ var line = scenario.getLine();
+
+ //console.log(scenario);
+ var text = uri + ":" + line + " # Scenario: " + name + "";
+ // handle errors
+ if(statsJournal.isCurrentScenarioPending() || statsJournal.isCurrentScenarioUndefined()){
+
+ // increment the error counters
+ testSuitesErrorTestCount ++;
+ testSuiteErrorTestCount ++;
+
+ // create the error element
+ testCase.element('error', {message:"Scenario: "+name+" ERROR"}, text);
+
+ // handle failures
+ } else if (statsJournal.isCurrentScenarioFailing()) {
+
+ // increment the failure counters
+ testSuitesFailureTestCount ++;
+ testSuiteFailureTestCount ++;
+
+ // create the failure element
+ testCase.element('failure', {message:"Scenario: "+name+" FAILED"}, text);
+ }
+ callback();
+
+ };
+ self.getTime = function getTime(){
+ return (new Date).getTime();
+ };
+ self.getTimeDiff = function getTimeDiff(before, after){
+ return (after - before)/1000;
+ };
+
+ return self;
+};
+module.exports = JunitFormatter;
View
3  package.json
@@ -54,7 +54,8 @@
"mkdirp": "0.3.3",
"cucumber-html": "0.2.2",
"walkdir": "0.0.4",
- "coffee-script": "1.4.0"
+ "coffee-script": "1.4.0",
+ "xmlbuilder":"0.4.2"
},
"scripts": {
"test": "./bin/cucumber.js && jasmine-node spec"
Something went wrong with that request. Please try again.