JUnit-style reporter for Serenity/JS

[igTkachov]

Please tell
Have we hook for convert serenity report to some xUnit reports for import to Azure?

[jan-molak]

Hey @igTkachov - I'm not that familiar with Azure, but do I understand correctly that you'd like Serenity/JS to produce a test report in an xUnit format so that it can be imported into another tool? Is that right?

[igTkachov]

Hey @jan-molak
Yes, you are right.
Now I try to parse serenity reports and convert to necessary xml format.

[jan-molak]

I see; A better approach would be to implement a custom JUnit reporter (or a StageCrewMember in Serenity-speak). I've been planning on doing it at some point, but if you'd be willing to contribute one that would be absolutely amazing. Let me know if that's the case and I could guide you through it.

But to address your immediate challenge. What you'd need to do is to implement a JUnitReporter that receives DomainEvents and converts them into an appropriate report you then store in a file.

If you're using 1.x have a look at the ConsoleReporter or the SerenityBDDReporter.

If you're on version 2, have a look at the crew package in @serenity-js/core and some of the StageCrewMember implementations there.

The main difference between the v1 and v2 implementations of StageCrewMembers is that in v1 the StageCrewMember used to be responsible for writing the report to a file, in v2 the ArtifactArchiver has the responsibility of performing side effects so instead of making your JUnitReporter write a report to a file you simply broadcast a domain event with the report and the ArtifactArchiver will store it in a file for you. Here's an example.

[igTkachov]

I'm not expert in js, but I want to try it.

[andreasmarkussen]

Yup! Got the same issue :) I need it for circleci.com .
I will look into doing it as well, but not sure that it will be easy.

Alternatively, I would be happy with a cucumber compatible JSON file.

[abhinaba-ghosh]

@igTkachov Can I provide a workaround?

cucumber-junit-convert can be your friend in this case:

npm i --save-dev cucumber-junit-convert

in your protractor conf, add format options to emit cucumber results in a JSON format to a specific file, like:

cucumberOpts: {
    format: ['pretty', 'json:target/cucumber_report.json'],
    'format-options': '{"colorsEnabled": true }',
    require: [`${process.cwd()}/transform/**/*.js`],
    compiler: 'ts:ts-node/register'
  }

Then create a JS file which will convert this JSON report to JUNIT style report. Let's say, the file name is jenkins.report.js

const cucumberJunitConvert = require('cucumber-junit-convert');

const options = {
  inputJsonFile: 'target/cucumber_report.json',
  outputXmlFile: 'target/cucumber_report.xml'
};

cucumberJunitConvert.convert(options);

You can add the conversion process to the serenity report step in your package.json like:

 "report": "serenity run --cacheDir ./lib/serenity-cli && node ./jenkins.report.js",

Now, every time you produce Serenity HTML reports, it will generate a JUNIT style report as well in specified folder.

[igTkachov]

@abhinaba1080 I've created my local convertor for Azure report. But your convertor should work, because Azure can parse cucumber report in xml format.

[andreasmarkussen]

@abhinaba1080 what is in your /transform/ folder?

And my cucumber_report.json file is just empty? i.e. 0 bytes.
Update: It is 0 bytes when the tests fail, but with success, it contains the expected content.

[test:execute] When the user logs in with correct credentials
[test:execute] [17:46:40] E/launcher - Error: TypeError: Cannot read property 'children' of undefined

[abhinaba-ghosh]

Hi @andreasmarkussen ,

The transform folder is where I keep all of the JS files.
I have tested the logic with both Serenity V1 and V2. It just worked fine.
If you manage to get the cucumber_report.json printed with the correct value, then only you will be able to produce the Junit report.

Can you share your protractor.conf file?

[andreasmarkussen]

OK - got it working now.

There are two learnings I would like to highlight:

I fixed my steps code, that was not implemented as it should.
And if you end up having exceptions from the Serenity-JS framework, you most likely have race conditions, caused by promises or and in my condition, I did not return the promise from the last function.

The folder location must be created e.g. reports/cucumber must exist before the reporter can write to the file.

Here is my protractor.conf.js

// @ts-check
const
    { ConsoleReporter } = require('@serenity-js/console-reporter'),
    { ArtifactArchiver } = require('@serenity-js/core'),
    { Photographer, TakePhotosOfFailures, TakePhotosOfInteractions, TakePhotosBeforeAndAfterInteractions } = require('@serenity-js/protractor'),
    { SerenityBDDReporter } = require('@serenity-js/serenity-bdd'),
    isCI = require('is-ci');

const { lambdaTestConf } = require("./.circleci/lambdaTest.conf.js")
const { sauceLabsConf } = require("./.circleci/sauceLabs.conf.js")
const { Authenticator } = require('authenticator-browser-extension');

/** Browser Location - CI or local machine
 * Direct 
 * SauceLabs (Only Cloud)
 * LambdaTest (Only Cloud)
 */
/** We use sauceLab in the cloud and direct connection locally */
const targetEnvironment = isCI ? 'sauceLab' : ''; 
// targetEnvironment = 'lambdaTest'; 

/**
 * Local url : http://localhost:10007/
 * Some url : 
 */

const determineHeadlessMode = () => {
    if (process.env.DRIVER === 'HEADLESS')
        return true;
    return false;
}

const shouldBeHeadlessMode = determineHeadlessMode();

/**
 * @type { import("protractor").Config }
 * @see https://www.protractortest.org/#/browser-setup
*/
const protractorConfig = {
    baseUrl: 'http://test.dk',

    chromeDriver: require(`chromedriver/lib/chromedriver`).path,

    SELENIUM_PROMISE_MANAGER: false,

   directConnect: true,

    // https://github.com/angular/protractor/blob/master/docs/timeouts.md
    allScriptsTimeout: 300000,

    framework: 'custom',
    frameworkPath: require.resolve('@serenity-js/protractor/adapter'),

    specs: ['features/**/*.feature'],

    serenity: {
        runner: 'cucumber',
        crew: [
            ArtifactArchiver.storingArtifactsAt('./target/site/serenity'),
            ConsoleReporter.forDarkTerminals(),
            //Photographer.whoWill(TakePhotosOfInteractions),
            Photographer.whoWill(TakePhotosOfFailures),
            //Photographer.whoWill(TakePhotosBeforeAndAfterInteractions),
            new SerenityBDDReporter(),
        ]
    },
    plugins: [
        {
            // The module name
            package: "protractor-react-selector"
        },
    ],


    /**
     * If you're interacting with a non-Angular application,
     * uncomment the below onPrepare section,
     * which disables Angular-specific test synchronisation.
     */
    
    onPrepare: function () {
        /** @type { import("protractor").ProtractorBrowser } */
        let browser = global['browser'];
        browser.waitForAngularEnabled(false); // This can be removed if the 'protractor-testability-plugin' is enabled
    },

    cucumberOpts: {
        require: ['features/**/*.ts',],
        'require-module': ['ts-node/register'],
        format: [ 'json:reports/cucumber/cucumber_report.json'],
        tags: ['~@wip'],
        strict: false,
    },

    capabilities: {
        browserName: 'chrome',
        
        // see https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities#loggingpreferences-json-object
        loggingPrefs: {
            browser: 'ALL' // "OFF", "SEVERE", "WARNING", "INFO", "CONFIG", "FINE", "FINER", "FINEST", "ALL".
        },

        chromeOptions: {
            args: [
                '--no-sandbox',
                '--disable-infobars',
                '--log-level=3',
                '--disable-gpu',
                '--window-size=1024,800',
            ].concat(shouldBeHeadlessMode ? ['--headless'] : []),    // run in headless mode on the CI server
            /** 
             * Note: We do not use headless in the sauceLabs cloud, 
             * since then we will not be able to use the authenticator extension, 
             * since Chrome Headless does not support extensions.
            */ extensions: [
                // The authenticator only has an effect locally and on sauceLab. 
                // Not on lambdaTest where the Url authentication is needed. 
            ].concat(shouldBeHeadlessMode ? [] : [ Authenticator.for('xxx', 'yyy').asBase64() ] )
        }
    }
};


function decideConfig(browserLocation) {
    console.log(`Browser Location set to ${browserLocation}`)
    switch (browserLocation) {
        case 'sauceLab':
            return { ...protractorConfig, ...sauceLabsConf }
        case 'lambdaTest':
            return { ...protractorConfig, ...lambdaTestConf }
    }
    return protractorConfig;
}

const finalConfig = decideConfig(targetEnvironment);

exports.config = finalConfig;

console.log("Protractor config  ", finalConfig)

[aksbenz]

@jan-molak : Hello,
Based on you previous guidelines, I wrote a custom JUnitReporter, but facing 2 issues:

1. There is no XML or Generic Text file Artifact available.
2. Report file is not created, I believe because ArtifactArchiver only handles Photo or TestReport Artifact. Unable to pass a generic artifact to write to file.

I create another artifact like:

export class XMLData extends Artifact {
  map<T>(fn: (decodedValue: string) => T): T {
    return fn(Buffer.from(this.base64EncodedValue, 'base64').toString());
  }

  static fromString (value: string): XMLData {
    return new XMLData(Buffer.from(value).toString('base64'));
  }
}

Then I do a broadcast like this:

private broadcast() {
    this.stage.announce(new ArtifactGenerated(
      new Name('junit-result.xml'), XMLData.fromString(this.testSuites.end({pretty: true} as XMLToStringOptions))
    ));
  }

Report is not created, I believe because ArtifactArchiver has a check for:

if (event.artifact instanceof Photo){...}
if (event.artifact instanceof TestReport) {...}

Any help on how to proceed is appreciated.
Thanks.

[jan-molak]

@aksbenz - If you'd like to raise a PR we could work on it together?

[aksbenz]

@jan-molak : Not experienced with open source projects. I am not sure how to go about creating another junit reporter package in serenity.

My Junit reporter looks like this:
https://github.com/aksbenz/serenityjs-playwright-jasmine-junit/blob/master/src/JUnitReporter.ts

Created a copy of ArtifactArchiver with XMLData:
https://github.com/aksbenz/serenityjs-playwright-jasmine-junit/blob/master/src/JUnitArchiver.ts

Thanks.

[aksbenz]

Please check pr for initial steps towards junit reporter. Thanks.
#571

[jan-molak]

Note to self: Looks like Gitlab supports attaching screenshots to JUnit reports. Same with Jenkins.

[jan-molak]

Hey folks! Looks like quite a few people have asked for this functionality.

I did have a closer look at what is required to make it happen and it feels like it would be a good opportunity to improve how Serenity/JS artifacts work in general. I'm planning to make the mechanism more flexible to support not just JUnit, but also other custom report and artifact types in the future.

If you think that it's a worthwhile effort, and you too would like to see this feature added, please consider sponsoring this goal at https://github.com/sponsors/serenity-js