What's the best way to report pass / fail to sauce labs?

[ericelliott]

Orchestrating sauce labs runs with selenium-sauce, what's the best way to report pass / fail to Sauce Labs?

var
  test = require('tape'),
  SeSauce = require('selenium-sauce');

// Loads the config file and invokes the callback once for each browser
new SeSauce(
    {   // Configuration options
        quiet: false,           // Silences the console output
        webdriver: {            // Options for Selenium WebDriver (WebdriverIO)
            user: process.env.SAUCE_USERNAME,
            key: process.env.SAUCE_ACCESS_KEY,
            desiredCapabilities: [{
                browserName: 'chrome',
                version: '27',
                platform: 'XP',
                tags: ['examples'],
                name: 'This is an example test'
            }]
        },
        httpServer: {           // Options for local http server (npmjs.org/package/http-server)
            port: 8080              // Non-standard option; it is passed into the httpServer.listen() method
        },
        sauceLabs: {            // Options for SauceLabs API wrapper (npmjs.org/package/saucelabs)
            username: process.env.SAUCE_USERNAME,
            password: process.env.SAUCE_ACCESS_KEY
        },
        sauceConnect: {         // Options for SauceLabs Connect (npmjs.org/package/sauce-connect-launcher)
            username: process.env.SAUCE_USERNAME,
            accessKey: process.env.SAUCE_ACCESS_KEY
        },
        selenium: {             // Options for Selenium Standalone Server (npmjs.org/package/selenium-standalone). Only used if you need Selenium running locally.
            args: []                // options to pass to `java -jar selenium-server-standalone-X.XX.X.jar`
        }
    },
    function(browser) {

        // Initialize the browser
        browser.init(function(err) {
            if(err) throw err;

            // Load a url into the browser
            browser.url('http://localhost:8080/test.html', function() {
                var result;
                // Tell SauceLabs that the test was successful
                result = // what do we put here?
                browser.passed(result, myCompletionCallback);
            });
        });
    }
);

[vvo]

@ericelliott Maybe see https://github.com/defunctzombie/zuul which is a test runner capable of running tape tests in saucelabs and marking them as success/error accordingly, see https://github.com/algolia/faux-jax tests and badge :)

[ericelliott]

I'm still interested in knowing if there is a canonical way to just get the results from tape to pass along. =)

[vvo]

https://docs.saucelabs.com/reference/test-configuration/#job-annotation-with-the-rest-api seems the way to go

[tomek-he-him]

@ericelliott did you manage to find a way to do this without pulling out the REST hammer?

[ericelliott]

Not yet... had to turn my attention to other things for a while.

[TehShrike]

I'm running into the same issue trying to integrate my tape tests into BrowserStack.

The results object looks like exactly what I need.

The results object is currently attached to the main test harness, but it doesn't look like the main test harness is ever exposed to where I can get at it.

I would like to be able to do something like this

var tapeResults = require('tape')()._results
tapeResults.on('done', function() {
  reportToBrowserStack(tapeResults.pass, tapeResults.fail)
}

I'm motivated enough to get good automated browser testing that I'm prepared to fork tape to expose the results object.

1. Am I missing anything here, or is there any way for me to get at the results object in tape currently?
2. Would the tape maintainers be willing to merge this functionality back upstream at some point, and if so, do you have any preferences that I could bear in mind as I implement this?

[TehShrike]

I suppose to be thorough, other changes would need to be made in order to collect the results from any test harnesses that were created manually. I'll worry about that later I guess?

[TehShrike]

This is my code to report tape output to BrowserStack using the code in this pull request:

var tapeResults = require('tape-with-automated-reporting').results()

var start = new Date()

tapeResults.on('done', function() {
    var finish = new Date()

    var browserStackResults = {
        runtime: finish.getTime() - start.getTime(),
        total: tapeResults.count,
        passed: tapeResults.pass,
        failed: tapeResults.fail,
        tracebacks: []
    }

    BrowserStack.post('/_report', browserStackResults, function(){})
})

I'll publish it in its own module for BrowserStack users when that pull request is merged. I assume that something similar will do the trick for SauceLabs.

[vvo]

@TehShrike Do you think we could also get events on passed/failed tests so that we can have a feedback loop of the test?

[TehShrike]

For Saucelabs? It doesn't look like the results object emits many events at the moment.

[dzcpy]

zuul currently doesn't update job's annotation data (especially passed field) after test finishes. However it's required by sauceslabs' documentation. So you can't get a correct status bardge or browser matrix widget. See #288.

[ericelliott]

Anybody using sauce-tap-runner? How's that working for you?

[TehShrike]

As an aside, I should have linked to this module in this thread after I published it: browserstack-tape-roperter. It looks like using something similar with the sauce-results module might be reasonable.