Support for VSTS & VSTS Test Attachments

[jessehouwing]

Current behavior:

I've installed Cypress into a project built on VSTS. To get cypress to run on the Linux Hosted agents I had to perform the following additional steps:

apt-get -qq -y update
apt-get -qq -y install xvfb libgtk2.0-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2

To get reporting to work (to get status information in the VSTS release dashboard) I configured the junit reporter:

cypress run --reporter junit

This works up to a certain point. Cypress runs, a junit report is generated, but all of the recorded videos are ignored after the run is completed.

VSTS supports uploading test attachments when they are referenced in the test report. I tried configuring the xunit-file reporter, but no attachment is added. I tried configuring the mocha-trx-reporter (native visual studio format), but it crashes when running with cypress.

TypeError: test.isPending is not a function
    at C:\temp\node_modules\mocha-trx-reporter\lib\trx.js:63:22
    at Array.forEach (<anonymous>:null:null)
    at Runner.<anonymous> (C:\temp\node_modules\mocha-trx-reporter\lib\trx.js:62:34)
    at emitOne (events.js:120:20)
    at Runner.emit (events.js:210:7)
    at Reporter.emit (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\reporter.js:191:55)
    at Object.server.startWebsockets.onMocha (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\project.js:288:22)
    at Socket.<anonymous> (C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\server\lib\socket.js:237:36)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at C:\temp\node_modules\cypress\dist\Cypress\resources\app\packages\socket\node_modules\socket.io\lib\socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Desired behavior:

Cypress logs the recorded video as test attachment in the generated test results so that Visual Studio Team Services can associate them to the test results and show them in the portal.

Steps to reproduce:

• Run cypress in a Visual Studio Team Services build.
• Run the "Upload test results" step and publish the results file

• Check the VSTS portal for the presence of test attachments.

Versions

• Windows 10 April 2018 or Ubuntu (https://github.com/Microsoft/vsts-agent-docker/blob/master/ubuntu/16.04/standard/Dockerfile)
• Cypress: 2.1.0
• VSTS Agent version: 2.133.3

[SeriousM]

any updates on this?

[erik-peterson-fox]

Please add support for this if it's not already there...This would be very useful.

[jennifer-shehane]

Some related work here: #2433

[SebastianSchoeberl]

Any updates here?
As far as I know cypres uses the Mocha-Junit-Reporter, which should support attachment files already. Anybody got that working with cypress and CI/CD?

[YOU54F]

I came across this error today, on a totally unrelated issue, see my CI build here

$(npm bin)/cypress run --spec cypress/integration/sample1.spec.js --reporter-options "reportFilename=test1"

For whatever reason, it was trying to use the xunit reporter, despite the fact that my reporterOptions are set to use cypress-multi-reporters, mochawesome and mocha-junit-reporters.

If i comment out the following function at node_modules/mocha/lib/reporters/xunit.js

https://github.com/mochajs/mocha/blob/f3ed729b6908f3893e6f87722c6e996cca5f7733/lib/reporters/xunit.js#L182

I get a xunit xml file at the root of my project called xunit.xml

<testsuite name="Mocha Tests" tests="1" failures="0" errors="0" skipped="0" timestamp="Tue, 04 Jun 2019 23:30:02 GMT" time="2.259">
<testcase classname="Cypress parallel run example - 1" name="should display the title" time="0.645"/>
</testsuite>

This is the equivalent junit report

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.527" tests="1" failures="0">
  <testsuite name="Root Suite" timestamp="2019-06-04T23:39:48" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="Cypress parallel run example - 1" timestamp="2019-06-04T23:39:48" tests="1" failures="0" time="0.527">
    <testcase name="Cypress parallel run example - 1 should display the title" time="0.527" classname="should display the title">
    </testcase>
  </testsuite>
</testsuites>

I'm not entirely sure why an xunit file is being emitted! as it isn't specified in any of my setup!

Here is my cypress.json

{
  "reporter": "cypress-multi-reporters",
  "reporterOptions": {
    "configFile": "reporterOpts.json"
  }
}

and reporterOpts.json

{
  "reporterEnabled": "mocha-junit-reporters, mochawesome",
  "mochaJunitReportersReporterOptions": {
    "mochaFile": "cypress/reports/junit/test_results[hash].xml",
    "toConsole": false
  },
  "mochawesomeReporterOptions": {
    "reportDir": "cypress/reports/mocha",
    "quiet": true,
    "overwrite": false,
    "html": false,
    "json": true
  }
} 

It may or may not be relevant to this issue, just thought I would help add some additional info!

[marceloavf]

Still nothing @SebastianSchoeberl

I saw that they support Selenium, publishing images and test results, but for now I'm only testing e2e and failing the build if they don't pass, not recording anything in portal.

[SebastianSchoeberl]

Thanks for your update @marceloavf , well I suppose we have to hold on a little more. But would be great if they made it possible!

[kesiyaabraham]

Hi ,

Do we have any other option available in cypress to attach screenshots and videos to azure test results other than directly using DevOps API?

[krileo]

I wrote a PowerShell script that can be run as a task in the Azure Pipeline. It will locate and attach the screenshots from Cypress to the tests results for failing tests.

You can find the code here and a blog post on how to use it here.