parallel reporting

[nprosenikov]

Hello,

For some reason the parallel run doest work with reporting and it only shows the results in the terminal, and doesn't create normal report if you use it with mochawesome reporting for example.
Best Regards

[prasanmgc]

This plugin uses custom reporter which prints the results in console and doesn't create any html reports. You need to create your own reports using various reporting plugins in the market. @tnicola mentioned that he will be working on html report in near future. so you can create using this plugin.

[Leop2020]

The HTML report generated using mochawesome reporter seems to be listing only the details of the last executed spec file.

Is there any specific configuration that I am missing while using this plugin for parallel testing ?

[hai-le-niteco]

The HTML report generated using mochawesome reporter seems to be listing only the details of the last executed spec file.

Is there any specific configuration that I am missing while using this plugin for parallel testing ?

You can use this: "overwrite": false to let it generate different html report for each thread.

[VicenteRuizWefoxgroup]

How can I keep using mochawesome as I used to do? I noticed that when I run the script for parallel execution, this is hardcoded:
"--reporter" "cypress-multi-reporters" "--reporter-options" "configFile=/Users/user/workspace/myProject/test-folder/multi-reporter-config.json" and the multi-reporter-config.json file is overwritten every time I launch the tests.

If I try to set the configFile parameter, the execution fails in the end because can't find the runner-results folder.

[stevenmchaves]

I have been struggling with the same.

…

On Wed, Nov 17, 2021 at 9:16 AM vicente.ruiz ***@***.***> wrote: How can I keep using mochawesome as I used to do? I noticed that when I run the script for parallel execution, this is hardcoded: "--reporter" "cypress-multi-reporters" "--reporter-options" "configFile=/Users/user/workspace/myProject/test-folder/multi-reporter-config.json" and the multi-reporter-config.json file is overwritten every time I launch the tests. If I try to set the configFile parameter, the execution fails in the end because can't find the runner-results folder. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMR4WMK4IGJKYGAW4S22HH3UMO2K5ANCNFSM4ZNCS2QA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[TenPotatoes]

I was confused by this too. It seems you cannot specify configFile, but you can pass in everything that was in your config file as args.

[psrebrny]

multi-reporter-config.json is overwritten every time when tests are run but you can keep your config in another file for instance reporter.config.json then the file multi-reporter-config.json will be overridden but the config will be read and the test result will be generated like on clear cypress run.

[nathaniel-k-woo]

multi-reporter-config.json is overwritten every time when tests are run but you can keep your config in another file for instance reporter.config.json then the file multi-reporter-config.json will be overridden but the config will be read and the test result will be generated like on clear cypress run.

I'm struggling with utilizing the reporter.config.json file. For my command, I'm running

cypress-parallel -s test-ct-cmd -t 5 -d src/test/**/**.spec.{js,ts,tsx,jsx} -r mochawesome -o configFile=reporter.config.json

and the generated file reflects the reporter option:

{ "reporterEnabled": "cypress-parallel/json-stream.reporter.js, mochawesome", "mochawesomeReporterOptions": { "configFile": "reporter.config.json" } }

but none of the options set within the reporter.config.json file are being respected. Any ideas?

[psrebrny]

multi-reporter-config.json is overwritten every time when tests are run but you can keep your config in another file for instance reporter.config.json then the file multi-reporter-config.json will be overridden but the config will be read and the test result will be generated like on clear cypress run.

I'm struggling with utilizing the reporter.config.json file. For my command, I'm running

cypress-parallel -s test-ct-cmd -t 5 -d src/test/**/**.spec.{js,ts,tsx,jsx} -r mochawesome -o configFile=reporter.config.json

and the generated file reflects the reporter option:

{ "reporterEnabled": "cypress-parallel/json-stream.reporter.js, mochawesome", "mochawesomeReporterOptions": { "configFile": "reporter.config.json" } }

but none of the options set within the reporter.config.json file are being respected. Any ideas?

too less data to understand the problem,

At first tell me if the options are respected when you try run command by test-ct-cmd

I will show you my full command

"cy:parallel": "cypress-parallel -s cypress:run -t 4 -d cypress/integrationxxx -a '\"--browser chrome\"' --reporter cypress-multi-reporters --reporter-options configFile=reporter-config.json",

and "cypress:run": "./node_modules/.bin/cypress run",

and then in my reporter config file I'm using more that one reporter

{ "reporterEnabled": "mochawesome, mocha-junit-reporter", "mochaJunitReporterReporterOptions": { "mochaFile": "tests-results/results-[hash].xml" }, "mochawesomeReporterOptions": { "reportDir": "tests-results", "overwrite": false, "cdn": true, "json": true, "html": false } }

maybe try use cypress-multi-reporters and then join mochaawesome with mocha-junit-reporter

[barnie-karbonhq]

any luck with this, any one? Thanks

[rgangal]

any update on this issue? mochawesome is only reporting the last run tests.

[barnie-karbonhq]

I have set it up as below and it's working for me...

On my package.json

"scripts": {
    "delete:reports": "rm -rf cypress/reports && rm -rf cypress/archive",
    "make:directories": "mkdir -p cypress/reports/mochareports && mkdir -p cypress/archive",
    "pretest": "npm run delete:reports && npm run make:directories",
    "combine-reports": "mochawesome-merge cypress/reports/mocha/*.json > cypress/reports/mochareports/report.json",
    "combine-xml-reports": "jrm test-results.xml cypress/reports/*.xml",
    "generate-report": "marge cypress/reports/mochareports/report.json -f report -o cypress/reports/mochareports",
    "posttest": "npm run combine-xml-reports && npm run combine-reports && npm run generate-report",
    "cy:run": "cypress run --config-file scripts/config/cypress_develop.config.js --config video=false",
    "cy:parallel:all" : "npx cypress-parallel -s cy:run -t 6 -d 'cypress/e2e/feature-tests/**' --reporter cypress-multi-reporters --reporter-options configFile=reporter-config.json",
  },

And I can execute the test using a .sh file with below commands, which generates the mocha reports fine.

npm run pretest
npm run cy:parallel:all
npm run posttest

[lems3]

I had a different approach for mochawesome in my project. There's some elements in common with previous solutions :

In package.json

  "scripts": {
    "cy:run": "cypress run --browser=chrome",
    "cy:parallel-test": "cypress-parallel -s cy:run -t 3 -d 'cypress/e2e/test/**/*.cy.js' -r 'mochawesome' -o 'reportDir=report/json,overwrite=false,html=false,json=true'",
    "report:clean": "rm -r report",
    "report:merge": "mochawesome-merge 'report/json/*.json' > report/mochawesome.json",
    "report:build": "marge report/mochawesome.json -f index -o report --assetsDir report/assets",
    "report:serve": "npx http-server ./report",
    "cy:run:test": "npm run report:clean; npm run cy:parallel-test; npm run report:merge && npm run report:build"
  },

So my cy:run:test takes care of everything. (test is the name of a folder regrouping all tests that target a test environment. I would use cy:run:staging if the folder and the target environment was named staging).

The workflow in all solutions is the same :

1. Manually clean the report folders
2. Launch test with cypress-parallel by telling mochawesome to generate json files and not to override files
3. Merge all report files
4. Generate the report

I have 0 mochawesome configuration in my cypress.config.js file. The reporter configuration needs to be done trough cypress-parallel to work properly.

Sources for this solutions comes from this thread, and Cypress doc : https://github.com/cypress-io/testing-workshop-cypress/blob/master/slides/09-reporters/PITCHME.md#mochawesome

[SaulGoode77]

I had a different approach for mochawesome in my project. There's some elements in common with previous solutions :

In package.json

  "scripts": {
    "cy:run": "cypress run --browser=chrome",
    "cy:parallel-test": "cypress-parallel -s cy:run -t 3 -d 'cypress/e2e/test/**/*.cy.js' -r 'mochawesome' -o 'reportDir=report/json,overwrite=false,html=false,json=true'",
    "report:clean": "rm -r report",
    "report:merge": "mochawesome-merge 'report/json/*.json' > report/mochawesome.json",
    "report:build": "marge report/mochawesome.json -f index -o report --assetsDir report/assets",
    "report:serve": "npx http-server ./report",
    "cy:run:test": "npm run report:clean; npm run cy:parallel-test; npm run report:merge && npm run report:build"
  },

So my cy:run:test takes care of everything. (test is the name of a folder regrouping all tests that target a test environment. I would use cy:run:staging if the folder and the target environment was named staging).

The workflow in all solutions is the same :

1. Manually clean the report folders
2. Launch test with cypress-parallel by telling mochawesome to generate json files and not to override files
3. Merge all report files
4. Generate the report

I have 0 mochawesome configuration in my cypress.config.js file. The reporter configuration needs to be done trough cypress-parallel to work properly.

Sources for this solutions comes from this thread, and Cypress doc : https://github.com/cypress-io/testing-workshop-cypress/blob/master/slides/09-reporters/PITCHME.md#mochawesome

This seems to work perfectly, the only issue I have is that the Screenshots are not appearing in the report. Is there some other config I'm missing?

[scoca91]

solved this issue as following:

my configuration File:

const { defineConfig } = require("cypress");
module.exports = defineConfig({

e2e: {
chromeWebSecurity: false,
experimentalSessionAndOrigin: true,
// other e2e configurations
setupNodeEvents(on, config) {

  config.reporter = "mochawesome";
  // Specify the reporter options for mochawesome
  const reporterOptions = {
    reportDir: 'cypress/reports',
    charts: true,
    overwrite: false,
    reportPageTitle: "custom-title",
    embeddedScreenshots: true,
    inlineAssets: true,
    saveAllAttempts: false,
    html: true,
    json: false,
  };
  // Update the reporterOptions for mochawesome
  config.reporterOptions = Object.assign(
    {},
    config.reporterOptions,
    reporterOptions
  );
  // Return the updated config
  return config;

},


env: {
  //UR env data here
},
video: true,
videoUploadOnPasses: false,
viewportHeight: 1080,
viewportWidth: 1920,
pageLoadTimeout: 30000,
requestTimeout: 60000,
defaultCommandTimeout: 40000,
retries: 0,
numTestsKeptInMemory: 1,
chromeWebSecurity: false,

}
});

in package.json i have the following script:
"run": "npx cypress run",
"run-parallel": "npx cypress-parallel -d cypress/e2e/ParallelExecution -a '"--e2e --config-file cypress/configs/cypress.app.config.js"' -t 15 -s run -r mochawesome"

where:
-d is the folder where i have the spec files(test files) for parallel execution
-a = arguments that points to my config file

[umairmah]

The solution above is working fine. The only issue I am facing is that since I am running with 4 threads, 4 html files generated at the end of the run. Any idea how to to get 1 html instead of 4?

[lems3]

https://github.com/PremierTechDigital/cypress-parallel-report

For those interested, I built a repository to demonstrate our final solution to run Cypress parallel and get a report.

At the end, we have one report, with some tweaks so the number of success and failures can bubble up properly if you have nested describes.

This repo helped in this issue : LironEr/cypress-mochawesome-reporter#44 (comment) and there's a new version of cypress mochawesome reporter that should handle this properly. I didn't had time to test it though.

[SumanthMalisetty]

• I am using cypress-parallel package to run cypress suites. I am having 4 suites( 4 threads or 4 describe blocks ).
• How to generate a single html report for this kind of situation. Any idea. I tried, but none of the methods worked.
• Maybe I might be missing something. Can someone help me.....

@umairmah @lems3 @scoca91 @SaulGoode77

[LironEr]

cypress-mochawesome-reporter just released with support of cypress-parallel

Check out the setup guide

[umairmah]

The new released version of cypress-mochawesome-reporter is working perfectly with cypress-parallel. There is just one issue when ruuning the tests in the build pipeline:

Previously when the tests failed in the build pipeline, build failed with exit code based on the number of tests failed. But not the case anyomore after using the parallel execution. Any idea how to get the build failed with exit code in cypress-parallel incase some tests failed? Thanks

[grbaran]

The new released version of cypress-mochawesome-reporter is working perfectly with cypress-parallel. There is just one issue when ruuning the tests in the build pipeline:

Previously when the tests failed in the build pipeline, build failed with exit code based on the number of tests failed. But not the case anyomore after using the parallel execution. Any idea how to get the build failed with exit code in cypress-parallel incase some tests failed? Thanks

I'm just having the same issue. Looks like no matter how many tests fail now on CI (using github actions) - the job always ends up as 'passed'

[umairmah]

The new released version of cypress-mochawesome-reporter is working perfectly with cypress-parallel. There is just one issue when ruuning the tests in the build pipeline:
Previously when the tests failed in the build pipeline, build failed with exit code based on the number of tests failed. But not the case anyomore after using the parallel execution. Any idea how to get the build failed with exit code in cypress-parallel incase some tests failed? Thanks

I'm just having the same issue. Looks like no matter how many tests fail now on CI (using github actions) - the job always ends up as 'passed'

@LironEr @lems3 @scoca91 @barnie-karbonhq
Maybe any of you can assist on this issue. We are no longer getting exit code in case of failed tests in the CI. Maybe you guys came across this issue and have some resolution? Thanks

[LironEr]

Just released a new version of cypress-mochawesome-reporter (v3.7.0) and I updated the setup guide.

npx generate-mochawesome-report --set-exit-code

Will set the exit code to the number of failed tests.

[peacetrader]

how to use cypress-parallel with https://www.npmjs.com/package/multiple-cucumber-html-reporter?.
I would appreciate any help please. This would benefit many people as cucumber is very popular and widely used in testing.