[Feature] Combine sharding with html reporter #10437

bamanczak · 2021-11-19T12:56:43Z

Currently there is no way to combine test results into one html report, while using sharding on multiple machines. (v1.17.0-rc1)

I end up with multiple reports which makes it harder for sharing with others in case of failures.

It would be nice to be able to combine multiple executions into 1 report.

I believe anyone who wants to scale tests horizontally might benefit from such feature

JoelEinbinder · 2021-11-19T17:14:46Z

Sounds reasonable to me! What kind of api would you like to combine the reports together?

bamanczak · 2021-11-22T09:49:14Z

I personally have no preferences for how to merge the reports together. Whatever I can do on CI in a fast way.

MindaugasMateika · 2022-03-03T12:15:31Z

Since I'm currently using https://plugins.jenkins.io/allure-jenkins-plugin/ to merge multiple Allure reports and it is as simple as 👇 then I'd favor a similar approach to HTML reporter but any other out-of-the-box no-brainer solution would work as well.

            script {
                allure([
                        commandline      : 'allure-2.x',
                        includeProperties: false,
                        jdk              : '',
                        results          : [
                                [path: 'allure-results1'],
                                [path: 'allure-results2']
                        ]
                ])

liamdebeasi · 2022-03-21T17:19:22Z

This feature would be great to have. We are looking to migrate our E2E testing solution to Playwright. We have a component library where each test runs several variations of a particular test to check for visual regressions. Some of the variants we have are:

LTR vs RTL modes
Desktop vs mobile viewports
Different themes for the components

Right now the results for these tests can get split up across multiple shards. It would be great to have a single result where we can see all of the test results.

anooprav7 · 2022-03-28T17:23:26Z

I was able to build one solution for my use case (in Node.js). I had multiple reports from different shards (tests distributed with a custom logic) in a single run and wanted to send a single merged html report. Hope it helps.

Installation

npm install playwright-merge-html-reports --dev

Usage

const { mergeHTMLReports } = require("playwright-merge-html-reports");

mergeHTMLReports([
  process.cwd() + "/html_report-1",
  process.cwd() + "/html_report-2"
])

It creates a separate HTML merged report, folder name - merged-html-report (customizable)

⚠️ Tested on some use cases: Still at early stages of development

Github Repo
NPM package

btd · 2022-03-28T19:59:33Z

@anooprav7 thanks kind men. I will try this in a week or two.

ghost · 2022-04-28T06:43:12Z

Is this feature still in progress?

bamanczak · 2022-05-16T16:01:44Z

My test suite becomes bigger and bigger every week, and this is a blocker for my team to start using sharding.

The solution mentioned by anooprav7 seems cool, but we would like to have a stable version of such toolkit. I was really hoping for it to appear in 1.22 :(

mrmckeb · 2022-06-02T02:19:34Z

We're running our tests on GitHub Actions. We'd ideally like a single report, but for now will create a report per shard/matrix runner.

I'm not sure how this should work - there's no easy mechanism for the runners to talk to each other to combine a report.

mstepin · 2022-06-10T06:02:18Z

wanting this too. We run our tests back to back with various different configurations and want to combine the final test results into a single html. So pls team at MS and Playwright, at some point in future, make this a native playwright feature.
In the meantime, we have wired in the module provided by @anooprav7 . There are a bunch of issues with that module that we needed to address so just a heads up to anyone else try it.

We use NodeJS 12.X. This limits us to Playwright 1.21.X or lower. The module looks for fs/promises but cant find it. We've had to alias in the fs.promises module to that path.
If a test is successful and no data is generated as a result (in the 'data' directory) it also breaks, as it cant find the dir. So we've mitigated it by manually creating that dir no matter what.
I believe there is a PR to fix both the above waiting to merged in gitHub for that module. But as for now there is no ETA. @anooprav7 pls let us know when the fix is available.

mrmckeb · 2022-06-11T09:12:04Z

Feel free to install from my (unpublished) fork in the meantime, @mstepin. We're using this on one of our repositories at Vercel right now and it seems like we have everything working.
https://github.com/mrmckeb/playwright-merge-html-reports/tree/with-dist

npm i https://github.com/mrmckeb/playwright-merge-html-reports#aadf6a7

anooprav7 · 2022-06-11T10:20:37Z

The fix has been merged and published - playwright-merge-html-reports@0.2.2. Thank you @mrmckeb and @nickofthyme for the fixes (anooprav7/playwright-merge-html-reports#2 and anooprav7/playwright-merge-html-reports#3).

TimeInvestor · 2022-07-01T07:02:46Z

Looking forward to this feature.
Our tests take more than 1 hour to finish using one Jenkins job.
We have to split the tests into mutiple Jenkins jobs to run across mutiple Jenkins agents using Playwright sharding feature.
However, getting a single HTML test report is an issue with this solution. We need to pass HTML test report for release team to review before they do release.

nickofthyme · 2022-07-01T14:16:46Z

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use playwright-merge-html-reports@0.2.3 npm package to combine all the html reports into one.

TimeInvestor · 2022-07-01T14:37:55Z

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use playwright-merge-html-reports@0.2.3 npm package to combine all the html reports into one.

Thanks @nickofthyme. Will check it out although I am not sure if it works well for my scenario that the test results are residing with multiple Jenkins jobs over different Jenkins slaves.

damongabrielle · 2023-07-24T20:42:43Z

I came back to this issue today to check on any progress and I noticed the blob reporter merged back in April in v1.35.1. There was no doc on it so I wasn't sure if if the feature fully addressed this issue. For me at least, this turned out to be pretty great.

I enabled the blob reporter then ran:

npx playwright test --shard=1/3 
npx playwright test --shard=2/3 
npx playwright test --shard=3/3

All results were placed in the blob-report directory

then I ran
npx playwright merge-reports blob-report --reporter=html
and it produced a playwright-report directory with combined results.

and I then ran

npx playwright merge-reports blob-report --reporter=junit > playwright-report/results.xml

and it produced the xml output.
I guess the junit reporter must default standard out. I didn't expect to have to redirect the output but I guess it might not be reading the reporting configuration yet.
Overall, I'm pretty excited to have this functionality built in. Are there additional pieces coming that explain why this feature request is still open? Just trying to gauge the risks of using this before it's officially documented.

damongabrielle · 2023-07-24T20:57:15Z

It would be pretty cool if it produced a merged report for every report type that was already defined in the report config. If I left off the--reporter flag it didn't produce any reports and it didn't seem to fail either. Apologies if this is a work in progress. I am pretty psyched to have what's already there!

costa-collibra · 2023-07-25T09:43:06Z

@damongabrielle oh wow, i just replicated your steps and got it working against 3 shards.

Im a little reluctant to use it though without any official support from Playwright. This is pretty awesome though

swetrify · 2023-07-25T11:59:20Z

@damongabrielle @costa-collibra, yeah, it's working 😮, but I'm facing a problem when running on CI. The HTML report opens automatically if some of the tests fail. Unfortunately, merge-reports is ignoring the 'open' property in the Playwright config, and I don't see any other way to configure that. 😢 I'm not sure if I'm missing something, if there's a workaround for it, or maybe that's the reason why merging multiple reports is not officially supported yet.

There is even a documentation https://playwright.dev/docs/next/test-merge-report (it's marked as 'Version: Next' though)

MindaugasMateika · 2023-07-25T12:14:19Z

@damongabrielle @costa-collibra, yeah, it's working 😮, but I'm facing a problem when running on CI. The HTML report opens automatically if some of the tests fail. Unfortunately, merge-reports is ignoring the 'open' property in the Playwright config, and I don't see any other way to configure that. 😢 I'm not sure if I'm missing something, if there's a workaround for it, or maybe that's the reason why merging multiple reports is not officially supported yet.

There is even a documentation https://playwright.dev/docs/next/test-merge-report (it's marked as 'Version: Next' though)

try setting env variable PW_TEST_HTML_REPORT_OPEN: 'never'

swetrify · 2023-07-25T12:24:59Z

@MindaugasMateika Oh, now I see where I have made a mistake (I forgot to pass CI=true when running on my test setup). But setting it as an environment variable PW_TEST_HTML_REPORT_OPEN='never' works too! Thank you.

swetrify · 2023-07-25T13:00:06Z

I have noticed one problem with merging when using the setup test (https://playwright.dev/docs/test-global-setup-teardown#setup-example).

When running with --shard=1/2 and shard=2/2, the setup test is executed once per each shard (so 2x in my example). When merging these two tests, the resulting report is broken for these two tests.

When I click on these two tests, most of the resulting page is empty.

Other than that, it looks great!

costa-collibra · 2023-07-25T13:20:54Z

@swetrify is is running the global setup twice or just reporting on it twice?

If its running setup twice that seems a pretty big bug/oversight

costa-collibra · 2023-07-25T13:22:03Z

@damongabrielle @MindaugasMateika

I have this set

const reportList: ReporterDescription[] = [['list'], ['blob'], ['html', { open: 'never' }]];

to prevent the report from opening

swetrify · 2023-07-25T14:00:51Z

@swetrify is is running the global setup twice or just reporting on it twice?

If its running setup twice that seems a pretty big bug/oversight

The setup is running on each shard once, resulting in the global setup being executed twice (in my example with 2 shards). When I don't merge the reports, each HTML file contains a result of the global setup that differs from each other. I am not sure if it's a bug or a feature. The documentation for sharding and global setup is not clear on this matter.

Edit: I think it's a feature. It does make sense in the context of the login example. You need to create storageState on each shard; otherwise, one of the shards wouldn't be able to log in because it would be missing storageState.

damongabrielle · 2023-07-25T14:25:47Z

I'm surprised @costa-collibra that the report config open: 'never' is working for you during the execution of merge-reports. I have open: 'never' set but that doesn't work for me on merge-reports. That has always successfully prevented the report from opening at the end of a test run though. I wonder if you have additionally set CI=true or PW_TEST_HTML_REPORT_OPEN='never'. Both of those fixed it for me.

costa-collibra · 2023-07-25T14:29:41Z

@damongabrielle actually you are correct on that part I misread. The report does open when I merged. Its great though that you have a solution as we will need to set this in the pipeline.

angelo-loria · 2023-08-10T19:59:24Z

Here's another example of an Actions file using the blob reporter and merging CLI. This merges to XML for use with dorny-test-reporter and deploys the HTML report to GH Pages:

.
.
.
jobs:
  playwright_test:
    name: Playwright Test
    strategy:
      fail-fast: false
      matrix:
        shard: [1/3, 2/3, 3/3]
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v3

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODEJS_VERSION }}

      - name: Yarn Install
        run: yarn install
        
      - name: Setup
        run: yarn setup

      - name: Run Tests
        run: yarn test:ci --shard ${{ matrix.shard }}

      - name: Upload blob report to GitHub Actions Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: all-blob-reports
          path: ./playwright/blob-report/
          retention-days: 10
      
      # screenshot/video/trace is stored in test-results if a test fails, otherwise empty if all tests pass
      - name: Upload test-results to GitHub Actions Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-results
          path: ./playwright/test-results/
          retention-days: 10
          if-no-files-found: warn
          
  reporting:
    if: always()
    needs: [playwright_test]
    name: Reporting
    outputs:
      page_url: ${{ steps.deployment.outputs.page_url }}
    environment:
      name: ${{ inputs.environment }}-report
      url: ${{ steps.deployment.outputs.page_url }}
    defaults:
      run:
        working-directory: ./playwright

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
 
    - uses: actions/setup-node@v3
 
    - name: Yarn Install
      run: yarn install

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: all-blob-reports
        path: ./playwright/all-blob-reports
    
    # test-results directory is only created when a test fails, if all pass there is no dir to download
    - name: Check for test-results Artifact
      uses: xSAVIKx/artifact-exists-action@v0
      id: check-artifact
      with:
        name: test-results

    # TODO - watch for upcoming ignore feature on this so that the above step can be removed
    - name: Download test-results from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      if: ${{ steps.check-artifact.outputs.exists == 'true' }}
      with:
        name: test-results
        path: ./playwright/test-results
      
    # xml report use by dorny-test-reporter
    - name: Merge into XML Report
      run: PLAYWRIGHT_JUNIT_OUTPUT_NAME=junit.xml npx playwright merge-reports --reporter=junit ./all-blob-reports
        
    - name: Test Report
      uses: dorny/test-reporter@v1
      if: success() || failure()
      with:
        name: Playwright Test Report
        path: "**/junit.xml"
        reporter: java-junit
        fail-on-error: 'false'

    - name: Merge into HTML Report
      run: npx playwright merge-reports --reporter html ./all-blob-reports 

    - name: Upload HTML report
      uses: actions/upload-pages-artifact@v1
      with:
        name: github-pages
        path: './playwright/all-blob-reports'
        retention-days: 14
        
    - name: Setup Pages
      uses: actions/configure-pages@v3

    - uses: actions/download-artifact@v3
      with:
        name: github-pages

    - name: Deploy to GitHub Pages
      id: deployment
      uses: actions/deploy-pages@v1

aslushnikov · 2023-08-11T10:29:37Z

Hey everybody,

This has been released in 1.37.

Documentation: https://playwright.dev/docs/test-sharding
Release notes: https://github.com/microsoft/playwright/releases/tag/v1.37.0
Video overview: https://youtu.be/cEd4SH_Xf5U

Thank you all for the feedback!

marwie · 2023-08-12T12:55:33Z

Hi, does this also support mergin of custom reports? For example we use a custom json reporter.
If not do you have any hints on how we could accomplish this?

I assume I can implement a merge method in my reporter?

aslushnikov · 2023-08-14T12:00:24Z

@marwie custom reports are supported as well. Make sure to specify it when using the merge-repots CLI tool and it should all just work.

marwie · 2023-08-14T12:21:40Z

Hi @aslushnikov thanks for the reply.

Should i pass in the path to my reporter as an argument then and is it possible to specify multiple reporters (e.g. can I both use the html and my reporter like so: --reporter html ./myreporter.js ?)

aslushnikov · 2023-08-14T13:36:14Z

@marwie you'd need to:

create a custom config for the merge-reports tool
configure your custom reporter in that config: https://playwright.dev/docs/next/test-reporters#custom-reporters
use the --config option to the merge-reports tool to pass in the config: https://playwright.dev/docs/next/test-sharding#merge-reports-cli

shaiagga · 2024-03-04T06:10:00Z

Hi,
I have a requirement where I have to combine the reports for that I am using playwright-merge-html-reports
@anooprav7
This is generating the reports with html content only, I also want to merge the xml report with that, do we have a way to do this via this library

rwoll mentioned this issue Nov 19, 2021

[Feature] HTML Reporter Time-Series #10158

Closed

JoelEinbinder added the P3-collecting-feedback label Nov 19, 2021

Nav-2d mentioned this issue Dec 6, 2021

[Feature] make possible merge html reports #10733

Closed

dgozman assigned pavelfeldman Dec 7, 2021

dgozman added the v1.19 label Dec 7, 2021

Nav-2d mentioned this issue Dec 10, 2021

[BUG] Retain retry history in Playwright Html report #10859

Closed

pavelfeldman added v1.20 feature-html-reporter and removed v1.19 labels Jan 25, 2022

aslushnikov added v1.21 and removed v1.20 labels Mar 1, 2022

mxschmitt added v1.22 and removed v1.21 P3-collecting-feedback labels Apr 4, 2022

mxschmitt removed the v1.22 label Apr 20, 2022

aslushnikov added the v1.35 label May 18, 2023

aslushnikov added v1.36 and removed v1.35 labels Jun 5, 2023

tihuan mentioned this issue Jun 15, 2023

Shard E2E tests in the CI chanzuckerberg/single-cell-data-portal#4488

Closed

inomdzhon mentioned this issue Jun 19, 2023

ci(e2e): use playwright shard feature VKCOM/VKUI#5308

Closed

aslushnikov added v1.37 and removed v1.36 labels Jul 7, 2023

mxschmitt mentioned this issue Jul 11, 2023

Observation_How can i run Multiple scripts with different cross-env values and fetch in single report #24158

Closed

aslushnikov closed this as completed Aug 11, 2023

tihuan mentioned this issue Aug 11, 2023

Shard tests to support multiple browser tests and combine test report chanzuckerberg/single-cell-data-portal#5449

Closed

2 tasks

kyranjamie mentioned this issue Aug 14, 2023

test: merge sharded test reports leather-io/extension#4129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Combine sharding with html reporter #10437

[Feature] Combine sharding with html reporter #10437

bamanczak commented Nov 19, 2021

JoelEinbinder commented Nov 19, 2021

bamanczak commented Nov 22, 2021

MindaugasMateika commented Mar 3, 2022 •

edited

Loading

liamdebeasi commented Mar 21, 2022

anooprav7 commented Mar 28, 2022 •

edited

Loading

btd commented Mar 28, 2022

ghost commented Apr 28, 2022

bamanczak commented May 16, 2022

mrmckeb commented Jun 2, 2022

mstepin commented Jun 10, 2022 •

edited

Loading

mrmckeb commented Jun 11, 2022 •

edited

Loading

anooprav7 commented Jun 11, 2022 •

edited

Loading

TimeInvestor commented Jul 1, 2022

nickofthyme commented Jul 1, 2022

TimeInvestor commented Jul 1, 2022

damongabrielle commented Jul 24, 2023 •

edited

Loading

damongabrielle commented Jul 24, 2023

costa-collibra commented Jul 25, 2023

swetrify commented Jul 25, 2023 •

edited

Loading

MindaugasMateika commented Jul 25, 2023

swetrify commented Jul 25, 2023 •

edited

Loading

swetrify commented Jul 25, 2023

costa-collibra commented Jul 25, 2023

costa-collibra commented Jul 25, 2023

swetrify commented Jul 25, 2023 •

edited

Loading

damongabrielle commented Jul 25, 2023 •

edited

Loading

costa-collibra commented Jul 25, 2023

angelo-loria commented Aug 10, 2023 •

edited

Loading

aslushnikov commented Aug 11, 2023

marwie commented Aug 12, 2023 •

edited

Loading

aslushnikov commented Aug 14, 2023

marwie commented Aug 14, 2023 •

edited

Loading

aslushnikov commented Aug 14, 2023

shaiagga commented Mar 4, 2024

[Feature] Combine sharding with html reporter #10437

[Feature] Combine sharding with html reporter #10437

Comments

bamanczak commented Nov 19, 2021

JoelEinbinder commented Nov 19, 2021

bamanczak commented Nov 22, 2021

MindaugasMateika commented Mar 3, 2022 • edited Loading

liamdebeasi commented Mar 21, 2022

anooprav7 commented Mar 28, 2022 • edited Loading

Installation

Usage

btd commented Mar 28, 2022

ghost commented Apr 28, 2022

bamanczak commented May 16, 2022

mrmckeb commented Jun 2, 2022

mstepin commented Jun 10, 2022 • edited Loading

mrmckeb commented Jun 11, 2022 • edited Loading

anooprav7 commented Jun 11, 2022 • edited Loading

TimeInvestor commented Jul 1, 2022

nickofthyme commented Jul 1, 2022

TimeInvestor commented Jul 1, 2022

damongabrielle commented Jul 24, 2023 • edited Loading

damongabrielle commented Jul 24, 2023

costa-collibra commented Jul 25, 2023

swetrify commented Jul 25, 2023 • edited Loading

MindaugasMateika commented Jul 25, 2023

swetrify commented Jul 25, 2023 • edited Loading

swetrify commented Jul 25, 2023

costa-collibra commented Jul 25, 2023

costa-collibra commented Jul 25, 2023

swetrify commented Jul 25, 2023 • edited Loading

damongabrielle commented Jul 25, 2023 • edited Loading

costa-collibra commented Jul 25, 2023

angelo-loria commented Aug 10, 2023 • edited Loading

aslushnikov commented Aug 11, 2023

marwie commented Aug 12, 2023 • edited Loading

aslushnikov commented Aug 14, 2023

marwie commented Aug 14, 2023 • edited Loading

aslushnikov commented Aug 14, 2023

shaiagga commented Mar 4, 2024

MindaugasMateika commented Mar 3, 2022 •

edited

Loading

anooprav7 commented Mar 28, 2022 •

edited

Loading

mstepin commented Jun 10, 2022 •

edited

Loading

mrmckeb commented Jun 11, 2022 •

edited

Loading

anooprav7 commented Jun 11, 2022 •

edited

Loading

damongabrielle commented Jul 24, 2023 •

edited

Loading

swetrify commented Jul 25, 2023 •

edited

Loading

swetrify commented Jul 25, 2023 •

edited

Loading

swetrify commented Jul 25, 2023 •

edited

Loading

damongabrielle commented Jul 25, 2023 •

edited

Loading

angelo-loria commented Aug 10, 2023 •

edited

Loading

marwie commented Aug 12, 2023 •

edited

Loading

marwie commented Aug 14, 2023 •

edited

Loading