Support Cucumber reporters

[vitalets]

Cucumber has several built-in reporters (also called formatters). It would be useful to output results of Playwright test runner to these reporters (especially html-reporter or messages reporter).
There are different approaches:

1. Write custom Playwright reporter that will create Cucumber report
2. Use Playwright json report and convert it into Cucumber messages, and then convert to html

Ideally, final usage should allow to use any Cucumber report:

import { defineConfig, devices } from '@playwright/test';
import { cucumberReporter } from 'playwright-bdd';

export default defineConfig({
  // ...
  reporter: cucumberReporter('json:cucumber-report.json'),
});

Known problem - reporting Scenario Outline.
In Cucumber it is possible to have the same test title with different data. In Playwright each test must have unique title (it was also discussed in #3 (comment)). Example of Cucumber html report for Scenario Outline:

I would appreciate more ideas on that from your reporting usage.

[evix1101]

Not elegant, and my needs are currently minimal. I'm supporting tests against multiple environments at the project level via extending PlaywrightTestProject

import { defineConfig, devices, PlaywrightTestProject } from '@playwright/test';

export interface ExtendedPlaywrightTestProject extends PlaywrightTestProject {
  testEnv?: string;
}

const config = defineConfig({ 
  testDir: './.features-gen',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: process.env.CI ? [ ['html'], ['junit', { outputFile: 'test-results/results.xml' }] ] : [ ['html'],['list']],
  use: {
    trace: 'retain-on-failure',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'prod-chromium',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: 'https://www.google.com/',
      },
      testEnv: 'prod',
    } as ExtendedPlaywrightTestProject, 
    {
      name: 'dev-chromium',
      use: {
        ...devices['Desktop Chrome'],
        baseURL: 'https://dev.google.com/',
      },
      testEnv: 'dev',
    } as ExtendedPlaywrightTestProject,
  ],
});

export default config;

My thought was there might be a slick way to make this more dynamic to support the scenario outlines.

[Jag46]

import { defineConfig, devices } from '@playwright/test';
import { cucumberReporter } from 'playwright-bdd';

export default defineConfig({
// ...
reporter: cucumberReporter('json:cucumber-report.json'),
});

Hi @vitalets , when can we expect cucumberReporter function? It would be great if get this up running soon..Thank you..!

[vitalets]

@Jag46 can't give concrete date but will increase the priority for that.
Btw are you interested in particular json reporter or html?

[Jag46]

Thanks @vitalets , we are looking for a cucumber html report. That would great addition if you can help us with this blocker. Thanks

[harikishanl]

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

[vitalets]

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

Ok, got it. Could you clarify what parts of cucumber html reporter are the most important for you? E.g. formatting Scenario Outlines or something else? As there are a lot of events in both playwright and cucumber test run flows.

[harikishanl]

We are looking for a cucumber style html report as well, the current report is poorly formatted in playwright style and is not really making use of the BDD style reporting. If we can bring this enhancement soon, it will help our team use the html report without much hassle.

Ok, got it. Could you clarify what parts of cucumber html reporter are the most important for you? E.g. formatting Scenario Outlines or something else? As there are a lot of events in both playwright and cucumber test run flows.

Hi @vitalets - It would be helpful if we can cover: 'Feature, Scenario Outline, Scenario, Given, When, Then & Examples' in the report

[vitalets]

Cucumber messages can be validated with https://github.com/cucumber/compatibility-kit

[dlitmano]

@vitalets please inform me if i can help you. I love this project and want to support it!

[vitalets]

Thank you @dlitmano ! Answered in #82

[magarwal19]

Looking forward for this feature!!

[vitalets]

Looking forward for this feature!!

It's already 80% complete. Planning to include it in the nearest release.

[magarwal19]

Hi @vitalets
do we have any planned ETA for this?
My team is working on integrating test management tool and it needs the results to be pushed using cucumber jsons

[vitalets]

Hi @vitalets do we have any planned ETA for this? My team is working on integrating test management tool and it needs the results to be pushed using cucumber jsons

Hi @magarwal19 , I would say it's within 1-2 weeks. Cucumber json will be supported.

[magarwal19]

This is great news!!
thank you :)

[magarwal19]

Hi @vitalets
can you please let us know if we are planning to deploy this soon?

[vitalets]

Hi @magarwal19
Base part is ready, I need some time to write documentation.
Would it be helpful for you, if I push it in a separate branch and you will pull playwright-bdd directly from github and check Cucumber reporters?

[magarwal19]

Hi @magarwal19 Base part is ready, I need some time to write documentation. Would it be helpful for you, if I push it in a separate branch and you will pull playwright-bdd directly from github and check Cucumber reporters?

Hi @vitalets
i have integrated it by installing the version of playwrigh-bdd npm package..
if we can get a beta release, then i can surely use that and check the cucumber reports

[vitalets]

I've published playwright-bdd beta with Cucumber reporters support.
Please follow the instructions below to check it with your projects.
I will appreciate any feedback on that!
@magarwal19 and everybody :)

1.Install playwright-bdd@next:

npm i -D playwright-bdd@next

2.Use cucumberReporter helper in the playwright config:

import { defineConfig } from '@playwright/test';
import { defineBddConfig, cucumberReporter } from 'playwright-bdd';

const testDir = defineBddConfig({
  importTestFrom: 'steps/fixtures.ts',
  paths: ['features'],
  require: ['steps/*.ts'],
});

export default defineConfig({
  testDir,
  reporter: [
    cucumberReporter('html', { outputFile: 'cucumber-report/report.html' }),
    cucumberReporter('json', { outputFile: 'cucumber-report/report.json' }),
    cucumberReporter('junit', { outputFile: 'cucumber-report/report.xml' }),
    cucumberReporter('message', { outputFile: 'cucumber-report/report.ndjson' }),
  ],
});

Here is the diff example.

Example of html report:

Please note that minimal Playwright version is 1.34.

[magarwal19]

Hi @vitalets
thank you for sharing this!
two things i observed:

1. It is adding keyword as After, even when i do not have after. Also After is not BDD keyword so it throws error when i import in test management tool
2. There is huge data added as part of key "embeddings", not sure for what part do we need this..

HTML reports look good :)

[vitalets]

@magarwal19 could you provide more details on both two points?

[magarwal19]

Hi @vitalets
few updates here..

1. When we are creating junit reports, we need to add <testsuites> </testsuites>
   and then add the current report under this.. Due to this, whenever we use any utility to parse our junit xml it is failing
   also not sure if we can add encoding at first step like
   <?xml version="1.0" encoding="UTF-8"?>
   currently we are getting it like <?xml version="1.0"?>

[magarwal19]

@magarwal19 could you provide more details on both two points?

Hi @vitalets
i analysed this and this seems to be dependent on Playwright execution. Since i have Screenshot and videos to be on in my playwright.config.js file, there gets an entry added for After in results.json

If we can remove this, it will make this result json more light and easy to read

after.json

[vitalets]

If we can remove this, it will make this result json more light and easy to read

Yes, Playwright's automatic screenshots are attached as an after step, b/c we need some entity to attach them.
So if you don't need screenshots and videos in the report, why you can't just disable them in playwright.config.js?

[magarwal19]

Hi @vitalets few updates here..

1. When we are creating junit reports, we need to add <testsuites> </testsuites>
   and then add the current report under this.. Due to this, whenever we use any utility to parse our junit xml it is failing
   also not sure if we can add encoding at first step like
   <?xml version="1.0" encoding="UTF-8"?>
   currently we are getting it like <?xml version="1.0"?>

Hi @vitalets
have updated this comment.. some how it was not showing complete information earlier

[magarwal19]

If we can remove this, it will make this result json more light and easy to read

Yes, Playwright's automatic screenshots are attached as an after step, b/c we need some entity to attach them. So if you don't need screenshots and videos in the report, why you can't just disable them in playwright.config.js?

We would definitely need them in report, but in my case we did not need in json file but in html file..
will figure out using a script at my end

[vitalets]

@magarwal19

1. Will check on <testsuites> </testsuites> and <?xml version="1.0" encoding="UTF-8"?>. Btw, does your parser correctly handles Cucumber's original junit report? b/c I've ported this code from Cucumber without changes.
2. Got it. So the problem is large files due to embedded screenshots and videos, that are actually not needed? I will think about it. Also related to Support for having large attachments saved externally to the report cucumber/html-formatter#281

[magarwal19]

@magarwal19

1. Will check on <testsuites> </testsuites> and <?xml version="1.0" encoding="UTF-8"?>. Btw, does your parser correctly handles Cucumber's original junit report? b/c I've ported this code from Cucumber without changes.
2. Got it. So the problem is large files due to embedded screenshots and videos, that are actually not needed? I will think about it. Also related to Support for having large attachments saved externally to the report cucumber/html-formatter#281

@vitalets
I was using junit xml...in junit it generate data with testsuites... this might be a change that in junit it shows but in cucumber it might not...

thank you for thinking about point 2 :)

[vitalets]

@magarwal19 I've checked original Cucumber junit report, it returns the same structure as playwright-bdd:

• starts with <?xml version="1.0"?>, not <?xml version="1.0" encoding="UTF-8"?>
• does not contain <testsuites> </testsuites>

This is the full Cucumber report. Could you try to load it with your tool?

<?xml version="1.0"?>
<testsuite failures="4" skipped="0" name="cucumber-js" time="0.004758075" tests="4">
  <testcase classname="Playwright site" name="Check title" time="0.000913249">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="Check title 2" time="0.000599414">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="A Get started step is Playwright" time="0.002910996">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Playwright".........................................skipped]]></system-out>
  </testcase>
  <testcase classname="Playwright site" name="A Get started step is Play" time="0.00033441600000000003">
    <failure type="Error" message="foo"><![CDATA[Error: foo
    at World.<anonymous> (/Users/vitalets/projects/bdd-playground/steps/cucumber-raw/index.ts:15:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)]]></failure>
    <system-out><![CDATA[Given I open url "https://playwright.dev".................................failed
When I click link "Get started"..........................................skipped
Then I see in title "Play"...............................................skipped]]></system-out>
  </testcase>
</testsuite>

[magarwal19]

Hi @vitalets
sure will check this.. thank you for sharing this!
i think teams integration plugin expects testsuites to be there..

[NikkTod]

Hi @vitalets, the html report looks great.
One thing I wanted to ask if you can make it possible the traces of a failed test to be opened directly within the browser with a click. Currently I need to download the trace file and open it with command 'npx playwright show-trace ....'.

Within TeamCity when I am using the playwright html report I also have the option to just click it and it will open in a frame, so no need to download it.

this is sample from the html how it is opened in TeamCity iframe

Maybe passing the URL of the traces could help

[vitalets]

One thing I wanted to ask if you can make it possible the traces of a failed test to be opened directly within the browser with a click. Currently I need to download the trace file and open it with command 'npx playwright show-trace ....'.

@NikkTod
Created a separate issue for that, pls check out questions there: #103

[vitalets]

Finally, Cucumber reporters are released in playwright-bdd v6 🥳
It was not-easy journey, but result is worth the efforts.
There were two main challenges during development:

1. Attachments. In Cucumber attachments are bound to the steps, but in Playwright attachments are bound to the whole test. Although there is no 100% way to map attachments to steps in Playwright, for BDD steps it's possible, b/c they are always serial. I've changed the approach several times, because I wanted to support Playwright's merge-reports mode, that provides a bit different data to the Reporter API. Now you can fully benefit from Playwright's sharding: run BDD tests on the multiple machines and get combined Cucumber report.

2. Projects. Cucumber reports do not support concept of projects that exists in Playwright. In Cucumber you get separate reports for Chrome, Firefox, etc, but in Playwright it's a single run and output. Playwright-bdd adopts Cucumber messaging protocol to handle several projects: basically it prepends project name to the feature uri making it unique. You can observe the result in the html reporter:
   
   Having all browsers in a single run & report is also a good performance boost!

Thank you all for your help to achieve this goal. Feel free to share your feedback in separate issues ❤️