feat: Add basic test steps support for 'test.step' category only

[apatelia]

• Tested with passed, failed, and skipped tests

[Ma11hewThomas]

Hello, thanks for your contribution.

This change includes steps such as hooks and assertions whereas the CTRF steps property is intended specifically for BDD-style steps.

Could I suggest that we limit the inclusion to playwright steps category "test.step"?

[apatelia]

I have filtered steps with test.step category only.

[Ma11hewThomas]

Hello @apatelia,
Regarding steps within steps, I have tested the change with the below test, but the nested steps are not included in the report.

test('test with nested steps', async ({ page }) => {
  await test.step('Navigate and validate Playwright homepage', async () => {
    await test.step('Navigate to Playwright homepage', async () => {
      await page.goto('https://playwright.dev/');
    });

    await test.step('Verify page title contains Playwright', async () => {
      await expect(page).toHaveTitle(/Playwright/);
    });
  });

  await test.step('Interact with the Get Started section', async () => {
    await test.step('Click on Get Started link', async () => {
      await page.getByRole('link', { name: 'Get started' }).click();
    });

    await test.step('Verify Installation heading is visible', async () => {
      await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
    });
  });
});

  "steps": [
          {
            "name": "Navigate and validate Playwright homepage",
            "status": "passed"
          },
          {
            "name": "Interact with the Get Started section",
            "status": "passed"
          }
        ],

[Ma11hewThomas]

Please update the README.md with the new supported property.

[apatelia]

Hello @Ma11hewThomas,
First of all, thanks for guiding with the implementation and pointing out the issues. Appreciated!

Based on your earlier comment and CTRF schema doc, I (wrongly) assumed that the steps property is exclusive for BDD tests and hence processed nested test.step for test.category = 'hook' only, which covered steps of Background section of a BDD feature file.

I have removed this category condition now while looking for child steps, so it iterates over the child steps of all types but includes steps having test.step category only.

I have also updated README.md as requested.

My tests were:

A BDD scenario

Feature: Products

    Background:
        Given the User is on login page
        When the User tries to login with "standard_user" as username and "secret_sauce" as password
        Then the User should be on Products page

    Scenario: Test that the user is able to remove a product from the cart
        When the user adds "Sauce Labs Bike Light" to the cart
        Then the user should be able to remove "Sauce Labs Bike Light" from the cart, using the `Remove` button
        Then the cart item badge must not be displayed

which generated

"steps": [
          {
            "name": "Given the User is on login page",
            "status": "passed"
          },
          {
            "name": "When the User tries to login with \"standard_user\" as username and \"secret_sauce\" as password",
            "status": "passed"
          },
          {
            "name": "Then the User should be on Products page",
            "status": "passed"
          },
          {
            "name": "When the user adds \"Sauce Labs Bike Light\" to the cart",
            "status": "passed"
          },
          {
            "name": "Then the user should be able to remove \"Sauce Labs Bike Light\" from the cart, using the `Remove` button",
            "status": "passed"
          },
          {
            "name": "Then the cart item badge must not be displayed",
            "status": "passed"
          }
        ]

and a regular playwright test with nested steps (same as given by you above)

test('test with nested steps', async ({ page }) => {
    await test.step('Navigate and validate Playwright homepage', async () => {
        await test.step('Navigate to Playwright homepage', async () => {
            await page.goto('https://playwright.dev/');
        });

        await test.step('Verify page title contains Playwright', async () => {
            await expect(page).toHaveTitle(/Playwright/);
        });
    });

    await test.step('Interact with the Get Started section', async () => {
        await test.step('Click on Get Started link', async () => {
            await page.getByRole('link', { name: 'Get started' }).click();
        });

        await test.step('Verify Installation heading is visible', async () => {
            await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
        });
    });
});

which generated

"steps": [
          {
            "name": "Navigate and validate Playwright homepage",
            "status": "passed"
          },
          {
            "name": "Navigate to Playwright homepage",
            "status": "passed"
          },
          {
            "name": "Verify page title contains Playwright",
            "status": "passed"
          },
          {
            "name": "Interact with the Get Started section",
            "status": "passed"
          },
          {
            "name": "Click on Get Started link",
            "status": "passed"
          },
          {
            "name": "Verify Installation heading is visible",
            "status": "passed"
          }
        ]

[apatelia]

@Ma11hewThomas, one question.

CTRF schema supports only flattened steps while BDD tests and Playwright tests both, support nested steps through Background keyword and test.step() function, respectively. If one wants to generate a HTML report from CTRF json report, that person won't be able to produce/display Test Steps accurately in HTML due to flat structure of the steps.

For example, it won't be possible to generate HTML like below.

[Ma11hewThomas]

Thank you for your contribution @apatelia! Appreciated! This has been published in v0.0.16

Regarding your question, this is a valid point and your expertise with BDD is valuable. Please can you raise this as a discussion for us to consider a change to the schema for steps.

Thanks