Skip to content

Latest commit

 

History

History
341 lines (199 loc) · 18.9 KB

quickstart-run-end-to-end-tests.md

File metadata and controls

341 lines (199 loc) · 18.9 KB
Raw
title description ms.topic ms.date ms.custom
Quickstart: Run Playwright tests at scale
This quickstart shows how to run your Playwright tests with highly parallel cloud browsers using Microsoft Playwright Testing Preview. The cloud-hosted browsers support multiple operating systems and all modern browsers.
quickstart
10/04/2023
playwright-testing-preview, build-2024

Quickstart: Run end-to-end tests at scale with Microsoft Playwright Testing Preview

In this quickstart, you learn how to run your Playwright tests with highly parallel cloud browsers using Microsoft Playwright Testing Preview. Use cloud infrastructure to validate your application across multiple browsers, devices, and operating systems.

After you complete this quickstart, you have a Microsoft Playwright Testing workspace to run your Playwright tests at scale.

Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

Create a workspace

To get started with running your Playwright tests at scale on cloud browsers, you first create a Microsoft Playwright Testing workspace in the Playwright portal.

[!INCLUDE Create workspace in Playwright portal]

When the workspace creation finishes, you're redirected to the setup guide.

Create an access token for service authentication

Microsoft Playwright Testing uses access tokens to authorize users to run Playwright tests with the service. You first generate a service access token in the Playwright portal, and then store the value in an environment variable.

To generate the access token, perform the following steps:

  1. In the workspace setup guide, in Create an access token, select Generate token.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-generate-token.png" alt-text="Screenshot that shows setup guide in the Playwright Testing portal, highlighting the 'Generate token' button." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-generate-token.png":::

  2. Copy the access token for the workspace.

    You need the access token value for configuring your environment in a later step.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-copy-access-token.png" alt-text="Screenshot that shows how to copy the generated access token in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-copy-access-token.png":::

Configure the service region endpoint

In the service configuration, you have to provide the region-specific service endpoint. The endpoint depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL, perform the following steps:

  1. In Add region endpoint in your setup, copy the region endpoint for your workspace.

    The endpoint URL matches the Azure region that you selected when creating the workspace.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-region-endpoint.png" alt-text="Screenshot that shows how to copy the workspace region endpoint in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-region-endpoint.png":::

Set up your environment

To set up your environment, you have to configure the PLAYWRIGHT_SERVICE_ACCESS_TOKEN and PLAYWRIGHT_SERVICE_URL environment variables with the values you obtained in the previous steps.

We recommend that you use the dotenv module to manage your environment. With dotenv, you define your environment variables in the .env file.

  1. Add the dotenv module to your project:

    npm i --save-dev dotenv

  2. Create a .env file alongside the playwright.config.ts file in your Playwright project:

    PLAYWRIGHT_SERVICE_ACCESS_TOKEN={MY-ACCESS-TOKEN}
PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}

    Make sure to replace the {MY-ACCESS-TOKEN} and {MY-REGION-ENDPOINT} text placeholders with the values you copied earlier.

Caution

Make sure that you don't add the .env file to your source code repository to avoid leaking your access token value.

Add a service configuration file

To run your Playwright tests in your Microsoft Playwright Testing workspace, you need to add a service configuration file alongside your Playwright configuration file. The service configuration file references the environment variables to get the workspace endpoint and your access token.

To add the service configuration to your project:

  1. Create a new file playwright.service.config.ts alongside the playwright.config.ts file.

    Optionally, use the playwright.service.config.ts file in the sample repository.

  2. Add the following content to it:

    :::code language="typescript" source="~/playwright-testing-service/samples/get-started/playwright.service.config.ts":::

  3. Save the file.

Run your tests at scale with Microsoft Playwright Testing

You've now prepared the configuration for running your Playwright tests in the cloud with Microsoft Playwright Testing. You can either use the Playwright CLI to run your tests, or use the Playwright Test Visual Studio Code extension.

Run a single test at scale

With Microsoft Playwright Testing, you get charged based on the number of total test minutes. If you're a first-time user or getting started with a free trial, you might start with running a single test at scale instead of your full test suite to avoid exhausting your free test minutes.

After you validate that the test runs successfully, you can gradually increase the test load by running more tests with the service.

Perform the following steps to run a single Playwright test with Microsoft Playwright Testing:

Playwright CLI

To use the Playwright CLI to run your tests with Microsoft Playwright Testing, pass the service configuration file as a command-line parameter.

  1. Open a terminal window.

  2. Enter the following command to run your Playwright test on remote browsers in your workspace:

    Replace the {name-of-file.spec.ts} text placeholder with the name of your test specification file.

    npx playwright test {name-of-file.spec.ts} --config=playwright.service.config.ts

    After the test completes, you can view the test status in the terminal.

    Running 1 test using 1 worker
    1 passed (2.2s)

To open last HTML report run:

npx playwright show-report

Visual Studio Code

To run a single Playwright test in Visual Studio Code with Microsoft Playwright Testing, select the service configuration file in the Test Explorer view. Then select and run the test from the list of tests.

  1. Install the Playwright Test Visual Studio Code extension.

  2. Open the Test Explorer view in the activity bar.

    The test explorer automatically detects your Playwright tests and the service configuration in your project.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-test-explorer.png" alt-text="Screenshot that shows the Test Explorer view in Visual Studio Code, which lists the Playwright tests." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-test-explorer.png":::

  3. Select Select Default Profile, and then select your default projects from the service configuration file.

    Notice that the service run profiles are coming from the playwright.service.config.ts file you added previously.

    By setting a default profile, you can automatically run your tests with the service, or run multiple Playwright projects simultaneously.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-choose-run-profile.png" alt-text="Screenshot that shows the menu to choose a run profile for your tests, highlighting the projects from the service configuration file." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-choose-run-profile.png":::

  4. From the list of tests, select the Run test button next to a test to run it.

    The test runs on the projects you selected in the default profile. If you selected one or more projects from the service configuration, the test runs on remote browsers in your workspace.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-test.png" alt-text="Screenshot that shows how to run a single test in Visual Studio Code." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-test.png":::

    [!TIP] You can still debug your test code when you run your tests on remote browsers by using the Debug test button.

  5. You can view the test results directly in Visual Studio Code.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-test-results.png" alt-text="Screenshot that shows the Playwright test results in Visual Studio Code." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-test-results.png":::

You can now run multiple tests with the service, or run your entire test suite on remote browsers.

Caution

Depending on the size of your test suite, you might incur additional charges for the test minutes beyond your allotted free test minutes.

Run a full test suite at scale

Now that you've validated that you can run a single test with Microsoft Playwright Testing, you can run a full Playwright test suite at scale.

Perform the following steps to run a full Playwright test suite with Microsoft Playwright Testing:

Playwright CLI

When you run multiple Playwright tests or a full test suite with Microsoft Playwright Testing, you can optionally specify the number of parallel workers as a command-line parameter.

  1. Open a terminal window.

  2. Enter the following command to run your Playwright test suite on remote browsers in your workspace:

    npx playwright test --config=playwright.service.config.ts --workers=20

    Depending on the size of your test suite, this command runs your tests on up to 20 parallel workers.

    After the test completes, you can view the test status in the terminal.

    Running 6 tests using 6 workers
    6 passed (18.2s)

To open last HTML report run:

    npx playwright show-report

Visual Studio Code

To run your Playwright test suite in Visual Studio Code with Microsoft Playwright Testing:

  1. Open the Test Explorer view in the activity bar.

  2. Select the Run tests button to run all tests with Microsoft Playwright Testing.

    When you run all tests, the default profile is used. In the previous step, you configured the default profile to use projects from the service configuration.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-all-tests.png" alt-text="Screenshot that shows how to run all tests in Visual Studio Code." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-all-tests.png":::

    [!TIP] You can still debug your test code when you run your tests on remote browsers by using the Debug tests button.

  3. Alternately, you can select a specific service configuration from the list to only run the tests for a specific browser configuration.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-all-tests-select-project.png" alt-text="Screenshot that shows how to run all tests for a specific browser configuration, by selecting the project in Visual Studio Code." lightbox="./media/quickstart-run-end-to-end-tests/visual-studio-code-run-all-tests-select-project.png":::

  4. You can view all test results in the Test results tab.

View test runs in the Playwright portal

Go to the Playwright portal to view the test run metadata and activity log for your workspace.

:::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-activity-log.png" alt-text="Screenshot that shows the activity log for a workspace in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-activity-log.png":::

The activity log lists for each test run the following details: the total test completion time, the number of parallel workers, and the number of test minutes.

View test results in the Playwright portal

Microsoft Playwright Testing now supports viewing test results in the Playwright Portal. This feature is only available as an invite only feature.

Important

The reporting feature of Microsoft Playwright Testing service is free of charge during the invite-only preview. However, existing functionality of any cloud-hosted browsers continues to bill per the Azure pricing plan.

Once you have access to the reporting tool, use the following steps to set up your tests.

  1. From the workspace home page, navigate to Settings.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-select-settings.png" alt-text="Screenshot that shows settings selection for a workspace in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-select-settings.png":::

  2. From Settings, select General and make sure reporting is Enabled.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-enable-reporting.png" alt-text="Screenshot that shows how to enable reporting for a workspace in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-enable-reporting.png":::

  3. Make sure the environment is set up correctly as mentioned in the section Set up your environment.

  4. Install reporting package

    Since the feature is currently not public, you need to perform a few extra steps to install the package. These steps won't be needed once the feature becomes public.

    1. Create a file with name .npmrc at the same location as your Playwright config file.

    2. Add the following content to the file and save.

      @microsoft:registry=https://npm.pkg.github.com

    3. Create a GitHub Personal Access Token by following these steps.

    You need to provide read:packages permissions to the token. This token is referred to as PAT_TOKEN_PACKAGE for the rest of this article.

    1. Run the following command in your terminal, at the location of your Playwright config file. Replace PAT_TOKEN_PACKAGE with the token generated in the previous step.

          npm set //npm.pkg.github.com/:_authToken PAT_TOKEN_PACKAGE

    2. Update package.json file with the package.

       "dependencies": {
            "@microsoft/mpt-reporter": "0.1.1-alpha-8839338250-1.0"
    }

    3. Run npm install to install the package.

  5. Update Playwright.config file

    Add Playwright Testing reporter to Playwright.config.ts in the same way you use other reporters.

    import { defineConfig } from '@playwright/test';

export default defineConfig({
    reporter: [
    ['list'],
    ['json', {  outputFile: 'test-results.json' }],
    ['@microsoft/mpt-reporter'] // Microsoft Playwright Testing reporter
    ],
});

    Make sure that the artifacts are enabled in the config for better troubleshooting.

    use: {
    // ...
    trace: 'on-first-retry',
    video:'retain-on-failure',
    screenshot:'only-on-failure',
    }

  6. Run Playwright tests

    You can run npx playwright test command and view the results and artifacts on Playwright Testing portal.

    :::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-test-run-page.png" alt-text="Screenshot that shows the test runs for a workspace in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-test-run-page.png":::

Tip

You can use Microsoft Playwright Testing service to publish test results to the portal independent of the cloud-hosted browsers feature.

Optimize parallel worker configuration

Once your tests are running smoothly with the service, experiment with varying the number of parallel workers to determine the optimal configuration that minimizes test completion time.

With Microsoft Playwright Testing, you can run with up to 50 parallel workers. Several factors influence the best configuration for your project, such as the CPU, memory, and network resources of your client machine, the target application's load-handling capacity, and the type of actions carried out in your tests.

You can specify the number of parallel workers on the Playwright CLI command-line, or configure the workers property in the Playwright service configuration file.

Learn more about how to determine the optimal configuration for optimizing test suite completion.

Next step

You've successfully created a Microsoft Playwright Testing workspace in the Playwright portal and run your Playwright tests on cloud browsers.

Advance to the next quickstart to set up continuous end-to-end testing by running your Playwright tests in your CI/CD workflow.

[!div class="nextstepaction"] Set up continuous end-to-end testing in CI/CD