Playwright executor fails to run the browsers in case of any Playwright version mismatch

[tkonieczny]

Describe the bug
Playwright executor fails to run the browsers and start the test because of missing browser binaries in case of any playwright version mismatch.
It is recommended to always pin your Docker image to a specific version if possible. If the Playwright version in your Docker image does not match the version in your project/tests, Playwright will be unable to locate browser executables.
Browser download is disabled (PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=true), but downloading it for every single execution is probably not an option.

To Reproduce
Steps to reproduce the behavior:

1. Create the test:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: playwright-test
  namespace: testkube
spec:
  type: playwright/test
  content:
    type: git-dir
    repository:
      type: git
      uri: https://github.com/kubeshop/testkube
      branch: main
      path: test/playwright/executor-tests/playwright-project

2. Run it
3. The test fails:

) [chromium] › smoke.spec.js:4:1 › has title ────────────────────────────────────────────────────

    browserType.launch: Executable doesn't exist at /ms-playwright/chromium-1048/chrome-linux/chrome
    ╔══════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just updated to 1.31.1. ║
    ║ Please update docker image as well.                                  ║
    ║ -  current: mcr.microsoft.com/playwright:v1.30.0-focal               ║
    ║ - required: mcr.microsoft.com/playwright:v1.31.1-focal               ║
    ║                                                                      ║
    ║ <3 Playwright Team                                                   ║
    ╚══════════════════════════════════════════════════════════════════════╝





  2) [firefox] › smoke.spec.js:4:1 › has title ─────────────────────────────────────────────────────

    browserType.launch: Executable doesn't exist at /ms-playwright/firefox-1378/firefox/firefox
    ╔══════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just updated to 1.31.1. ║
    ║ Please update docker image as well.                                  ║
    ║ -  current: mcr.microsoft.com/playwright:v1.30.0-focal               ║
    ║ - required: mcr.microsoft.com/playwright:v1.31.1-focal               ║
    ║                                                                      ║
    ║ <3 Playwright Team                                                   ║
    ╚══════════════════════════════════════════════════════════════════════╝




  2 failed
    [chromium] › smoke.spec.js:4:1 › has title ─────────────────────────────────────────────────────
    [firefox] › smoke.spec.js:4:1 › has title ──────────────────────────────────────────────────────

[dev-jonghoonpark]

It seems that all three must match.

1. The playwright version of the client
2. version of playwright execute in the executor
3. The docker version of the executor

for example

if run test using

1. client playwright version is "@playwright/test": "^1.30.0"

2. the executor execute
   npx playwright@1.30.0 test
   or
   pnpm dlx playwright@1.30.0 test(in pnpm)

3. 1.30.0 playwright docker version (current using version)
   mcr.microsoft.com/playwright:v1.30.0-focal

it should be work well
I have verified that this works without any problems. (in locally)

[dev-jonghoonpark]

To fix the executor version, modify this part as follows.
https://github.com/kubeshop/testkube-executor-playwright/blob/main/pkg/runner/playwright.go#L75

var runner string
var args []string

if r.dependency == "pnpm" {
    runner = "pnpm"
    args = []string{"dlx", "playwright@1.30.0", "test"}
} else {
    runner = "npx"
    args = []string{"playwright@1.30.0", "test"}
}

args = append(args, execution.Args...)

[dev-jonghoonpark]

How about making it possible to select the docker tag to use when setting up an executor while creating a test in the dashboard?

[vLia]

We could do something similar like for cypress, where we build different images for the different versions: https://github.com/kubeshop/testkube-executor-cypress/tree/main/build/agent.
But I am not sure how feasible this is given that Playwright has monthly release cycles.

[Bernix01]

any update on this issue?

[vsukhin]

hey, @Bernix01 yes, we started refactoing our executor system in order to suppot complex versioning and dependencies, work still is in progress, but it should allow to solve such issues