[BUG]: env PLAYWRIGHT_BROWSERS_PATH does not work with webkit

[GermanJablo]

System info

• Playwright Version: v1.32.1
• Operating System: macOS-latest (GA)
• Browser: WebKit
• Other info:

Source code

• I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

MRE: https://github.com/johny-ss/actions

or

Github action file

name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

# Cancel current job when pushing new commit into the PR
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  e2e:
    # timeout-minutes: 60
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false # keep running other jobs if one fails
      matrix:
        os: [macos-latest, windows-latest, ubuntu-latest]
        browser: [chromium, firefox]
        include:
          - os: macos-latest
            browser: webkit
    env:
      # Playwright uses different paths on different OS's. 
      # To make it work cross-platform, we need to set the path manually.
      # Alternatively, `PLAYWRIGHT_BROWSERS_PATH=0` install binaries in node_modules
      PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/ms-playwright
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      # installing dependencies with cache
      - name: Cache dependencies
        id: cache-deps
        uses: actions/cache@v3
        with:
          path: ./node_modules
          key: modules-${{ hashFiles('package-lock.json') }}
      - name: Install dependencies
        if: steps.cache-deps.outputs.cache-hit != 'true'
        run: npm ci --ignore-scripts

      # installing browsers with cache
      - name: Store Playwright's Version
        shell: bash #necessary for it to work on Windows, which uses powershell by default
        run: echo "PLAYWRIGHT_VERSION=$(node -p 'require("@playwright/test/package.json").version')" >> $GITHUB_OUTPUT
        id: playwright-version
      - name: Cache browsers
        id: cache-browsers
        uses: actions/cache@v3
        with:
          key: ${{ runner.os }}-${{ matrix.browser }}-playwright-${{ steps.playwright-version.outputs.PLAYWRIGHT_VERSION }}
          path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
      - name: Download browsers
        if: steps.cache-browsers.outputs.cache-hit != 'true'
        run: npx playwright install --with-deps ${{ matrix.browser }}

      # running tests
      - run: npx playwright test --project=${{ matrix.browser }}
      - name: Upload Artifacts
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: ${{ matrix.browser }}-${{ matrix.os }}-test-results
          path: test-results/
          retention-days: 7

Test file (self-contained): It can be any test, it doesn't matter

Steps

• Run the github actions of my repository. macOS-webkit throws error, and if you run again ubunto-webkit (now with cache), also throw error.

Expected

The test on webkit should succeed even when the browser and its dependencies are cached.

Actual

• In ubuntu-webkit, if the browser IS NOT cached, the execution of the tests gives this error:

• in macOS webkit, the execution of the tests gives the following error:

My guess is that the PLAYWRIGHT_BROWSERS_PATH environment variable doesn't work well with webkit, because if I remove it and don't cache at all, the tests work fine.

note: If you want to reproduce in another repo, you can change the --project flag to --browser to avoid plawright.config.ts.

[mxschmitt]

This is expected, be aware that there is a difference between browser binaries - which you are actually caching - and browser operating system dependencies - which you are not able to cache, since they get installed via apt.

To "fix" this, you need to call npx playwright install-deps all the time. Also be aware that the majority of the time (usually a minute), gets spent on installing the operating system dependencies rather on downloading the actual browsers (which usually takes 1-2 seconds). So we don't recommend caching as of today.

For macOS and Windows there are no operating system dependencies required to run the browsers, thats why its working there. Also for Chromium on ubuntu GitHub Actions works mostly, because they include most of the dependencies already - hence you only surface this issue on WebKit.

[GermanJablo]

browser operating system dependencies - which you are not able to cache, since they get installed via apt.

But this only applies to Linux, right?

To "fix" this, you need to call npx playwright install-deps all the time. Also be aware that the majority of the time (usually a minute), gets spent on installing the operating system dependencies rather on downloading the actual browsers (which usually takes 1-2 seconds). So we don't recommend caching as of today.

For macOS and Windows there are no operating system dependencies required to run the browsers, thats why its working there. Also for Chromium on ubuntu GitHub Actions works mostly, because they include most of the dependencies already - hence you only surface this issue on WebKit.

I have compared a run without cache vs one with cache and this is the installation time saved by caching:

• windows-chromium, 1m 52s.
• ubuntu-chromium, 38s
• ubuntu-firefox, 36s

I don't know how much of this difference is due to the browsers and how much to the dependencies, but what is evident is that one of these 2 things that you told me is not being fulfilled in my situation:

• Browsers take only 1-2 seconds to install.
• macOS, windows, ubuntu-chome [¿and ubuntu-firefox?] they do not require dependencies, (i.e. the only one that needs them is ubuntu-webkit.

---

On the other hand, I have moved webkit to a separate job by caching the following path:

path: |
             ~/Library/Caches/ms-playwright
             ~/.cache/ms-playwright
             %LOCALAPPDATA%\ms-playwright

And macOS-webkit doesn't give any problem anymore, so can't it be that in that combination there is a problem with PLAYWRIGHT_BROWSERS_PATH?

[mxschmitt]

Yes on Linux there need to be apt dependencies installed and on Windows the Server-Media-Foundation feature. Both you are not able to cache.

Sure it can take longer to download browsers, in my case of https://github.com/mxschmitt/myproject0404 browsers took 12 seconds to download and dependencies 54 seconds. See here.

If you don't install the dependencies for Google Chrome or Firefox, you might experience issues with e.g. graphics, fonts, video playback etc. Such configurations we are not able to support.

There is no issue with PLAYWRIGHT_BROWSERS_PATH from what I can tell. It looks all expected from my side, so I guess we can close this issue?

[GermanJablo]

Thanks, I think I understand it better now. I've decided to go with this:

      - name: Install Playwright browsers
        if: steps.cache-browsers.outputs.cache-hit != 'true'
        run: npx playwright install ${{ matrix.browser }}
      - name: Install Playwright dependencies
        # Linux and Windows cannot cache deps.
        # See: https://github.com/microsoft/playwright/issues/22146#issuecomment-1495821016
        # if: steps.cache-browsers.outputs.cache-hit != 'true' || runner.os != 'macOS'
        # BUT except for linux-webkit, they already include most of the necessary dependencies.
        # So unless you use things like graphics, fonts, video playback etc. and you run into a problem,
        # You can speed up your CI by using this less strict condition:
        if: steps.cache-browsers.outputs.cache-hit != 'true' || (runner.os == 'Linux' && matrix.browser == 'webkit')
        run: npx playwright install-deps ${{ matrix.browser }}

By the way, I don't know if it's possible but it would be great if Playwright could automatically detect the necessary dependencies based on tests.

---

There is no issue with PLAYWRIGHT_BROWSERS_PATH from what I can tell. It looks all expected from my side, so I guess we can close this issue?

Regarding this, have you tried the macOS-webkit combination using that environment variable?
Right now it's the only problem I'm encountering, and it doesn't seem to be related to the cache:

• no cache and using PLAYWRIGHT_BROWSERS_PATH (note that both browsers and dependencies are installed).
• using cache but not using PLAYWRIGHT_BROWSERS_PATH

[mxschmitt]

Looks like you extract your cached browsers to CWD, and then @playwright/test tries to run a test on ms-playwright/webkit-1811/WebInspectorUI.framework/Versions/A/Resources/Test/Test.js:26.

Since the original bug is fixed, I'm closing this bug for now.