Skip to content

chore(e2e): run Playwright visual tests in Docker (#DS-5311) #1845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lskramarov wants to merge 2 commits into
base: main
Choose a base branch
from
Open

chore(e2e): run Playwright visual tests in Docker (#DS-5311) #1845

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
53ccb2b
chore(e2e): run Playwright visual tests in Docker (#DS-5311)
lskramarov Aug 5, 2026
b49c949
fix: after review
lskramarov Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .dockerignore
View file
Open in desktop

@artembelik artembelik Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

удобный подход, когда ты игнорируешь все по умолчанию, и кладешь в образ только то, что действительно необходимо, пример:

# Exclude everything by default, then explicitly allow only what is needed.
**

# Yarn Berry binary and configuration
!.yarn/releases/
!.yarn/releases/**
!.yarnrc.yml
!package.json
!yarn.lock

# Application and test source code
!packages/components/

я к тому что, в данной реализации в докер образ попадают например packages/components-dev, packages/docs-examples и тд

сам файл можно положить рядом с Dockerfile в tools/e2e/

Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Build context for tools/e2e/Dockerfile, whose build context is the repository root.
#
# It lives here rather than beside the Dockerfile because BuildKit only reads
# `<context>/.dockerignore` or `<dockerfile>.dockerignore`, and this is the location that works on
# every toolchain. A plain `.dockerignore` next to the Dockerfile is read by nothing and fails
# silently, shipping the entire working tree.
#
# Deliberately a denylist. An allowlist omits files silently: forgetting `tools/builders/` breaks
# `yarn install`'s postinstall, and forgetting `packages/e2e/utils/` breaks deep inside `ng serve`
# with an error that points nowhere near the cause.
#
# NOTE: these patterns are not recursive by default. A bare `node_modules` would match only the
# repository root and miss `.opencode/node_modules` (56 MB, self-ignored so it never appears in
# `git status`) — hence the `**/` prefixes.
#
# Verify with:
# docker build --progress=plain --no-cache -f tools/e2e/Dockerfile \
# --build-arg PLAYWRIGHT_VERSION=<version> . 2>&1 | grep "transferring context"
# Expect roughly 57 MB. Substantially more means one of the patterns below stopped matching.

**/node_modules
**/dist
**/*.log

.git
.angular
.nx
.yarn/cache
.yarn/install-state.gz
.yarn/unplugged

# Playwright outputs. The container produces its own; the committed __screenshots__ baselines it
# compares against are deliberately not excluded.
blob-report
playwright-report
playwright-report-docs
test-results
.playwright-mcp

# Editor, agent and local tooling state.
.ai
.claude
.idea
.opencode
.vscode
coverage
tmp
11 changes: 11 additions & 0 deletions .gitattributes
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Deliberately narrow. `* text=auto` is tempting but would renormalize every tracked file in a
# single commit, so line endings are left alone except where they are load-bearing.

# The Docker build inputs are consumed by a Linux shell. A contributor with core.autocrlf=true
# would otherwise commit CRLF into the Dockerfile's `RUN` continuations and the yarn shim it
# writes.
tools/e2e/** text eol=lf

# Git already detects these as binary; declaring it means no future filter or `text=auto` change
# can start mangling the screenshot baselines, which are compared byte-for-byte at threshold: 0.
*.png binary
22 changes: 16 additions & 6 deletions .github/workflows/e2e-approve-snapshots.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,23 @@ jobs:
- uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3.0.1
with:
message: 🔄 [Updating](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}) snapshots.
- uses: ./.github/workflows/actions/setup-node
# Same as e2e.yml: no --with-deps, and a cap so a stalled download fails fast.
- run: yarn run e2e:setup
timeout-minutes: 20
# Regenerated in the same container that e2e.yml compares against. Doing it on the bare
# runner instead would mean the baselines are written by one renderer and checked by another,
# and this workflow would happily commit screenshots that fail the very next run.
#
# docker compose creates a missing bind-mount source as root; packages/components already
# exists from the checkout, so only the report directories need creating up front.
- run: mkdir -p playwright-report test-results
# npm rather than yarn: no setup-node here, so the repository's Yarn 4 release is not on
# PATH. The container does its own install. e2e:docker:update-snapshots additionally mounts
# packages/components, which is how the rewritten PNGs reach the working tree for the commit
# step below.
- id: update-snapshots
run: |
yarn run e2e:components --update-snapshots
run: npm run e2e:docker:update-snapshots
env:
# As in e2e.yml: the compose default is tuned for developer machines, the runner wants
# its own core count.
PLAYWRIGHT_WORKERS: 100%
- uses: stefanzweifel/git-auto-commit-action@4a55954c782fc1ea30b9056cd3e7a2b40ca8887d # v7.2.0
id: commit-and-push
with:
Expand Down
51 changes: 35 additions & 16 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,30 +11,49 @@ permissions:
pull-requests: write

jobs:
# Runs in the container built from tools/e2e/, not on the runner directly. The screenshots are
# compared with threshold: 0 against baselines that carry no {platform} suffix, so the thing that
# produces them has to be pinned; a bare runner is only pinned by whatever `ubuntu-latest` happens
# to mean this week. The same image is what `yarn run e2e:docker` gives a developer locally, which
# is the point — a failure here is reproducible off CI.
#
# Regeneration must go through the same image: see .github/workflows/e2e-approve-snapshots.yml.
tests:
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
contents: read # for actions/checkout to read the repository
pull-requests: write # for thollander/actions-comment-pull-request to comment on PRs
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: ./.github/workflows/actions/setup-node
# e2e:setup deliberately omits --with-deps: that shells out to sudo apt-get to install the
# browser's system libraries, and the ubuntu-latest image already ships Chrome, Chromium,
# Edge and Firefox, so they are present before the job starts. Please do not add it back
# without checking the runner image first.
# No setup-node and no browser install: node is already on the runner, tools/e2e/run.js only
# reads package.json, and the browsers come baked into the image. That removes the ~174 MB
# `playwright install` download this job used to nurse through a timeout — but it is not a
# net time saving and should not be read as one. GitHub-hosted runners keep no Docker layer
# cache between runs, so every run rebuilds the image from scratch: the Node install, the
# font layer and `yarn install --immutable` all re-run, and the base image is pulled again
# (872 MB compressed, 3.6 GB on disk). Measured cold on a 64-core machine, the build alone is
# ~157s, and a 4-vCPU runner will be slower. Against the old job this is roughly a wash.
#
# Capping the step means a stalled download fails fast instead of consuming the whole job
# budget and taking the test run down with it — which is what happened in run 30790866073,
# where the job hit the wall during setup and produced no report at all. The cap only bites
# while it stays below the job timeout.
# What is bought with that is reproducibility, not speed. If the wall clock ever does become
# the problem, the answer is a prebuilt image pulled from GHCR by tag — not
# `cache-to: type=gha`, which would push well over a gigabyte of layers into the same 10 GB
# Actions cache that every other job's yarn cache is competing for.
#
# 8 minutes was too tight: the ~174 MB browser download alone has been overrunning it, so
# the cap was killing otherwise-healthy runs. Keep this comfortably above the observed
# duration — it is a guard against a hang, not a performance budget.
- run: yarn run e2e:setup
timeout-minutes: 20
# docker compose creates a missing bind-mount source as root. Creating them up front keeps
# the workspace owned by the runner user, which matters on self-hosted runners where the
# next job has to clean it up.
- run: mkdir -p playwright-report test-results
# npm rather than yarn: without setup-node the repository's Yarn 4 release is never put on
# PATH, and the runner's own `yarn` is v1, which cannot read this manifest. Nothing is
# installed here either — the container does its own yarn install.
- id: run-e2e-tests
run: |
yarn run e2e:components
run: npm run e2e:docker
env:
# Back to the runner's own setting. The compose file caps workers for developer machines,
# where a container sees far more cores than one dev server can be driven from; a 4-vCPU
# runner has the opposite problem and wants all of them.
PLAYWRIGHT_WORKERS: 100%
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: ${{ always() }}
id: upload-report
Expand Down
9 changes: 9 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,17 @@ yarn run e2e:setup # Install Playwright browsers (run once)
yarn run e2e:components # Run all component E2E tests
yarn run e2e:docs # Run the docs site smoke suite (needs `yarn run docs:build` first)
npx playwright test <TEST_PATH_PATTERN> # Run specific E2E tests (e.g., npx playwright test packages/components/button/e2e.playwright-spec.ts)

# Screenshots differ across operating systems — always use Docker for anything visual:
yarn run e2e:docker # Run E2E tests in Docker (matches CI)
yarn run e2e:docker:update-snapshots # Run E2E tests in Docker and update the baselines
```

The committed baselines under `__screenshots__` are compared with `threshold: 0` and have no

@artembelik artembelik Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

это не нужно в AGENTS.md, дубль из packages/e2e/README.md

@lskramarov lskramarov Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Одно для человека другое для агента, так же ? и тут скорее из ридми нужно выбрасывать и для агента больше писать..

platform suffix, so a native run outside Linux fails on font rasterization alone. `e2e:components`
is still useful for the assertion-based specs; use `e2e:docker` whenever screenshots are involved,
and never regenerate a baseline any other way.

### Linting

```bash
Expand Down
54 changes: 54 additions & 0 deletions docs/guides/06-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,57 @@ exist first:
yarn run docs:build
yarn run e2e:docs
```

### Visual regression tests and Docker

The screenshot baselines committed under `__screenshots__` are compared with `threshold: 0` and carry
no platform suffix, so they are tied to one operating system and one browser build. Running the suite
natively on Windows or macOS compares your machine's font rasterization against Linux bytes and fails
regardless of whether anything actually changed.

Run anything visual in Docker instead. The image is built from the Playwright release matching
`@playwright/test` in `package.json`, which is what CI runs too:

```bash
yarn run e2e:docker
```

To accept intentional visual changes, regenerate the baselines the same way and commit the result:

```bash
yarn run e2e:docker:update-snapshots
```

Arguments are passed through, replacing the container's command — for example, to run one component:

```bash
yarn run e2e:docker yarn playwright test packages/components/button
```

The container always runs with `CI=true`, so that Playwright behaves the way it does on the runner.
Two consequences matter when debugging inside it: `test.only` is rejected outright rather than
honoured (`forbidOnly`), and a failing test is retried twice before being reported. Narrow a run with
a path and `-g` instead of `test.only`:

```bash
yarn run e2e:docker yarn playwright test packages/components/select -g "single select"
```

Requires Docker with Compose v2. On Windows, Docker Engine installed inside WSL puts no `docker.exe`
on the Windows PATH, so run these commands from inside the WSL distribution rather than from
PowerShell.

### Worker count

A container reports every core on the host, and Playwright sizes its worker pool from that. Since all
workers drive one shared Angular dev server, the useful ceiling comes from that server rather than
from the core count — on a 32-core machine `workers: '100%'` means 64 browsers, and the suite
collapses into timeouts that look like failures but are not. The compose file therefore caps workers
at 8. Override it when a machine wants something different:

```bash
PLAYWRIGHT_WORKERS=16 yarn run e2e:docker
```

Baselines can also be regenerated without a local Docker install by commenting `/approve-snapshots`
on a pull request.
2 changes: 2 additions & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,8 @@
"dev:e2e": "ng serve dev-e2e",
"e2e:setup": "playwright install chromium --with-deps && playwright install webkit --with-deps",
"e2e:components": "playwright test packages/components",
"e2e:docker": "node tools/e2e/run.js",
"e2e:docker:update-snapshots": "node tools/e2e/run.js yarn run e2e:components --update-snapshots",
"e2e:docs": "playwright test --config playwright.docs.config.ts",
"serve:docs": "node tools/serve-docs.mjs",
"-----API-----": "--------------------------------------------------------------------------------------------",
Expand Down
21 changes: 21 additions & 0 deletions packages/e2e/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,24 @@ yarn run e2e:components
# Run a specific E2E test file
yarn playwright test packages/components/button/e2e.playwright-spec.ts
```

## Screenshots

The baselines under each component's `__screenshots__` directory are compared with `threshold: 0` and
have no platform suffix, so they belong to one operating system and one browser build. The commands
above only compare them meaningfully on Linux; anywhere else they fail on font rasterization alone.

Run anything visual in Docker, which uses the Playwright image matching `@playwright/test` and is what
CI runs as well:

```bash
# Run the suite in Docker
yarn run e2e:docker

# Accept intentional visual changes and rewrite the baselines
yarn run e2e:docker:update-snapshots
```

Requires Docker with Compose v2. On Windows, Docker Engine installed inside WSL puts no `docker.exe` on
the Windows PATH, so run these from inside the WSL distribution. Without a local Docker install,
comment `/approve-snapshots` on a pull request to regenerate the baselines in CI.
45 changes: 44 additions & 1 deletion playwright.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,49 @@ const viewport: ViewportSize = {
const baseURL = process.env.BASE_URL || 'http://localhost:4200';
const webServerCommand = process.env.WEB_SERVER_COMMAND || 'yarn run dev:e2e --configuration=production';

/**
* Every worker drives its own browser against one shared Angular dev server, so the useful ceiling
* comes from that server rather than from the core count. '100%' suits a 4-vCPU CI runner, but not
* Docker: a container reports every core on the host (Playwright reads `os.cpus()`, which no cgroup
* or cpuset limit affects), so on a 32-core machine it means 64 browsers and the suite collapses
* into timeouts. tools/e2e's compose file caps it via PLAYWRIGHT_WORKERS and CI sets it back.
*
* Playwright only accepts a string when it is a percentage, so anything else has to become a number.
* With the variable unset this behaves exactly as it did before.
*
* The value is validated rather than passed through, because Playwright's own guard only rejects
* `workers <= 0` — and `NaN <= 0` is false. A typo like `PLAYWRIGHT_WORKERS=amx` would therefore
* reach the dispatcher's `for (i = 0; i < workers; i++)` loop, spawn zero workers, run zero tests,
* write no report, and still exit 0: a green suite that tested nothing.
*/
const resolveWorkers = () => {
const override = process.env.PLAYWRIGHT_WORKERS?.trim();

if (!override) {
return isCI ? '100%' : undefined;
}

if (override.endsWith('%')) {
const percentage = Number(override.slice(0, -1));

if (!Number.isFinite(percentage) || percentage <= 0) {
throw new Error(`PLAYWRIGHT_WORKERS must be a positive percentage, got ${JSON.stringify(override)}.`);
}

return override;
}

const workers = Number(override);

if (!Number.isInteger(workers) || workers <= 0) {
throw new Error(
`PLAYWRIGHT_WORKERS must be a positive integer or a percentage, got ${JSON.stringify(override)}.`
);
}

return workers;
};

/** @see https://playwright.dev/docs/test-configuration */
export default defineConfig({
testDir: __dirname,
Expand All @@ -17,7 +60,7 @@ export default defineConfig({
fullyParallel: true,
forbidOnly: isCI,
retries: isCI ? 2 : 0,

@artembelik artembelik Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

в идеале конечно отключить retries в 0, чтобы исключить нестабильные тесты на этапе разработки, но лучше это отдельно сделать

workers: isCI ? '100%' : undefined,
workers: resolveWorkers(),
reporter: [
['list', { printSteps: true }],
['html', { open: 'never' }]
Expand Down
Loading