Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"start": "cd examples && pnpm start",
"start:prod": "cd examples && pnpm start:prod",
"build": "tsc --build",
"build:docker": "docker build -t visual-regression .",
"build:docker": "cd visual-regression && pnpm build:docker",
"watch": "tsc --build --watch",
"test": "vitest",
"coverage": "vitest run --coverage",
Expand Down
178 changes: 126 additions & 52 deletions visual-regression/DOCKER.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,149 @@
# Visual Regression Test Docker Instructions

The Visual Regression Tests utilize headless browsers provided by the
[Playwright](https://playwright.dev/) project. Browsers tend to be very platform
specific in terms of exactly how pixels end up being rendered out. The
differences should be small, but they tend to be big enough such that a simple
image comparison algorithm cannot reliably tell what is a real regression and
what is just a platform difference.
The Visual Regression Tests utilize headless browsers provided by the [Playwright](https://playwright.dev/) project. Browsers are highly platform-specific, and even small pixel differences can cause significant issues for image comparison algorithms. These differences can prevent reliable detection of regressions.

In order to prevent this issue, the Visual Regression Tests are built to run
inside a Docker container which guarantees a consistent platform environment
for the given headless browser to run in.
To avoid these issues, Visual Regression Tests run inside a containerized environment. This guarantees a consistent platform for headless browsers, ensuring reproducible results.

Whenever new commits to a PR are pushed, a GitHub Action runs the Visual
Regression Tests in a Linux-based Docker container to determine if the PR should
be allowed to merge or not.
For PRs, a GitHub Action runs these tests in a Linux-based container. Locally, you must use `--ci` mode, which launches tests in a container to produce snapshots identical to the GitHub Action environment.

When you need to capture new snapshots or update existing ones, you must run the
Visual Regression Test Runner locally in `--ci` mode. This launches the tests in
a Docker container giving you exactly the same snapshot results as the tests
that run in the cloud on GitHub Actions.
This guide covers installing the required tools (`docker`, `colima`, or `podman`) and building the Visual Regression Test image.

Below are the instructions for both installing Docker and building the Visual
Regression Test Image that is used when you run the tests in `--ci`.
---

## Installing Docker
## Installing a Container Runtime

### Mac

If you have a license for [Docker Desktop](https://www.docker.com/products/docker-desktop/)
you can install that and all should be well. However, if you don't follow the
alternative steps below:
You can use Docker Desktop if you have a license. If you don’t, use Colima or Podman as alternatives.

#### Option 1: Docker Desktop (Requires License)

1. Download and install [Docker Desktop](https://www.docker.com/products/docker-desktop).
2. After installation, test Docker:
```bash
docker ps
```

#### Option 2: Colima (Open Source Docker Alternative)

1. Install the Docker CLI using [Homebrew](https://brew.sh/):
```bash
brew install docker
```
2. Install [Colima](https://colima.dev/):
```bash
brew install colima
```
3. Start Colima:
```bash
colima start
```
4. Test Docker with Colima:
```bash
docker ps
```
It should run without errors.

#### Option 3: Podman (Docker Alternative)

1. Install [Podman](https://podman.io/):
```bash
brew install podman
```
2. Start Podman:
```bash
podman machine init
podman machine start
```
3. Test Podman:
```bash
podman ps
```

### Linux

Docker is natively supported on Linux, but you can also use Podman for a rootless container environment.

#### Option 1: Docker

1. Follow the instructions for your Linux distribution to install Docker:
- [Ubuntu/Debian](https://docs.docker.com/engine/install/debian/)
- [Fedora/CentOS](https://docs.docker.com/engine/install/centos/)
2. After installation, test Docker:
```bash
docker ps
```

#### Option 2: Podman

1. Install [Podman](https://podman.io/) for your Linux distribution:
- [Podman Installation Guide](https://podman.io/getting-started/installation)
2. Test Podman:
```bash
podman ps
```

### Windows

Windows users can use Docker Desktop if they have a license or install Podman as an alternative.

#### Option 1: Docker Desktop (Requires License)

1. Download and install [Docker Desktop](https://www.docker.com/products/docker-desktop).
2. After installation, test Docker:
```powershell
docker ps
```

#### Option 2: Podman (Open Source Alternative)

1. Install [Podman](https://podman.io/) via the Windows installer:
- [Podman for Windows](https://podman.io/getting-started/installation)
2. Start the Podman machine:
```powershell
podman machine init
podman machine start
```
3. Test Podman:
```powershell
podman ps
```

---

1. Using [Homebrew](https://brew.sh/) install the Docker Client:

```
brew install docker
```
## Building the Test Image

2. Install [Colima](https://github.com/abiosoft/colima)
After installing a container runtime, you must build the Visual Regression Test image.

```
brew install colima
```
1. Run the build script:

3. Start Colima
```bash
pnpm build:docker
```

```
colima start
```
2. The script automatically detects your runtime (`docker` or `podman`) and builds the image. After a successful build, you should see the image:

4. Test Docker
```bash
docker images
```

```
docker ps
```
Or, if using Podman:

It should run without errors.
```bash
podman images
```

## Building the Test Image
Example output:

After installing Docker, you must build the Visual Regression Test Image before
running the test runner in `--ci` mode.
```
REPOSITORY TAG IMAGE ID CREATED SIZE
visual-regression latest 40476ed4acae 3 minutes ago 2.09GB
```

```
pnpm build:docker
```
---

If all goes well it should create an image called **visual-regression** locally.
## References

```
❯ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
visual-regression latest 40476ed4acae 3 minutes ago 2.09GB
```
- **Docker Desktop**: [docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop)
- **Colima**: [colima.dev](https://colima.dev/)
- **Podman**: [podman.io](https://podman.io/)
2 changes: 1 addition & 1 deletion visual-regression/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"build": "tsc",
"build:renderer": "cd .. && pnpm build",
"build:examples": "cd ../examples && pnpm build",
"build:docker": "cd .. && pnpm build:docker",
"build:docker": "tsc && node dist/src/build-docker.js",
"serve-examples": "cd ../examples && pnpm preview:automation",
"node-version": "node --version"
},
Expand Down
48 changes: 48 additions & 0 deletions visual-regression/src/build-docker.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env ts-node

import { $ } from 'execa';
import { argv } from 'process';
import path from 'path';
import { fileURLToPath } from 'url';

import { detectContainerRuntime } from './detectDockerRuntime.js';

/**
* Builds a container image using the detected container runtime.
* Changes the working directory to one level higher than the script's location.
* @param runtime - The container runtime (`podman` or `docker`).
* @param imageName - The name of the container image to build.
*/
async function buildContainer(
runtime: string,
imageName: string,
): Promise<void> {
// Change working directory to one level higher than the script's location
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const scriptDir = path.resolve(__dirname, '../../../');
process.chdir(scriptDir);

console.log(`Working directory changed to: ${scriptDir}`);
console.log(`Using ${runtime} to build the container image: ${imageName}`);
try {
await $({ stdio: 'inherit' })`${runtime} build -t ${imageName} .`;
} catch (error) {
console.error(`Failed to build the image with ${runtime}.`, error);
process.exit(1);
}
}

(async () => {
const imageName = argv[2] || 'visual-regression'; // Default image name
try {
const runtime = await detectContainerRuntime();
await buildContainer(runtime, imageName);
} catch (error) {
if (error instanceof Error) {
console.error(error.message);
} else {
console.error(error);
}
process.exit(1);
}
})();
22 changes: 22 additions & 0 deletions visual-regression/src/detectDockerRuntime.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { $ } from 'execa';

/**
* Detects the available container runtime (podman or docker).
* @returns {Promise<string>} The name of the container runtime (`podman` or `docker`).
* @throws {Error} If neither runtime is found.
*/
export async function detectContainerRuntime(): Promise<'docker' | 'podman'> {
try {
await $`podman -v`;
return 'podman';
} catch {
try {
await $`docker -v`;
return 'docker';
} catch {
throw new Error(
'Neither podman nor docker is installed. Please install one of them.',
);
}
}
}
7 changes: 6 additions & 1 deletion visual-regression/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ import {
export const certifiedSnapshotDir = 'certified-snapshots';
export const failedResultsDir = 'failed-results';

import { detectContainerRuntime } from './detectDockerRuntime.js';

const browsers = { chromium };
let snapshotsTested = 0;
let snapshotsPassed = 0;
Expand Down Expand Up @@ -135,6 +137,9 @@ const argv = yargs(hideBin(process.argv))
* @returns Exit code
*/
async function dockerCiMode(): Promise<number> {
// Detect container runtime
const runtime = await detectContainerRuntime();

// Relay the command line arguments to the docker container
const commandLineStr = [
argv.capture ? '--capture' : '',
Expand All @@ -149,7 +154,7 @@ async function dockerCiMode(): Promise<number> {
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const rootDir = path.resolve(__dirname, '..', '..', '..');

const childProc = $({ stdio: 'inherit' })`docker run --network host \
const childProc = $({ stdio: 'inherit' })`${runtime} run --network host \
-v ${rootDir}:/work/ \
-v /work/node_modules \
-v /work/.pnpm-store \
Expand Down