[BUG] Issues with VRT tests

[edumserrano]

System info

• Playwright Version: [v1.34.3]
• Operating System: [Windows 10, Ubuntu 22.04]
• Browser: [Chromium]

Source code

• I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

edumserrano/playwright-bug-vrt-diff

Bug report

The issue I have is that some scenarios with screenshot comparison don't seem to work as expected, or at least as I'd expect. Perhaps some of these issues aren't qualified to be bugs.

I've found issues with unexpected differences in screenshot tests:

• running in headless vs not headless mode.
• running on Chromium Windows vs Chromium Ubuntu.
• running outside debug vs in debug mode.

Please see the details below.

Issues trying to get consistent VRT

Note

All these scenarios were executed from a Windows OS as the base OS.

Scenario 1: Hardware acceleration and headless mode

• Go to the root of the repo.
• Run the tests via npm run test.
• Run the app via npm run start.
• Compare the screenshot generated via npm run test at tests/example.spec.ts-snapshots/basic-test-1-chromium-win32.png with how the app running via npm run start looks like in regards to the blur effect.

Screenshot from npm run test

Screenshot from npm run start

Note

My PC has a higher resolution than 1920x1080, I've used the browser tools to try and get close for the manual screenshot I took. Anyways, what's important in this screenshot is looking at how the blur looks correct and how different it is from when running the test via npm run test.

When running the tests with npm run test it seems the chromium browser running in headless mode does not have hardware acceleration enabled, therefore the image produced is not as expected.

You can compare the image produced from the test with how the app looks like when running via npm run start. I believe this is related with hardware acceleration because when running the app via npm run start if you enable/disable hardware accelearation in the browser settings, the blur applied looks different.

Another way to view this is related with hardware acceleration is that if you add:

launchOptions: {
  args: ["--disable-gpu"],
},

to the chromium project settings in the playwright.config.ts file, then the blur of screenshot from debug will look equal to when running without debug.

Note

The way the blur is intended to look like is how it looks when hardware acceleration is enabled

The blur effect is being created with:

background-color: transparent;
-webkit-backdrop-filter: blur(50px);
backdrop-filter: blur(50px);

Scenario 2: Chromium screenshot in Windows vs Chromium screenshot in Ubuntu

• Go to the root of the repo.
• Uncomment the line // ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file.
• Run the tests via npm run test and generate the screenshot in Windows.
• Duplicate the screenshot basic-test-1-chromium-win32.png at /tests/example.spec.ts-snapshots and rename it to basic-test-1-chromium-linux.png.
• Run the tests in Ubuntu via docker by executing docker run -it --rm --ipc=host --workdir=/app -v ${PWD}:/app mcr.microsoft.com/playwright:v1.34.3-jammy npx playwright test.
• Open the html report via npx playwright show-report tests/test-results/reporters/html and look at the differences in the screenshots.

The screenshot generated from Ubuntu will have some slight differences regarding the font (looks bolder in windows?). Furthermore, the page scrollbar on Windows is a bit wider than the one in Ubuntu.

Diff

Screenshot from Windows

Screenshot from Ubuntu

Note

The font I'm using on the app is Lato Regular 400 from google fonts.

See the below at index.html:

<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Lato&display=swap" rel="stylesheet">

And CSS at styles.less:

@font-family-lato: 'Lato', sans-serif;

body {
  font-family: @font-family-lato;
}

Scenario 3: Differences when running the test in/out of debug mode

This one might not be a bug but it still feels like a bit of an issue.

• Install the playwright extension for VSCode.
• Go to the root of the repo.
• Uncomment the line // ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file.
• Generate the test screenshot by running the tests with npm run test.
• Go to VS Code and debug the test using the playwright extension for VS Code.
• Open the html report via npx playwright show-report tests/test-results/reporters/html and look at the differences in the screenshots.

The debug will fail because there are differences in the screenshots:

• the screenshot from debug mode will have will have the appropriate blur. Perhaps it runs with hardware acceleration enabled and therefore shows as expected (equal to what is visible when running the app with npm run start).
• the screenshot from debug mode will have slight differences in the font. Or perhaps this is incorrect, perhaps the issue is not with the font but with the fact that the scrollbar width is slightly different and the effect that the difference in blur also has on the fonts.
• the screenshot from debug mode seems to have "weaker" coloured red line (the red line below the label Where do you study or work?). This again might just be related with the blur difference.
• the screenshot from debug mode has a scrollbar with smaller width.

Diff

Screenshot from running via "npm run test"

Screenshot from running in debug via VS Code playwright extension

Note

To make the comparison easier in debug vs not debug I've added ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file. This makes sure scrollbar is always visible whether running the test in debug mode or not.

However I'd say that this was also unexpected to me even if it's default behavior. Couldn't find any mention to the fact that the scrollbar is disabled in headless mode when searching the playwright docs. Perhaps a chance for improving the docs?

Related

Perhaps related with the effort on #8161

[edumserrano]

Update for Scenario 2: Chromium screenshot in Windows vs Chromium screenshot in Ubuntu.

If I run the steps described in this scenario but simplify the demo code to focus only on the font by removing the CSS for the blur, the images and the footer section, then the diff clearly highlights that the font on Chromium Ubuntu renders differently from Chromium Windows. On Windows the font seems bolder.

Diff

Screenshot from Chromium Windows

Screenshot from Chromium Ubuntu

[aslushnikov]

Hey @edumserrano,

These are all indeed well-known issues:

• for the headed vs headless rendering issues, see [BUG] Chromium and webkit headless/head rendering differences #12683, especially [BUG] Chromium and webkit headless/head rendering differences #12683 (comment)
• for the ubuntu vs windows: font rendering & font stack resolution is handled by OS, not specced & fundamentally different, so the common agreement as of today is that there's no way to achieve consistent rendering across operating systems. The only way is to use a well-defined consistent execution environment, e.g. VM / containers
• debug vs no-debug mode: seems like a headless vs headed rendering in your case, as well.

Overall, I'll close this issue in favor of other VRT-related bugs we already have on the tracker.

[edumserrano]

Thank you for the reply @aslushnikov . I did see some of those, like the webkit headless/head issue for instance, but since:

1. it mentioned webkit and not chromium
2. there are lots of issues on the repo when searching for font related issues

I didn't know for sure if these things were duplicates or not. Apologies for creating more "noise".

At least I created a demo repo and left all these details here, perhaps it helps others 🤞

[edumserrano]

@aslushnikov Do your comments also apply to the issue regarding Scenario 1: Hardware acceleration and headless mode ?

Specifically with the fact that the headless mode seems to run with --disable-gpu or some equivalent which then makes the blur effect look different from when running in head mode. I'm asking to confirm this is somehow captured in some other VRT-related bug already on the tracker ?

[aslushnikov]

I didn't know for sure if these things were duplicates or not. Apologies for creating more "noise".

@edumserrano no worries for the noise! We're here to help navigate the project.

Do your comments also apply to the issue regarding Scenario 1: Hardware acceleration and headless mode ? Specifically with the fact that the headless mode seems to run with --disable-gpu or some equivalent which then makes the blur effect look different from when running in head mode.

I don't think I've looked into this one particularly, but this falls in the headed vs headless larger category. So far it looks like all the screenshots should be always taken in either headed or headless configuration; mixing these won't work.