[BUG] 1px height difference in screenshot dimensions when comparing locally generated to ci-generated screenshots

[thraizz]

Context:

• Playwright Version: 1.29.2
• Operating System: macOS Ventura 13.1
• Node.js version: v18.13 (but tested 16.19 as well)
• Browser: Chromium
• Extra:
  • Does happen for both M1 and Intel chips
  • We use the mcr.microsoft.com/playwright:v1.29.2-focal docker image in CI, the difference is the same when comparing local vs in the image.

envinfo output

## System:
 - OS: macOS 13.1
 - Memory: 1.71 GB / 32.00 GB
## Binaries:
 - Node: 18.13.0 - ~/.nvm/versions/node/v18.13.0/bin/node
 - npm: 8.19.3 - ~/.nvm/versions/node/v18.13.0/bin/npm
## Languages:
 - Bash: 3.2.57 - /bin/bash

Code Snippet
More importantly than the test: This snippet has to be run locally and then inside the docker image / some sort of CI for the bug to happen.

import { expect, test } from "@playwright/test";

test("Onboarding Step 1", async ({ page }) => {
  await page.goto("/");
  await page.getByText("Login").click();
  await expect(page.getByTestId("onboarding-heading")).toContainText(
    "Onboarding Step 1"
  );
  await expect(page).toHaveScreenshot(
    "onboarding-step-1.png",
    { fullPage: true }
  );
});

Describe the bug

Screenshot have flaky dimensions. Our locally generated, full-page, screenshots sometimes differ from those created inside docker by 1px, sometimes 2px height. If running the same test multiple times in the same environment (either docker or local), the screenshots match all the time. All tests are run with headless chromium. I've also experimented with any antialiasing and font rendering setup. This did not help either.

Result, for example:

    Error: Screenshot comparison failed:

      Expected an image 1280px by 1740px, received 1280px by 1739px. 

    Call log:
      - expect.toHaveScreenshot with timeout 5000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - Expected an image 1280px by 1740px, received 1280px by 1739px. 
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - Expected an image 1280px by 1740px, received 1280px by 1739px. 

[thraizz]

This is probably a duplicate of #18827

[pavelfeldman]

Looks like your full page lays out to different fractions of the number between 1739 and 1740. This can be due to fonts or any other differences. Could you confirm that both local and remote scenarios are using the same docker image? Are you seeing this variability within the process architecture (within M1, depending on the test run)? Does it never happen when running on the same machine?

[thraizz]

I suspect some font rendering to two different heights as well. To answer your questions:

1. Yes. Both CI as well as local docker runs use the mcr.microsoft.com/playwright:v1.29.2-focal image, c4b51b122b0f - both generate the "smaller" image, where either one or two pixel is missing. The local, non-docker, environment produces "longer" images, using the 1.29.2 playwright package.
2. I see this across architectures. The non-arm linux pipeline of our CI environment produces the same 1px shorter image screenshot as my local docker environment. The non-docker environments across our team produce the 1px longer screenshots, both on intel and M1 chips.
3. No, it's not flaky when running in either environment repeatedly. If I regenerate the screenshots and run the test 10 times, it succeeds ten times. It just the docker vs. non-docker which fails.

Interesting is that some views produce differences and some do not.
I will have a look at the height of the elements in failing screens and report back, maybe I can just eliminate elements with heights "between valid values".

[pavelfeldman]

• Having different screenshots between local and docker environments is expected. You can only compare screenshots that are taken in the same exact environment, OS, fonts, etc.
• We've also seen a 1px fluctuation between Intel and M1, looks like there is a rounding inconsistency that manifests itself as such.

@aslushnikov, is this a dupe, do we still have that M1 1px issue open?

[thraizz]

Okay, I hadn't considered that the OS made a difference even when running headless! We always run our tests in the official docker image now, so that screenshot comparisons don't fail. If it's expected, it's okay 😸 Thanks for your fast responses and the help!

[aslushnikov]

We've also seen a 1px fluctuation between Intel and M1, looks like there is a rounding inconsistency that manifests itself as such.

@pavelfeldman we observed color blending differences between M1 and intel, but screenshot sizes were not affected by architecture differences so far. I'll experiment with this separately.

@thraizz oh yes, running OS does make difference even for headless! Looks like this is resolved, so I'll close this now. Feel free to file new issues if you think we can be helpful!