Visual regression is flaky for pages with images

[anandkedari]

This issue is specific to pages which has screens and when we want to perform visual regression on page. This becomes significantly flaky for mobile native apps. Consider a situation where UI(web or mobile) fetches/downloads content through backend api or server. The content is downloaded at client and then the content is rendered at client i.e., on browser or native app. Let us also consider that the loader normally comes when api is triggered and goes away when the response is received.

The problem occurs when the image is downloaded and all api response is received from server, but while rendering on UI, there is a slight delay, probably 0.5 milisecond to 2 seconds. Since the DOM is fully loaded and there is no loader anymore, UI is available for interaction, thereby visual regression kicks-in and starts performing checkScreen flow. But the test becomes flaky when the image on screen is not yet fully loaded(downloaded but partially rendered). As there is a difference in baseline(fully loaded image) and actual runtime screenshot (partially rendered), tests becomes flaky. This is usually higher for native apps as few devices might take higher time to render the image thats already downloaded.

Which approach wouldn't work?

• document.ready and img.complete is a very good alternative to tackle this, but this becomes difficult to handle on mobile native apps as these methods don't work there
• waitFor visibility or interaction will not halt the visual comparison because logically DOM is loaded and elements are available for interactions. Even if we use wait on element, it will always return true as element is interactable
• There is no way to interpret if image is fully rendered on page in native apps
• Hard pause can solve the problem, but it has to be placed on each page where there are images in page. Eventually it will increase the execution time

Expected behavior
Can there be a away to add a capability that will halt visual regression till the page and images is fully loaded

Steps to reproduce :

• In chrome, open network inspector
• Switch throttling speed to Slow 3G and go to https://stackoverflow.com/questions/60782997/slow-rendering-of-image-in-react
• You will notice image is rendered slowly on screen

Partially rendered image (Halt visual regression)

Fully rendered image (Perform visual regression)

[erwinheitzman]

I see two solutions, either stub the API call so the image never loads, or wait for the request(s). This is not an issue with WebdriverIO in my opinion.

[christian-bromann]

I do agree with @erwinheitzman. WebdriverIO doesn't have any checks in place that ensures that all images on the page are loaded. The devtools protocol would allow it to check for network idleness but this wouldn't work in a native context.

@anandkedari do you have any suggestions how WebdriverIO can help with this issue?

[anandkedari]

Thanks for your reply. I do agree with you both but this problem is quite often and I guess most of the people will face if they use wdio's visual regression. I did try doing some research to figure out how to fix it, but I couldn't figure out. Hence raised an issue as visual regression is a feature of WDIO and not android/ios sdks. The problem occurs when it does visual compare during automation, not sure why you ppl think its not related to Webdriverio. Even if it is so, can you guide where this issue actually belongs?

@erwinheitzman The purpose of having visual regression is to have visual compare of run-time image vs baseline and eventually give us feedbacks for any deviations. So if image never loads will it give feedback for the whole page, not right? And whats the point of doing visual regression if part of the page is not rendered? Wait for Request - As i said earlier there is a loader that usually appears when there is an api call. But this problem is specific from the time api response is received and image is slowly plotted on screen. Hence there is no clear distinguisher that helps to determine if page is fully ready or something is getting build on screen.

@christian-bromann yeah, with devtools & img.complete we can somehow achieve to wait for the images to be loaded before visual regression is performed on web (Hard way). But for mobile, its nearly difficult and not feasible at the moment. My concern was do we first accept if this is an issue? If so, then let's brainstorm and think through if there are any ways we can fix for native apps? If not, then I might need a guidance on where should I route these questions and issue to?

[christian-bromann]

@anandkedari while I don't believe this is a specific problem with WebdriverIO and more a general automation problem, I am happy to keep the issue around in the hope someone who has solved the problem or has ideas can chime in.

[praveendvd]

First thing, You have browser capabilities that allow to "not render any images" you can do that if that's the final outcome

 { pref :{"profile.managed_default_content_settings.images":2}}

Else , use browser.wait ( custom condition)
Where condition is the rendered size of the element , wait till the size is as expected .

Third , write a wrapper that retries image comparison say for 2 second

[anandkedari]

@praveendvd Yes this problem is still solvable for web with img.complete dev tool option. But there is no workaround for mobile native apps, there is no way to figure out if the image is fully rendered on screen.

[justusromijn]

@anandkedari did you try that setting that flag where it won't actually renders the images (suggested by @praveendvd)? Visual regression testing is, IMO, mainly to check if pages are still rendered as expected with regard to structure and spacing. The images itself would be out of scope, so skipping those for regression comparison seems a explainable approach?

[anandkedari]

What you are saying is true. But how do we validate if the structure itself varies when there is an image and without an image. Or may be the spacing. Isn't it good to perform a visual regression on a prod like screen just how the way it will eventually end up when went live? How can a good visual test be by omitting images. If we omit, it will be just a text only page, for which we can even put text assertions and get away with it. Having a visual comparison on a fully rendered page with image is an valid expectation. We can add a capability to omit it for the ones who don't need it.

Also I have a thought, how do we handle scenario where there are many images on a page. Some of it is static across website (like company logo), some of it is contextual (like mastercard image on specific payment page) and some of it is the actual content we want to display (like product listing with images). If we plan to omit images from visual, wouldn't all the images in the page disappear? So the screen will look mostly empty, incomplete or partially rendered, isn't it? If the link/path to logo on one of the sub-pages is broken, then will it fail the test case? I guess not. So, do we really want visual regression to be performed on half-baked/partially rendered page?

[justusromijn]

Hmm yeah I think it is indeed most straightforward to make snapshots on fully loaded pages. However, if browsers already do not provide any dependable hooks for knowing if an image is not just loaded but also rendered, I don't think making such an expectation from a framework like Webdriver valid: if Webdriver would implement something, it would be prone to issues, quirky and might break easily with new browser updates. I think such an request should be made to the W3C specifications.

For your case, I'm afraid doing small waits until the images are rendered are you best bet. You can wait for the network requests to be loaded, then give the browser an N amount of time to let them render. And tweak the value based on the flakiness of your test to get it just right.

[erwinheitzman]

Since I have been running visual tests in my current project I can share the following recommendations:

1. add a delay when creating baselines. this can be done by adding a flag to your command, followed with something along these lines:

async function takeScreenshot() {
  if (process.argv.includes('--autoSaveBaseline')) {
    await browser.pause(1000);
  }
  // take screenshot and return differences
}

This can be run like so:

npx wdio --autoSaveBaseline

2. retry until the screenshot matches the baseline or until it times out. this can be done like so (I'm leaving out some details but I paint the bigger picture):

let differences;
await browser.addCommand('verifyVisualDifferences', async () => {
  await browser.waitUntil(async () => {
     differences = await takeScreenshot();
     return differences === 0; // should be pixel perfect
  }).catch((error) => {
    // will throw a useful error message when timed out
    throw Error(`A difference of ${differences} was found.`);
  });
});

3. mock/stub the requests

The suggestions I just gave are the most important but here's some other things that you can also do.

4. You can point your browser to a profile in which you can add all your assets so that they are already cached in your browser. This is similar to mocking the requests, I have not done this but I did want to share that this should be a possibility although personally I would just mock what I need.

5. ignore/remove things that do not matter for your usecase. if the images do not matter for your usecase you can remove them or ignore them with different techniques. For example by querying said images and removing them from the DOM before making the screenshot.

I hope this helps anyone that is struggling with this.

@christian-bromann I feel that with this we can close the issue, what do you think?

[christian-bromann]

I agree, I don't see anything much actionable. Thanks for the recommendations.