Print failure reason for unstable tests

[sorin-davidoi]

What is your Test Scenario?

We have some unstable tests during continous integration. We run them in quarantine mode so they (mostly) pass, but we would like to identify at which step they fail.

What are you suggesting?

Introduce a new flag (or make it the default behavior) to print the errors that happen when tests run in quarantine mode but they pass (e.g. only one run fails).

[miherlosev]

Hi @sorin-davidoi

As I understand, the suggested option should help you to find the cause of the problem in the tests and fix these problems gradually. We will think about it and possible provide a solution in the future.

[sorin-davidoi]

Yes, printing the following during CI doesn't really help us understand why it is unstable (should we just increase a timeout for a network request? does something not exist in the DOM? etc).

[API] t.takeScreenshot()
    ✓ Should take a screenshot (unstable)

[elfia]

I am testing Iframe and for some tests I often get "Could not connect to iframe" or something missing in the DOM. Fixing flaky tests on CI without knowing the error is not easy especially for iframes where i can't see loading status (loading circle on the browser) or the current url.

[Dmitry-Ostashev]

@elfia If you have any unstable behavior with the switchToIframe action, you may consider implementing an iframe-waiting ClientFunction. Something like this:

    const waitForIframeLoaded = ClientFunction((iframeSelector) => {
        return new Promise((resolve, reject) => {
            let i = 1;
            const intervalId = window.setInterval(() => {
                var iframeElement = document.querySelector(iframeSelector);
                if (iframeElement && iframeElement.contentWindow && iframeElement.contentWindow.location.href !== 'about:blank' && iframeElement.contentDocument) {                    
                    window.clearInterval(intervalId);
                    resolve();                    
                }

                if(i > 10) {                 
                    window.clearInterval(intervalId);
                    reject(new Error("Iframe content loading timeout."));
                }
                i++;
            }, 1000);
        });
    });

If you need further assistance with switching to iframe, feel free to create a new ticket and describe the situation in detail there.

[grant-mccarriagher]

Definitely would be interested in this feature. Without error output, it's hard to diagnose unstable tests.