[🐛 Bug]: afterTest hook does not fetch retry attempts

[user7699]

WebdriverIO Version

7.13.2

Node.js Version

v14.16.1

Mode

WDIO Testrunner

Which capabilities are you using?

{
            maxInstances: 1,
            browserName: 'chrome',
            'goog:chromeOptions': {
                useAutomationExtension: false,
                args: ['--ignore-certificate-errors', '--window-size=1000,1080', '--headless'], 
                binary: puppeteer.executablePath(),
            },
        },

What happened?

I should be able to fetch the amount of retry attemps in the afterTest hook. When a test is retried, the output is always { attempts: 0, limit: 0 }.

I'm using the following setup to retry a failing test:

specFileRetries: 2,
specFileRetriesDelay: 2,
specFileRetriesDeferred: false,

What is your expected behavior?

In the afterTest hook, "retries" should log the amount of retry attempts.

How to reproduce the bug.

In your config file, add:

specFileRetries: 2,
specFileRetriesDelay: 2,
specFileRetriesDeferred: false,

and the afterTest hook in the same config file

afterTest: function (test, context, { error, result, duration, passed, retries }) {
    console.log('-----------');
    console.log(retries);
    console.log('-----------');
},

Run a spec file that always fail, e.g

describe('A failing suite', () => {
    it('A failing test', () => {
        expect(1).toEqual(0);
    });
});

Relevant log output

[0-0] RETRYING after 2s in chrome 
[0-0] { attempts: 0, limit: 0 }
[0-0] { attempts: 0, limit: 0 }

Despite the spec-file being retried, the console still logs { attempts: 0, limit: 0 }

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[christian-bromann]

@user7699 thanks for filling the issue.

The object you are accessing describes spec related retries, e.g. defined like this in Mocha:

it('should rerun a test at least 3 times', function () {
    // ...
}, 3)

You are retrying whole spec files. I added a patch to make this differentiation a bit clearer. Thanks!

[user7699]

@christian-bromann thank you for clearing that out. Is there any way for me the count the attempts when using the strategy shown above, i.e.

specFileRetries: 2,
specFileRetriesDelay: 2,
specFileRetriesDeferred: false,

I stumbled upon this ticket but the example is not working for me: #4722

"[...] we can use browser.config.specFileRetryAttempts instead

example

afterTest () {
browser.config.specFileRetryAttempts // works!
this.specFileRetryAttempts // not available
}, "

The command "browser.config.specFileRetryAttempts" doesn't return a usable value

[christian-bromann]

@user7699 you can store this in a file or somewhere else. I would not advice though to make test runs depending on some retry counts.

[user7699]

@christian-bromann I have a visual testing suite that runs through browserstack which is a bit flaky. Appium have a tendency to crash and/or not being able to load a specific webside when a spec file runs for the first time. When that happen - I would like to have the possibility to only save images for a specific test on the second retry attempt. A test that fail due to one of the reasons previously mentioned will (almost) never fail for the same reason again. Therefore, retrying spec files is essential for me.

But perhaps I can come up with a workaround for this problem. Instead of trying to count retries - I could instead use 'error' from the afterTest hook:

ERROR: JestAssertionError: expect(received).toBeLessThanOrEqual(expected)

That's the error that I'm expecting when images should be saved from the test. That would give me the possibility to ignore timeout errors.