Tests that rely on a Cookie header fail after first tests run

[adgad]

We have a test suite that sets a Cookie header with a different value for each test. With the default concurrency of 2, what happens is:

1. First two tests run, with the correct cookie, all is well
2. Subsequent tests run, they have a different set of cookies, not the ones that are set in the header.

What I think is happening is that some JS from the first two pages is setting cookies that's overriding what the header tries to use.

We've worked around this by bumping up the concurrency so all the tests are run at once - but wondering if there's a better way to make the tests more idempotent (e.g. clearing cookies after each test?)

/cc @remybach

[JakeChampion]

I've been speaking to @rowanmanning about this issue and we think it might be solvable by using separate browser contexts for each "url". Puppeteer has browser context functionality already built-in, it would take some development work to integrate this into Pa11y.

[rowanmanning]

Yep what Jake said. If anybody is interested in trying to resolve this then I'd appreciate the help :) it would probably require changes to https://github.com/pa11y/pa11y rather than CI though

[josebolos]

Hi @adgad

Pa11y-ci v3 uses an incognito context by default in order to avoid these kinds of issues. Concurrency has also been set to 1 by default so race conditions like this are avoided.

Feel free to reopen the issue if you're still experiencing this with a recent version of pa11y-ci.