-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove integration tests dependence on DNS failures #132
Comments
I took a look at options, trying to find cases that did not rely on external dependencies, but failed to execute. They obviously also had to work given the URL sanitization that pa11y and pa11y-ci perform (e.g. puppeteer passes Pass no url If either an empty string or Pass a URL with an invalid domain If a URL with an invalid domain is used, e.g. Pass a URL with an invalid file name This is based on the example already used for the CLI paths test, but passing an invalid file URL, e.g. |
This is great! Thank you so much for picking this one up, @aarongoldenthal. It's very very much appreciated. |
The replacement of Travis with GitHub Actions for CI is failing, apparently due to github doing something funky with the DNS that is causing tests to fail unexpectedly.
A number of integration tests rely of puppeteer failing to resolve hostnames like
notahost
in order to verify that errors during the test running are propagated properly. This reliance on the hostname failing to resolve correctly goes against best practices, as there are a number of reasons why this may not be the case, and the domain may resolve successfully.We have an example in pa11y, where a similar test was already disabled:
https://github.com/pa11y/pa11y/blob/d602acaad1db8ed90f55ab59cf73ebb3e782c768/test/integration/cli/exit-codes.test.js#L33
We need to remove all the testing that relies on external factors like the DNS failing to resolve a hostname, so we can continue to make changes to pa11y-ci. This is currently blocking any development or migration work.
The text was updated successfully, but these errors were encountered: