Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Only run aria-live tests in Chrome, bump timeout
We were seeing the new integration tests fail as they didn't complete within the set time limit of 2000ms. This would happen when they were run through Saucelabs as done on CI or when you have the Saucelabs set up to run locally. When Saucelabs is not available, the tests are run through Karma/headless Chrome, which seems to reliably pass. Solutions 2 and 3 to keep some test coverage, while also trying to ensure that our PRs don't start failing due to Saucelabs timer issues. With this in mind, we should keep an eye on the tests on CI going forward. Setting timeout on [waitUntil](https://webdriver.io/docs/api/browser/waitUntil.html) is optional. However removing it throws ``` Promise was rejected for the following reason: timeout ``` -Chrome on Saucelabs would intermittently fail within 2000ms; Firefox and IE would always fail - On 3000ms Firefox and IE11 would still fail. - On 4000ms IE11 was flaky. - IE10 needed at least 6000ms to pass, whilst IE9 wouldn't pass even at 10000ms. - Additionally, Firefox would fail intermittently regardless of the timeout limit set. As Chrome seemed most reliable to run the tests with, we only run these tests through Saucelabs only in Chrome on Windows 10. However even then we would occassionally see the tests in Chrome fail within the 2000ms limit. We increased the limit to 10000ms, at which point no issues were observed. Saucelabs runs tests in tests/integration. karma/headless chrome run tests in tests/functional. Tests on the original PR would have been run through karma/headless Chrome as Saucelabs wasn't available. They seem too flaky to run through Saucelabs because of timer issues so we investigated moving them to /tests/functional. However we found that trying to create the interactive component for testing was really difficult - this is probably why these tests were originally in placed in tests/integration. Investigate adding a file that has the same capabilities as tests in /integration (Webdriver?) but is run through Karma/headless Chrome rather than Saucelabs. This could be helpful in the future for creating integration tests that don't require Saucelabs. Saucelabs is flaky when running timer based tests. Remove these tests and leave a code comment to say if the aria-live region code is changed, it should be tested manually. Co-authored-by: Nick Colley <nick.colley@digital.cabinet-office.gov.uk>
- Loading branch information