Skip to content

Commit

Permalink
Only run aria-live tests in Chrome, bump timeout
Browse files Browse the repository at this point in the history
We were seeing the new integration tests fail as they didn't complete
within the set time limit of 2000ms.

This would happen when they were run through Saucelabs as done on CI
or when you have the Saucelabs set up to run locally. When Saucelabs
is not available, the tests are run through Karma/headless Chrome,
which seems to reliably pass.

Solutions 2 and 3 to keep some test coverage, while also trying to ensure that
our PRs don't start failing due to Saucelabs timer issues. With this in
mind, we should keep an eye on the tests on CI going forward.

Setting timeout on
[waitUntil](https://webdriver.io/docs/api/browser/waitUntil.html)
is optional.

However removing it throws
```
Promise was rejected for the following reason: timeout
```

-Chrome on Saucelabs would intermittently fail within 2000ms; Firefox
and IE would always fail
- On 3000ms Firefox and IE11 would still fail.
- On 4000ms IE11 was flaky.
- IE10 needed at least 6000ms to pass, whilst IE9 wouldn't pass even
at 10000ms.
- Additionally, Firefox would fail intermittently regardless of the timeout
limit set.

As Chrome seemed most reliable to run the tests with, we only run these tests
through Saucelabs only in Chrome on Windows 10.

However even then we would occassionally see the tests in Chrome fail within
the 2000ms limit.

We increased the limit to 10000ms, at which point no issues were observed.

Saucelabs runs tests in tests/integration. karma/headless chrome run tests in
tests/functional.

Tests on the original PR would have been run through karma/headless Chrome as
Saucelabs wasn't available. They seem too flaky to run through Saucelabs because
of timer issues so we investigated moving them to /tests/functional.

However we found that trying to create the interactive component for testing
was really difficult - this is probably why these tests were originally in
placed in tests/integration.

Investigate adding a file that has the same capabilities as tests in
/integration (Webdriver?) but is run through Karma/headless Chrome
rather than Saucelabs. This could be helpful in the future for creating
integration tests that don't require Saucelabs.

Saucelabs is flaky when running timer based tests. Remove these tests
and leave a code comment to say if the aria-live region code is changed,
it should be tested manually.

Co-authored-by: Nick Colley <nick.colley@digital.cabinet-office.gov.uk>
  • Loading branch information
hannalaakso and NickColley committed Sep 19, 2019
1 parent 39c6176 commit f0adce0
Showing 1 changed file with 29 additions and 25 deletions.
54 changes: 29 additions & 25 deletions test/integration/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ const isIE = browserName === 'internet explorer'
// const isIE9 = isIE && version === '9'
// const isIE10 = isIE && version === '10'
// const isIE11 = isIE && version === '11.103'
const liveRegionWaitTimeMillis = 2000
const liveRegionWaitTimeMillis = 10000

const basicExample = () => {
describe('basic example', function () {
Expand Down Expand Up @@ -39,31 +39,35 @@ const basicExample = () => {
expect(browser.isVisible(menu)).to.equal(true)
})

it('should announce status changes using two alternately updated aria live regions', () => {
const flip = browser.$('div#ariaLiveA')
const flop = browser.$('div#ariaLiveB')
// These tests are flakey when run through Saucelabs so we only run them
// in Chrome
if (isChrome) {
it('should announce status changes using two alternately updated aria live regions', () => {
const flip = browser.$('div#ariaLiveA')
const flop = browser.$('div#ariaLiveB')

browser.click(input)
browser.setValue(input, 'a')
expect(flip.getText()).to.equal('')
expect(flop.getText()).to.equal('')
browser.waitUntil(() => { return flip.getText() !== '' },
liveRegionWaitTimeMillis,
'expected the first aria live region to be populated within ' + liveRegionWaitTimeMillis + ' milliseconds'
)
browser.addValue(input, 's')
browser.waitUntil(() => { return (flip.getText() === '' && flop.getText() !== '') },
liveRegionWaitTimeMillis,
'expected the first aria live region to be cleared, and the second to be populated within ' +
liveRegionWaitTimeMillis + ' milliseconds'
)
browser.addValue(input, 'h')
browser.waitUntil(() => { return (flip.getText() !== '' && flop.getText() === '') },
liveRegionWaitTimeMillis,
'expected the first aria live region to be populated, and the second to be cleared within ' +
liveRegionWaitTimeMillis + ' milliseconds'
)
})
browser.click(input)
browser.setValue(input, 'a')
expect(flip.getText()).to.equal('')
expect(flop.getText()).to.equal('')
browser.waitUntil(() => { return flip.getText() !== '' },
liveRegionWaitTimeMillis,
'expected the first aria live region to be populated within ' + liveRegionWaitTimeMillis + ' milliseconds'
)
browser.addValue(input, 's')
browser.waitUntil(() => { return (flip.getText() === '' && flop.getText() !== '') },
liveRegionWaitTimeMillis,
'expected the first aria live region to be cleared, and the second to be populated within ' +
liveRegionWaitTimeMillis + ' milliseconds'
)
browser.addValue(input, 'h')
browser.waitUntil(() => { return (flip.getText() !== '' && flop.getText() === '') },
liveRegionWaitTimeMillis,
'expected the first aria live region to be populated, and the second to be cleared within ' +
liveRegionWaitTimeMillis + ' milliseconds'
)
})
}

describe('keyboard use', () => {
it('should allow typing', () => {
Expand Down

0 comments on commit f0adce0

Please sign in to comment.