Switch Accessibility tests to @axe-core/puppeteer

[colinrotherham]

This PR closes:

• Improve our use of automated accessibility tools #1971

All components + examples are now tested in axe-core@4.7.0 via Puppeteer

The tests run locally via npm test or separately with:

npx jest --selectProjects "Accessibility tests"

Colour contrast checks are built in ✅

Known issues

This switch has flagged a few known issues which are now ignored + documented, including some tests with positive tabindex and "landmark-inside-landmark" issues due to preview HTML

• All page content must be contained by landmarks
  Axe issue: "All page content must be contained by landmarks" #1604
  https://dequeuniversity.com/rules/axe/4.7/region

• Accordion "aria-labelledby attribute cannot be used on a div with no valid role attribute"
  Accordion - Elements must only use allowed ARIA attributes #2472
  https://dequeuniversity.com/rules/axe/4.7/aria-allowed-attr

• Conditional reveal "ARIA attribute is not allowed: aria-expanded="false""
  Radios that conditionally reveal content use aria-expanded which is not a valid aria-attribute for those roles #979
  https://dequeuniversity.com/rules/axe/4.7/aria-allowed-attr

• Invalid ARIA attribute value: aria-controls="example-id" for missing IDs
  https://dequeuniversity.com/rules/axe/4.7/aria-valid-attr-value

• Colour contrast for disabled form controls
  https://dequeuniversity.com/rules/axe/4.7/color-contrast

Test reports

I've kept the jest-axe reporter but generated the report using @axe-core/puppeteer instead

GitHub Actions will log failures as below:

"Elements must meet minimum color contrast ratio thresholds (color-contrast)"

Fix any of the following:
  Element has insufficient color contrast of 2.03 (foreground color: #aaa9a9, background color: #eeefef, font size: 12.0pt (16px), font weight: bold). Expected contrast ratio of 4.5:1

You can find more information on this issue here:
https://dequeuniversity.com/rules/axe/4.7/color-contrast?application=axe-puppeteer

Test workflow

Test progress appears on the GitHub Actions "Checks" tab (see example Accessibility tests)

[colinrotherham]

We were previously on axe-core@4.5.1 which is bundled with jest-axe

[colinrotherham]

I've created full-width.njk (without wrapping landmarks) alongside full-width-landmarks.njk

This is so Axe can pass some of the "landmark-in-landmark" violations

You'll see some Percy related diff changes as a result

[36degrees]

This looks ace @colinrotherham, thanks! Seems like a sensible approach, and great commit structure.

I think it makes sense to keep a accessibility.test.mjs for each component whilst we have individual rule exceptions, but is there any other reason for doing it that way? Wondering if in the future we might move to having this in a single place, like we have for HTML validation.

Can we make sure we add the accessibility tests to the required status checks for the main branch after this is merged?

[colinrotherham]

Thanks @36degrees 🙌

Ahh yep, it's to align with Running individual tests in testing-and-linting.md

The entire test suite is quite slow, so if you're only working on one component you can narrow things down:

Run Accordion tests only

Total time: 5s

npx jest --watch src/govuk/components/accordion

You can try npx jest --watch src/govuk/components/accordion --listFiles to see what's matched:

src/govuk/components/accordion/accessibility.test.mjs
src/govuk/components/accordion/accordion.test.js
src/govuk/components/accordion/template.test.js

Run Accessibility tests only

Total time: 55s

npx jest --selectProjects "Accessibility tests"

Run all non-build tests

Total time: 1m 20s (previously 25s)

npm test

Running npx jest will include build tasks but dist and package directories need building first

The added test time was a concern but well worth it

[36degrees]

That makes sense, thanks for explaining 👍🏻

[colinrotherham]

Can we make sure we add the accessibility tests to the required status checks for the main branch after this is merged?

@36degrees Great idea

I've just fixed the conflicts and pushed again

Some extra comments added to jest.config.mjs plus maxWorkers: '50%' to ensure Jest uses the same default across single project runs, multi-project runs, and with flags like --watch and --watchAll

Addresses a minor concern that 12x concurrent Chromium processes on my machine was a bit much!

npx jest --selectProjects "Accessibility tests"

Support for macOS ARM CPUs

Out of interest, but performance related, can you test this for me? I've got an Intel Mac

Enabling the ARM CPU experiment to see how much faster things run:

See config puppeteer.config.js

module.exports = {
  cacheDirectory: join(__dirname, '.cache', 'puppeteer'),
  experiments: {
    macArmChromiumEnabled: true
  }
}

Thanks

Update From @domoscargin: "Accessibility tests" total time 40% reduced to 24 seconds (~30% for all tests)

[colinrotherham]

Thought it best to push more commits to explain how to see these tests in action

Since our headless browser tests can be made visible with { headless: false } I've added:

1. Documentation for jestPuppeteer.debug() to pause tests
2. Jest config changes to only open 1x browser window
3. Jest teardown changes to keep browsers open

Update: With video demo

Radio.button.accessibility.tests.mp4

[36degrees]

The new commits look good to me, I'm just not sure about the documentation.

[36degrees]

This feels a bit backwards to me as it's not clear when you'd want to use it when you start reading the sentence. Would it make sense to flip it? Maybe something like:

If you want to inspect a test that's running in the browser, configure Jest Puppeteer to use non-headless mode by setting headless: false in the config and then use Jest Puppeteer's debug mode to pause the test execution.

Not blocking – just a suggestion.

Also, it might be nice to be able to change mode without editing the config – the default config seems to allow changing it using a HEADLESS env variable. Could we do similar? (Again, not-blocking, could be a follow up PR)

[colinrotherham]

Sounds fab let's do it

[colinrotherham]

I've made the suggested documentation change but with HEADLESS=false instead of { headless: false }

Branch protection settings also have Accessibility tests added 👍

(Other open PRs will need updating to include the new tests)