Enhance automated accessibility testing in our development pipeline

[dav-idc]

What

Timeline: Ongoing work, with a strong focus until March 2023, and then continuing afterwards
Priority level: 3 – what reduces risk of inaccessible implementations
Category: Design system fundamentals

Potential tasks

• Research new automated accessibility testing tools and determine whether they can add benefits beyond what our jest-axe automated testing is currently achieving.
• Expand automated testing to include all example code snippets for each style, component and pattern.
• Research the feasibility of testing example code snippets with dynamic JavaScript and multiple states, such as the cookie banner and character count components.

Epic lead

Not yet assigned

Why

As of November 2022, there are limitations to how the team implements jest-axe as part of our automated testing:

1. The team only tests the first example code snippet for each component and pattern – most of our components and patterns have multiple example code snippets.
2. The team only tests the static HTML version of example code snippets, as found in the HTML tab – some components and patterns include JavaScript that modifies the HTML, which tests do not capture.

Who needs to work on this

Developers, accessibility specialist

Who needs to review this

Developers, tech leads

Initial goals

• New automated accessibility testing tools are researched and changes to our existing testing are proposed
• Automated testing is expanded to include all example code snippets
• Components with multiple states get testing for each state

Future goals

• Automated testing is expanded to include multiple stages of complex interactions and journeys (for example, cookie banner, character count, Exit this Page)
• Research on automated testing focussed on screen reader outputs is completed and potential options are pursued
• Processes are put in place to make sure tests are updated when new components are added or changed

[dav-idc]

Note: this is a duplicate of an older placeholder epic. I've closed the previous issue: alphagov/govuk-design-system#1937

[dav-idc]

Note: this replaces an other older issue related to tooling: #1971

[NickColley]

Just so you know one limitation of jest-axe is that it runs using JSDOM so checks like colour contrast checks do not work. Moving towards something where a real browser is orchestrated (which would also allow testing between interaction states) would improve the coverage of axe-core testing.

[colinrotherham]

@davidc-gds @NickColley Have either of you tried @axe-core/puppeteer (from Deque Systems)?

[dav-idc]

Hey @colinrotherham I've only heard of Puppeteer, and I'm not sure how it compares to jest-axe (or even if comparing the two would be a valid apples-to-apples comparison).

That being said, I can think of a few reasons to warrant further investigation into Puppeteer:

• It's made by Deque, the people who make axe-core
• The public npm is newer than jest-axe (Puppeteer was released 2 years ago, while jest-axe has been in govuk-frontend for at least 3 years?)
• I think puppeteer might be related or closely tied to jest in some way? I don't know much about either, but we have a line in our Package.JSON that says 'jest-environment-puppeteer'. There's also this page about using jest with puppeteer: https://jestjs.io/docs/puppeteer

[colinrotherham]

Thanks @davidc-gds it definitely looks handy

But yeah it wouldn't be like-for-like as jest-axe adds nice "reporting" for us

Just so you know one limitation of jest-axe is that it runs using JSDOM so checks like colour contrast checks do not work. Moving towards something where a real browser is orchestrated (which would also allow testing between interaction states) would improve the coverage of axe-core testing.

@NickColley I'd imagine jest-axe could use Puppeteer's Page.setContent() to render static HTML just like JSDOM? Once rendered, could pass the page object into @axe-core/puppeteer versus passing an HTML element into axe-core?

[NickColley]

I think unless there's a really compelling way to opt into a real browser environment I'd not change jest-axe.

Instead my gut feeling it's better to pick a good orchestration suite e.g. something like https://testcafe.io/ or https://go.cypress.io/ (used by the GOV.UK Prototype Kit project) and run axe as part of that.

• https://www.npmjs.com/package/@testcafe-community/axe
• https://www.npmjs.com/package/cypress-axe

This way you can write easier to maintain tests that interact with components in a real browser environment and run axe between the states.