pa11y reports false positives for color contrast

[gcamacho079]

Expected behaviour

On the two pages that I'm testing, I'm expecting zero accessibility issues.

Actual behaviour

I'm getting 16 issues on these two pages, all related to color contrast. I've introduced a wait of 2000ms to ensure the pages are entirely loaded, and captured screenshots to verify that the pages are in the correct state.

The following is an issue on the login page:

{
  ...
  issues: [
    {
      code: 'color-contrast',
      type: 'error',
      typeCode: 1,
      message: 'Elements must have sufficient color contrast (https://dequeuniversity.com/rules/axe/4.2/color-contrast?application=axeAPI)',
      context: '<label for="rememberMe">\n    Keep me logged in\n    </label>',
      selector: '#rememberMe-field > div > label',
      runner: 'axe',
      runnerExtras: [Object]
    }
  ]
}

However, when I use Chrome DevTools to analyze the color contrast, it has a 7.33:1 ratio.

This is the screenshot captured by pa11y during the test:

Steps to reproduce

Running pa11y with the axe test runner

Environment:

System:
    OS: macOS 12.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 677.34 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.0 - ~/.nvm/versions/node/v16.13.0/bin/node
    npm: 8.1.0 - ~/.nvm/versions/node/v16.13.0/bin/npm
  pa11y: 6.2.0

[smustgrave]

I'm starting to this see this too. Getting a dozen errors on every page about this but testing with the color contrast analyzer app proves they pass

[josebolos]

Hi @gcamacho079, @smustgrave

Could you please provide a publicly-accessible example page that contains this problem that we can test?

Thanks.

[smustgrave]

Our site has not yet launched but here's a screenshot of the text that's failing. Oddly enough it's only the links with sub items.

[smustgrave]

And using the axe runner.

[josebolos]

Thanks but without some code to test there's no way for us to replicate the issue and determine if it's a problem in pa11y or in axe.

Feel free to update this issue if you manage to create a reduced test case in Codepen or a similar place and we'll have a look.

[smustgrave]

Here's another website we have I was able to test https://www.pandemicoversight.gov/
Error happens there too

[smustgrave]

See that pa11y is using axe-core 4.2 but they're on 4.4 wonder if that could be it

[smustgrave]

dequelabs/axe-core#3431 opened this too for axe

[dboulet]

In some cases, when axe can’t determine the contrast ratio for certain elements, for example if the element has a background image or contains a pseudo-element, it will report the issue as “needs review”. The problem is that pa11y is currently reporting these review items as errors, as described in #623.