BUG: pa11y scans failing on site

[echappen]

Expected Behavior

Pa11y scans all URL's in the CI check. Normal runtime for pa11y scan is ~25 mins.

Actual Behavior

The Pa11y CI check is passing, but pa11y is failing to scan any URL's. (CI example)

Steps to reproduce the behavior

• Observe a CI scan post Oct 16
• [ ]
• [ ]

This issue is done when

• Pa11y resumes scanning all urls on the site
• [ ]
• [ ]

[echappen]

Observations:

• The CircleCI job succeeds, which allows code to pass CI and to be merged, but the pa11y run inside the job is failing. You can tell this by the drop in build time from ~25 mins to ~5 mins
• pa11y starts silently failing in the CircleCI job on this build for the Ruby 3 upgrade. However, that commit is small, and nothing in that commit seems relevant to pa11y, so I chalked it up to timing coincidence.
  • Update: it looks like this bug was introduced on the very first "successful" ruby-3 build. For example, subsequent builds on the older ruby do not have this issue.
• Other projects started having trouble with pa11y builds around this same time

What I’ve tried:

• Pa11y job ends in SIGHUP after serving the site, but before any pa11y output
• Running the same pa11y script locally works: ./serve-pa11yci && npm run pa11y-ci
• Running pa11y in a GH Codespace failed in the same way that Matt H’s runs failed (as in, puppeteer exploded immediately)
  • Adding a few chrome dependencies before running pa11y got it to work in GH/Codespaces
• But trying the above fix in CircleCI did not solve it, same behavior as before (SIGHUP)
• Trying to SSH into the CircleCI build to further examine the environment is blocked by zscaler. Open ssh connection just hangs, even after checking for proper ssh keys. There are no consistent IP addresses provided by Circle for this purpose, so getting them allow-listed by GSA IT wouldn’t be effective.
• Another idea is to get pa11y to fail more loudly in the CI job, but pa11y-ci doesn’t really offer a --verbose or equivalent config
• I was able to ssh into a failing build. I'm able to curl URLs in the sitemap that pa11y is supposed to be hitting. Running pa11y manually while a detached server is running still returns a generic Failed to run message when scanning the first URL. I haven't been able to get it to return a more specific error using different configs.

I will update this comment as I try more things.