Adds site-with-errors test back, only checks PRs daily by lindseywild · Pull Request #156 · github/accessibility-scanner

lindseywild · 2026-03-13T13:49:16Z

Adds back the site-with-errors tests to run on every PR, but will only check issues.

PR creation validation will be run at a daily cadence so we can add enough of a timeout to ensure the Copilot PRs are created (sometimes they take several minutes). Also adds a check in the workflow if the test does fail, so we can address it. Current site-with-errors test added back passes, excluding the PR check 🎉

I'll have to run the test-copilot-pr-creation.yml workflow once this PR is merged but can make any follow-up changes necessary!

lindseywild · 2026-03-13T13:49:48Z

.github/workflows/test-copilot-pr-creation.yml

+name: Test Copilot PR creation
+on:
+  schedule:
+    - cron: "0 6 * * *" # Every day at 06:00 UTC


Random time; we can change to whenever.

lindseywild · 2026-03-13T13:50:19Z

.github/workflows/test-copilot-pr-creation.yml

@@ -0,0 +1,136 @@
+name: Test Copilot PR creation


This entire workflow is the same as test.yml, except it adds a PR boolean (see later comment)

lindseywild · 2026-03-13T13:50:32Z

.github/workflows/test-copilot-pr-creation.yml

+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
+          CACHE_PATH: ${{ steps.cache_key.outputs.cache_key }}
+          WAIT_FOR_PULL_REQUESTS: "true"


This is the only major change in this file.

lindseywild · 2026-03-13T13:51:42Z

tests/site-with-errors.test.ts

+ * Repeatedly calls `fn` until `predicate(result)` returns `true`, or until the timeout is exceeded.
+ * Errors thrown by `fn` (e.g. HTTP 404) are swallowed while polling continues.
+ */
+async function pollUntil<T>(


Basically just waiting until the PR is found, with a timeout of 15 minutes; that may be a bit too long, but hopefully it doesn't actually take this long for the PR to be created! We can experiment with lower timeouts if needed, but since this runs once a day I think it's probably fine.

lindseywild · 2026-03-13T13:52:56Z

tests/site-with-errors.test.ts


-  describe.runIf(!!process.env.GITHUB_TOKEN)('—', () => {
-    let octokit: Octokit
+  describe.runIf(!!process.env.GITHUB_TOKEN)('issues', () => {


Will only run against issues in the existing CI test, which are guaranteed -- but if they fail, then clearly there's real issue 😅

Copilot

Pull request overview

Re-enables the site-with-errors integration test suite and introduces a scheduled workflow intended to validate Copilot PR creation separately from per-PR checks, reducing PR CI latency while still monitoring PR-creation reliability.

Changes:

Unskips tests/site-with-errors.test.ts, adds polling helper, and gates PR validation behind WAIT_FOR_PULL_REQUESTS.
Adds a scheduled workflow to run the PR-creation validation daily (with cleanup and failure notification).
Adds a reusable workflow to open an issue when a workflow fails.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
tests/site-with-errors.test.ts	Re-enables the test, factors Octokit creation, adds polling, and splits issue vs PR assertions via env gating.
.github/workflows/test-copilot-pr-creation.yml	New daily/dispatch workflow to run scan + tests with PR polling enabled and cleanup/failure reporting.
.github/workflows/report-failure.yml	New reusable workflow intended to file an issue on failure.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

tests/site-with-errors.test.ts

 const OctokitWithThrottling = Octokit.plugin(throttling)

-describe.skip('site-with-errors', () => {
+const WAIT_FOR_PULL_REQUESTS = !!process.env.WAIT_FOR_PULL_REQUESTS


tests/site-with-errors.test.ts

+  const deadline = Date.now() + timeoutMs
+  let lastError: unknown
+  while (true) {
+    try {
+      const result = await fn()
+      if (predicate(result)) return result
+    } catch (error) {
+      lastError = error
+    }
+    if (Date.now() >= deadline) {
+      throw lastError ?? new Error(`Timed out after ${timeoutMs}ms waiting for condition`)
+    }


.github/workflows/test-copilot-pr-creation.yml

+                  branch="$(gh pr view "$URL" --json headRefName -q .headRefName || true)"
+                  if [[ -n "$branch" ]]; then
+                    echo "Deleting branch: $branch"
+                    gh api -X DELETE "repos/${{ env.TESTING_REPOSITORY }}/git/refs/heads/$branch" || echo "Failed to delete branch: $branch"


abdulahmad307 · 2026-03-13T15:18:07Z

.github/workflows/report-failure.yml

+      - uses: actions/github-script@v8
+        with:
+          script: |
+            github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo, title: "`${{ github.workflow }}` workflow failed", body: "The workflow, `${{ github.workflow }}`, failed to run. See [Action run output](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}) for more information. cc: @github/accessibility-reviewers." })


this seems like a legit issue, right? specifically about awaiting the issue creation?

.github/workflows/test-copilot-pr-creation.yml

+  notify-of-failures:
+    if: always() && github.ref_name == 'main' && needs.test_pull_request_creation.result == 'failure'
+    needs: [test_pull_request_creation]
+    uses: github/accessibility-scorecard/.github/workflows/report-failure.yml@main


abdulahmad307 · 2026-03-13T15:11:14Z

tests/site-with-errors.test.ts

+  while (true) {
+    try {
+      const result = await fn()
+      if (predicate(result)) return result


~~does a while loop respect the await call? (I don't think so, but I could be wrong). Should we add a time check? or we can use setInterval with a timeout of like 1 second.~~
actually 🤔 I'm prob wrong. I think it should be respecting it. but I do think this might be simpler if we use an interval instead of the while(true) loop?

would be something like this (I omitted the try/catch just to keep the example short):

return new Promise((resolve, reject) => { setInterval(async () => { const result = await fn() if (predicate(result)) { resolve(result) return; } if (Date.now() >= deadline) { reject(lastError ?? new Error(`Timed out after ${timeoutMs}ms waiting for condition`)) } }, 30 * 1000) })

although, idk if we have to clear the interval when we resolve/reject the promise with a clearInterval(intervalId. 🤷 I suppose either way is fine.

(to close the loop on this, we would have to clear the interval. I just tested and the interval keeps going even when the promise is resolved)

Yes, it does! It will still wait until the fn() resolves. The deadline var will handle a timeout if things are taking too long.

Oops sorry I missed this originally!!

abdulahmad307

I dont see anything major to be worried about. 🚀

Adds site-with-errors test back, only checks PRs daily

61e262c

lindseywild commented Mar 13, 2026

View reviewed changes

lindseywild marked this pull request as ready for review March 13, 2026 13:53

lindseywild requested a review from a team as a code owner March 13, 2026 13:53

lindseywild requested review from accessibility-bot and Copilot March 13, 2026 13:53

Copilot started reviewing on behalf of lindseywild March 13, 2026 13:53 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

Updates cache test with conditional PR check

a3ed935

abdulahmad307 reviewed Mar 13, 2026

View reviewed changes

abdulahmad307 approved these changes Mar 13, 2026

View reviewed changes

lindseywild merged commit b826e26 into main Mar 13, 2026
6 checks passed

lindseywild deleted the lw/adds-back-site-with-errors-test branch March 13, 2026 16:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds site-with-errors test back, only checks PRs daily#156

Adds site-with-errors test back, only checks PRs daily#156
lindseywild merged 2 commits intomainfrom
lw/adds-back-site-with-errors-test

lindseywild commented Mar 13, 2026 •

edited

Loading

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

lindseywild Mar 13, 2026 •

edited

Loading

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

abdulahmad307 Mar 13, 2026

Uh oh!

abdulahmad307 Mar 13, 2026 •

edited

Loading

Uh oh!

abdulahmad307 Mar 13, 2026 •

edited

Loading

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

lindseywild Mar 13, 2026

Uh oh!

abdulahmad307 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lindseywild commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

abdulahmad307 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

abdulahmad307 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abdulahmad307 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

lindseywild Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

abdulahmad307 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lindseywild commented Mar 13, 2026 •

edited

Loading

lindseywild Mar 13, 2026 •

edited

Loading

abdulahmad307 Mar 13, 2026 •

edited

Loading

abdulahmad307 Mar 13, 2026 •

edited

Loading