Skip to content

Fix intermittent Lighthouse failures and scope to changed pages#1333

Merged
nrichers merged 4 commits into
mainfrom
nrichers/sc-12702/fix-intermittent-lighthouse-fail
May 22, 2026
Merged

Fix intermittent Lighthouse failures and scope to changed pages#1333
nrichers merged 4 commits into
mainfrom
nrichers/sc-12702/fix-intermittent-lighthouse-fail

Conversation

@nrichers
Copy link
Copy Markdown
Contributor

@nrichers nrichers commented May 18, 2026 •
edited
Loading

Pull Request Description

What and why?

Fixes sc-12702.

This updates Lighthouse CI so it:

  • Runs after Validate docs site deploys the PR preview instead of polling for validate to finish.
  • Audits changed site/ pages by default instead of the same root pages on every PR.
  • Preserves broader audits through manual workflow_dispatch (mode: depth, depth 0–2) and the lighthouse:full label.
  • Fails clearly when audited pages score below 0.9 accessibility.

How to test

For this PR:

  1. Confirm Validate docs site passes and posts a preview link.
  2. Confirm Lighthouse check is dispatched after validate, not started in parallel.
  3. Confirm Lighthouse audits only changed pages when a site/**/*.qmd file changes.
  4. Optionally run Actions → Lighthouse check → Run workflow with:
    • branch: nrichers/sc-12702/fix-intermittent-lighthouse-fail
    • pr_number: 1333
    • mode: changed
    • depth: 0

For broader coverage, add lighthouse:full or run manual mode: depth.

What was tested

  • Initial CI-only version correctly skipped Lighthouse with No site pages to audit in this PR.
  • Manual workflow_dispatch confirmed the Lighthouse workflow can run from the PR branch.
  • A test .qmd change correctly caused Lighthouse to audit only developer/how-to/test-sandbox.html.
  • That page failed accessibility at 0.88 across 3 runs, confirming the new < 0.9 failure path works.

Lighthouse run: https://github.com/validmind/documentation/actions/runs/26069380803 [expected failure: pre-existing test-sandbox.html accessibility score of 0.88 confirms the new < 0.9 fail path]
Report: https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1779152057612-6670.report.html

The 0.88 result appears to be a pre-existing page issue, likely related to the embedded sandbox iframe, not a regression from this PR.

What needs special review?

  • The validate workflow dispatching lighthouse-check.yaml after preview deploy.
  • PR context resolution and permissions for workflow_dispatch.
  • site/scripts/lighthouse_urls.py mapping changed files to preview URLs.
  • The new failure behavior when accessibility is below 0.9.

Dependencies, breaking changes, and deployment notes

CI-only change. No production deployment.

Release notes

Internal CI improvement — no user-facing release notes.

@nrichers
Fix intermittent Lighthouse CI and scope audits to changed pages (sc-…
0e3a7f7 
…12702)

Chain Lighthouse after validate via workflow_run instead of polling, audit
PR-changed site pages by default, and keep depth dispatch and lighthouse:full
for thorough runs.
@nrichers nrichers added the internal Not to be externalized in the release notes label May 18, 2026
@nrichers
Dispatch Lighthouse from validate after preview deploy (sc-12702)
c82745b 
workflow_run only runs from the default branch, so trigger Lighthouse via
workflow_dispatch from validate on the PR branch instead.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: c82745b

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: c82745b

@nrichers
test: Trigger Lighthouse changed-page audit via minor wording tweak
4459f46 
Exercises the changed-files path on PR 1333 — Lighthouse should audit
developer/how-to/test-sandbox.html only.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: 4459f46

@nrichers
Fix diff pathspec so script works from any cwd (sc-12702)
8f0a727 
Use ':(top)site/' to anchor the pathspec at the repo root regardless of
where the script is invoked from. The workflow runs it from site/scripts/,
which previously caused git diff to return zero files.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

PR Summary

This PR introduces significant improvements to the Lighthouse audit process for PR preview sites. The changes include:

  • Workflow Trigger Updates: The Lighthouse check workflow has been updated to use a manual trigger (workflow_dispatch) with configurable inputs such as mode, depth, and PR number. The polling mechanism for the preview site has been reduced by checking preview availability in a more streamlined loop.

  • PR Context Resolution: The workflow now resolves PR metadata using the GitHub API to adjust parameters dynamically (e.g., switching from 'changed' audit mode to 'depth' if the PR has the 'lighthouse:full' label).

  • Checkout and Environment Setup: The steps to checkout the repository and set environment variables (e.g., preview URL, commit SHA) have been refactored for clarity and maintainability.

  • Updated URL Generation: The script (lighthouse_urls.py) used to generate audit URLs has been enhanced with improved logic to determine which pages to audit both by detecting file changes and crawling the sitemap based on the specified depth. This includes support for global fallback when shared configuration files are modified.

  • Improved Error Handling and Reporting: The workflow now includes more robust error checking (e.g., verifying HTTP status codes, handling missing sitemap or URLs) and posts detailed audit results directly onto the PR as a comment. The Lighthouse audit now explicitly fails if any page falls below an accessibility threshold (score < 0.9).

  • Documentation and Tests: The README has been updated to describe the new workflow behavior and available options. Additionally, unit tests for the URL generation logic have been added in site/scripts/test_lighthouse_urls.py to ensure correct behavior for mapping site files to their corresponding preview URLs.

Test Suggestions

  • Run the workflow manually using the workflow_dispatch event with various input parameters (mode, depth, and PR number) to ensure proper operation.
  • Commit changes affecting both global files and individual site pages to verify that the correct audit mode (changed vs. depth) is selected.
  • Simulate preview site failures (e.g., inaccessible preview URL or missing sitemap.xml) to confirm the workflow correctly handles errors.
  • Execute the unit tests in site/scripts/test_lighthouse_urls.py to validate correct URL mapping and global fallback behavior.
  • Test the end-to-end Lighthouse audit process in a staging environment to verify that the PR comments display the expected accessibility scores and report details.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview

@nrichers nrichers changed the title Fix intermittent Lighthouse failures and scope to changed pages (sc-12702) Fix intermittent Lighthouse failures and scope to changed pages May 19, 2026
@nrichers nrichers requested a review from validbeck May 19, 2026 18:01
@validbeck validbeck mentioned this pull request May 20, 2026
Merged
11 tasks
validbeck
validbeck approved these changes May 22, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@validbeck validbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤞🏻

@nrichers nrichers merged commit d9a39c4 into main May 22, 2026
13 of 14 checks passed
@nrichers nrichers deleted the nrichers/sc-12702/fix-intermittent-lighthouse-fail branch May 22, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@validbeck validbeck validbeck approved these changes

Assignees

No one assigned

Labels

internal Not to be externalized in the release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nrichers @validbeck