ci: check added links on PRs, sweep the rest weekly, fix two dead links - #712
Merged
rfay merged 2 commits intoAug 4, 2026
Merged
Conversation
…in linkspector ## The Issue The Link check workflow has failed on every push to main since 2026-07-31. Stack Overflow, tag1.com, and timeanddate.com return HTTP 403 to linkspector's headless Chrome. The 403 carries `cf-mitigated: challenge`, reproducible with plain curl, so it is a Cloudflare bot challenge rather than a link problem; Cloudflare's "Agent" category, offered to site owners on 2026-07-01, covers browser-driving agents. Auditing every URL the ignore list already suppressed turned up two genuinely dead links that the list was hiding. ## How This PR Solves The Issue Dead links fixed in the posts: - `news.extly.com` (February 2026 newsletter) resolves to 0.0.0.0. It was a syndicated copy; now points at the original Laravel News article. - A LinkedIn Pulse article (August 2025 newsletter) redirects to `article_not_found` with no Wayback snapshots. Repointed to the same tutorial on The Drop Times. - The timeanddate.com link to the past 2024-03-06 meeting is removed; the date remains as text. Ignore patterns added for hosts serving a Cloudflare challenge: stackoverflow.com, timeanddate.com (kept for future meeting announcements), pixabay.com. Widened the existing tag1consulting.com pattern to cover the newer tag1.com domain. ## Manual Testing Instructions Preview: https://20260803-rfay-linkspector-cl.ddev-com-front-end.pages.dev/ - /blog/ddev-february-2026-newsletter — "Read on Laravel News" resolves - /blog/ddev-august-2025-newsletter — Ollama item resolves to The Drop Times - /blog/2024-plans — meeting date reads as plain text - The Link check job passes ## Automated Testing Overview No new tests; the change is link-checker configuration and content. Verified the YAML parses, all 19 patterns compile, and they match the 12 failing URLs from run 30811952142 without matching the replacements. All 7 Stack Overflow URLs confirmed live via api.stackexchange.com. ## Release/Deployment Notes A Cloudflare challenge returns 403 whether or not the page exists, so the suppressed URLs on those hosts cannot be validated by CI and need occasional manual spot-checks. Cloudflare's new defaults land 2026-09-15, so more hosts may need this treatment. 🤖 Developed with assistance from [Claude Code](https://claude.ai/code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Deploying ddev-com-front-end with
|
| Latest commit: |
360c999
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0fe2ece2.ddev-com-front-end.pages.dev |
| Branch Preview URL: | https://20260803-rfay-linkspector-cl.ddev-com-front-end.pages.dev |
## The Issue Internal links are already validated by `astro-link-validator` during the Astro build, so this workflow only ever checked third-party URLs. Running the full sweep on every push to main meant any one of ~500 external links rotting turned the branch red: 9 of the last 20 pushes failed. A gate that is wrong half the time stops being read, and the ignore list grew an entry per newly-blocked host to keep it quiet. ## How This PR Solves The Issue Split the workflow into two jobs with the failure behavior each case deserves. The `pr` job keeps blocking with `filter_mode: added`, which is the case worth gating, since a typo'd URL in a new post fails while the author is still there to fix it. The `sweep` job takes over the full run, moves off push-to-main onto a weekly cron plus `workflow_dispatch`, and opens or updates a single tracking issue rather than failing; it closes that issue when every link resolves again. Link rot becomes a chore queue instead of a broken build. Also imported the hard-line-break guidance from ddev/ddev#8662 into `AGENTS.md`, extended to cover two cases this repo hits: commit bodies, which are reused verbatim as pull request descriptions here, and reports a workflow posts through `gh`. The sweep job's issue body is written one-line-per-paragraph for that reason. ## Manual Testing Instructions Preview: https://20260803-rfay-linkspector-cl.ddev-com-front-end.pages.dev/ - The blocking `Check blog links` job runs and passes on this PR - Run `Sweep all blog links` via workflow_dispatch on the branch; confirm it files an issue titled "Link check: broken external links in blog content" and that the job itself succeeds - Run it a second time and confirm the existing issue is updated rather than a duplicate created ## Automated Testing Overview No new tests. The workflow YAML parses and both job guards resolve as intended, every `run:` block is shellcheck-clean at warning level, and the report generation was tested end-to-end against real linkspector JSON output. The linkspector CLI contract the sweep depends on was verified directly: `-j` emits `{source, severity, diagnostics[]}` and exit status is non-zero when links fail. ## Release/Deployment Notes Link checking no longer runs on push to main, so merging will not surface link rot; the weekly sweep replaces it and first runs the following Monday. The sweep needs `issues: write`, granted at job scope only. If linkspector itself fails rather than merely finding bad links, the job fails loudly instead of silently reporting zero. 🤖 Developed with assistance from [Claude Code](https://claude.ai/code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
rfay
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Issue
The Link check workflow has failed on every push to main since 2026-07-31. Stack Overflow, tag1.com, and timeanddate.com return HTTP 403 to linkspector's headless Chrome. The 403 carries
cf-mitigated: challenge, reproducible with plain curl, so it is a Cloudflare bot challenge rather than a link problem; Cloudflare's "Agent" category, offered to site owners on 2026-07-01, covers browser-driving agents.Auditing every URL the ignore list already suppressed turned up two genuinely dead links that the list was hiding. Separately, the job's failure semantics are wrong: internal links are already validated by
astro-link-validatorduring the build, so this workflow only ever checks third-party URLs, and it failed on 9 of the last 20 pushes to main.How This PR Solves The Issue
Dead links fixed in the posts:
news.extly.com(February 2026 newsletter) resolves to 0.0.0.0. It was a syndicated copy; now points at the original Laravel News article.article_not_foundwith no Wayback snapshots. Repointed to the same tutorial on The Drop Times.Ignore patterns added for hosts serving a Cloudflare challenge: stackoverflow.com, timeanddate.com (kept for future meeting announcements), pixabay.com. Widened the existing tag1consulting.com pattern to cover the newer tag1.com domain.
Split the workflow into two jobs, each with the failure behavior its case deserves. PRs keep blocking on
filter_mode: added, which is the case worth gating — a typo'd URL in a new post fails while the author can still fix it. The full sweep moves off push-to-main onto a weekly schedule, where it opens or updates one tracking issue instead of breaking the branch, and closes that issue when everything resolves.Imported the hard-line-break guidance from ddev/ddev#8662 into
AGENTS.md, extended to cover commit bodies (reused verbatim as PR descriptions here) andgh-posted workflow reports.Manual Testing Instructions
Preview: https://20260803-rfay-linkspector-cl.ddev-com-front-end.pages.dev/
Sweep all blog linksjob via workflow_dispatch to confirm it reports without failingAutomated Testing Overview
No new tests; the change is link-checker configuration, CI wiring, and content. Verified the linkspector config YAML parses, all 19 patterns compile, and they match the 12 failing URLs from run 30811952142 without matching the replacements. All 7 Stack Overflow URLs confirmed live via api.stackexchange.com. The workflow YAML parses, every
run:block is shellcheck-clean at warning level, and the issue-report generation was tested end-to-end against real linkspector JSON.Release/Deployment Notes
Link checking no longer runs on push to main, so a merge will not surface link rot. The weekly sweep is the replacement; the first run lands the following Monday, and
workflow_dispatchtriggers it on demand.A Cloudflare challenge returns 403 whether or not the page exists, so the suppressed URLs on those hosts cannot be validated by CI and need occasional manual spot-checks. Cloudflare's new defaults land 2026-09-15, so more hosts may need this treatment.
🤖 Developed with assistance from Claude Code