Skip to content

v4.0.0: Significantly improved crawl reliability, performance, and result triage

Latest
Latest

Choose a tag to compare

View all tags
@rolandschuetz rolandschuetz released this 22 Jun 15:51
v4.0.0
e9e7c30

v4.0.0

Version 4.0.0 is a major release that moves the package from CodeQ to NEOSidekick and significantly improves crawl reliability, performance, and result triage.

screenshot

Breaking changes / Upgrade notes

  • The Composer package is now neosidekick/linkchecker.
  • The Neos package key changed from CodeQ.LinkChecker to NEOSidekick.LinkChecker.
  • PHP namespaces changed from CodeQ\LinkChecker to NEOSidekick\LinkChecker.
  • Settings now live below NEOSidekick.LinkChecker.
  • The persisted result table is renamed from codeq_linkchecker_domain_model_resultitem to neosidekick_linkchecker_domain_model_resultitem.
  • Backend-triggered crawls now use Flowpack.JobQueue.Common, with a Doctrine queue configured by default.

A Flow code migration and Doctrine migrations are included. After upgrading, run:

./flow flow:core:migrate Your.SitePackage --force
./flow doctrine:migrate
./flow flowpack.jobqueue.common:queue:setup NEOSidekick.LinkChecker.Crawl

Backend crawls require a worker, for example:

./flow flowpack.jobqueue.common:job:work NEOSidekick.LinkChecker.Crawl --verbose

NEW FEATURES

  • New grouped backend triage UI for reviewing link issues by broken target, source page, domain, status, and impact.
  • Link health summary and impact filters to prioritize issues affecting many pages.
  • Bulk actions to mark whole groups as fixed or ignore them.
  • JobQueue-backed backend crawls with queue status, duplicate-job prevention, and failed job visibility.
  • checklinks:audit command for inspecting stored result health, duplicates, ignored rows, and active issues.
  • Optional incremental node crawling via --only-changed.
  • Optional between-run cache for healthy external links.
  • HEAD-first external link checks with GET fallback for faster checking.
  • Per-host rate limiting, response byte limits, redirect following, retry/backoff handling, and configurable user agent.
  • OK / warning / broken classification to reduce false positives.
  • Configurable ignore rules for suppressing known findings.
  • CC recipients for email notifications.

Changed

  • Backend crawl logic was moved out of the controller into dedicated runner and queue services.
  • Link checker results now use stable fingerprints to prevent duplicates.
  • Existing duplicate findings are deduplicated by migration.
  • Notifications now count only truly broken links, excluding warnings such as auth walls, bot blocks, rate limits, and invalid phone-number hints.
  • External redirects are followed by default so working redirected links are no longer reported as broken.
  • Transient server errors are revalidated before being persisted.
  • Long-running crawls now persist more frequently and clean up crawler/context state to improve stability.
  • Result labels, node labels, backend edit links, source links, and status badges were improved in the backend module.
  • English and German translations were expanded and corrected.
  • README documentation and backend screenshot were updated.

Fixed

  • Duplicate link checker results.
  • Long-running crawl persistence issues.
  • Backend crawl reset status handling.
  • Translation mistakes and incomplete HTTP status labels.
  • Several result accuracy issues that caused false positives.
  • Code style issues from the previous release line.

Removed

  • Removed the direct dependency on symfony/polyfill-php80.

Full changelog: v3.1.0...v4.0.0

Assets 2
Loading