3.3.0

Added

• Max crawl depth (CrawlerOptions.MaxDepth, CLI --maxDepth; 0 = unlimited): entry points are depth 0 and
  each followed link one deeper, so a link beyond the limit is not crawled. The depth a page was reached at is
  surfaced per URL on UrlReport.Depth. A URL over the limit is left undiscovered, so a later shorter path can
  still reach it.
• URL normalization before de-duplication (CrawlerOptions.NormalizeUrls, CLI --normalizeUrls, on by
  default): drops the #fragment, lowercases scheme/host, removes the default port, and collapses a trailing
  slash; the query string is left as-is (no reordering). A page's <link rel="canonical"> still determines the
  emitted URL, so distinct request URLs that share a canonical collapse to one result line.
• Include/exclude link filters (CLI --include/--exclude, repeatable;
  CrawlerOptions.IncludePatterns/ExcludePatterns): robots.txt-style path globs (* wildcard, $
  end-anchor) applied to discovered links only — entry points are always crawled, on every backend. New
  SimpleCrawler.Core.Filtering.UrlFilter reuses the robots matcher's longest-match/allow-wins resolution;
  excludes deny, includes default-deny everything unmatched, and an exclude out-matches an include of equal
  length.
• Opt-in per-page signal capture (CrawlerOptions.CapturePageSignals, CLI --captureSignals, off by default):
  records each fetched page's response headers, cookie names, <script src> sources, meta tags, and
  application/ld+json blocks onto a new UrlReport.Signals, surfaced in the --report output. It runs on
  every backend — static, JS, and headless. Header keys are lower-cased and a header that appears more than
  once is joined with a newline (never comma-joined, since Set-Cookie values embed commas in their expiry
  dates); cookie values are dropped. Off by default because UrlReport is checkpointed and the whole
  checkpoint is rewritten on every autosave, so capturing every page would bloat each autosave with the full
  crawl history rather than just the in-flight URLs.
• Pluggable per-page collectors (new SimpleCrawler.Core.Collectors namespace): an ICrawlCollector observes
  every page's HTTP response, and an IDomCollector additionally derives data from the DOM. Because there is
  no shared C# DOM across backends, the DOM half is expressed once as an IPageDom walk for the static
  backends (IStaticDomCollector) and once as an in-page JavaScript fragment for the rendered backends
  (IRenderedDomCollector.DomScript) — no backend carries any collector-specific code. Register collectors
  via AddCrawlCollectors (idempotent per implementation, so several backends in one container never
  double-register); they run for every backend with no change to the core pipeline or the backends, and each
  fragment is isolated so a throw or an unserializable result yields no data for that collector without
  disturbing the crawl or the others. The built-in PageSignalsCollector behind --captureSignals is one
  such collector.

Changed

• Updated Jint dependency to 4.12.0
• Checkpoint format: the full discovered-URL set is no longer serialized. CrawlState now persists only the
  pending frontier with per-URL depth (Frontier) and rebuilds the in-memory Discovered on load from
  Processed plus the frontier. A completed crawl checkpoints an empty frontier, so the file shrinks rather
  than growing with depth. Checkpoints written by earlier versions are not resumed (the crawl restarts).

Security

• Narrowed the V8 rendering engine's document access from EnableAllLoading to EnableWebLoading. All
  module imports are served by the custom V8ModuleLoader, whose fetcher only accepts http/https, so file
  loading was never needed; withholding EnableFileLoading removes a latent local-file-read path that
  untrusted page JS could otherwise resolve an import() toward.

---

Full changelog: v3.2.0...v3.3.0