v2.2.0

2.2.0 (2026-08-15)

Bug Fixes

• crawler: judge a public crawl by the pages it looked at, not the ones it indexed (81e44ad)

  • read the no-session guard from the recorded pages, which are written when a
    page is checked, so a run of walls cannot end a public crawl whose
    documentation is still being extracted
  • drop the indexed-page counter the guard replaced
  • pin the guard against reading only the window, and the authenticated message
    prefix that no test asserted

• crawler: keep a public crawl that hit a run of login walls in its nav (29089cd)

  • apply the no-session guard to the mid-crawl window rule as well as the
    end-of-crawl proportion rule, so a public site keeps the documentation it
    gave up and reports the walls it skipped
  • hoist the wall-at-a-path fixture and drop the duplicated exactly-half test

• crawler: redact a URL's parameters without rewriting the rest of it (46efeaf)

  • Add redactUrlSecrets and use it where a URL is shown to someone who then has to act on it. The log
    redaction it replaced rewrites anything shaped like token: wherever it appears, so the
    documentation path /docs/token:refresh became /docs/token=[REDACTED] - a URL that exists nowhere and
    cannot be authenticated against or passed to pathPrefix. The URL recorded is the page the crawl
    asked for, so that was any documentation URL, not just a login one.
  • Apply it to the abort message too, where the URL is a login redirect and likeliest to carry a token.
  • Count pages with no indexable content as missing as well. The branch that stops an agent being told
    "no need to poll again" existed for pages that are absent from the index, and those are absent; the
    completion description now names them, so the entry it sends the agent to is not a dead end.
  • Pin the branch order: an operation that failed while carrying page counts is still reported as
    failed rather than as one that merely missed some pages. Swapping the two left 881 tests green.
  • Name the variable for what it counts, tie the workflow's crawler mock to WorkflowCrawler rather than
    a second copy of its key list, and say at the count why it is taken before the tracker caps.

• crawler: stop failing a public crawl over the login pages in its own nav (24e9831)

  Extending the login checks to crawls without a session, so a site that answers with a wall is not
  indexed as documentation, applied two rules that only made sense with one.

  • Walls being half of a short crawl is not a dead session when there was no session: a public site with
    a /login and a /signup in its nav is two walls, and on a four-page site that is half. It now fails
    only if the crawl indexed nothing at all, which is what a site that needs authenticating looks like.
    An expired session still fails a crawl that indexed pages, because those pages are stale.
  • The entry page fails on confidence alone only when a session was used, where a login page means the
    session died. Without one, it has to be a wall: a public article about building login forms scores as
    a login page, because a code sample containing type="password" is worth two of the six indicators.
  • A wall found on a later attempt moves to the position the crawl found it in. Recorded first by an
    attempt that could not read the page, it could sit outside every later window and a run of five
    walls would slip both rules.
  • The stderr line names the redacted URL the report carries. The logger's own redaction covers fewer
    parameter names, so logging the raw URL echoed what the report withheld.
  • One login page left out is "1 login page", and three kinds of missing page read as "A; B; and C".

• crawler: stop indexing login walls and end a crawl whose session has died (d138014)

  A login wall served at HTTP 200 was indexed as documentation. Nothing failed: the request succeeded,
  the page had content, and the crawl reported complete. A session that died part-way through left every
  page after it a wall, and a crawl that never authenticated was not checked at all, so a site that
  answers with a sign-in page became a one-page index.

  • A wall is a page the detector calls a login page, asking for a password, with no article around the
    form. Documentation about signing in carries the same wording and the same form but has an article;
    the heading does not separate them, since a wall has one of those too.
  • A wall is never indexed. One is not a dead session, though: a public site's own /login is a wall, and
    a crawl without a pathPrefix reaches it from the nav. Its links are still followed, so a sparse
    section index mistaken for a wall does not take its whole section with it.
  • A run of them is a dead session: five of the last eight pages checked ends the crawl. A crawl too
    short to fill that window falls back to walls being at least half of what it saw.
  • Counted by requested URL, so a wall reached by redirect from five URLs is five pages the crawl lost,
    and a page the crawl retried is one page.
  • The check runs whether or not the crawl authenticated, and the failure says which of the two
    happened: a session that expired, or a site that needed authenticating before it could be indexed at
    all. Both name the page it happened on and offer the two remedies.

• indexing: name the login pages that left a crawl with nothing to index (110158e)

  • make the two wall-count rules plainly conditional on a session, which is the
    only state they could ever fire in: a public crawl records its entry page as
    something other than a wall and armed the guard for the rest of the crawl
  • fail a crawl that reached nothing but login walls with their count, the first
    URL and both remedies, the stats never being reached on that path
  • rewrite the two tests that pinned a map state production cannot produce

• indexing: stop telling an agent a crawl is complete when it is missing pages (279bb66)

  • The all-complete instruction said "No need to poll again", which outranks the status entry that
    says what is missing and how to fix it: POLLING_INSTRUCTION sends the agent to that sentence. An
    operation that finished without indexing everything now points at its entries instead.
  • Cap the reported URLs where they are stored. The whole status is serialized to the client on every
    poll, and the type already claimed to carry "the first few" while carrying all of them - a
    thousand-page crawl was 56KB of URLs per poll. The count keeps the true total.
  • Redact and normalize the URL as it is recorded. The same string reaches stderr through the logger's
    redaction, so a query carrying a token was scrubbed in one place and echoed to the client in the
    other.
  • Reset the skipped URLs per crawl and hand out a copy, matching what the sibling failed-page count
    already does. A status object holds that array after the crawl.
  • Pin what the suite could not see: the sentence join, the "and N more" boundary, the bracket when no
    URL is named, both resets, both copies, and the redaction.
  • Type the two crawler mock factories against the real classes with satisfies Pick<>. Deleting a hop
    is now a compile error rather than a silently passing suite, which is how three of them went
    unnoticed last round.

Features

• crawler: report the login pages a crawl left out of the index (1dc1e37), closes #292

  A wall kept out of the index was a stderr line and nothing else, so a crawl missing pages behind a
  login reported a clean success. The count now reaches the client the way pagesFailed does, and it
  carries the URLs: a count on its own cannot separate a correct skip from a mistaken one, and both
  remedies - authenticate, or restrict the crawl with pathPrefix - need to know which part of the site
  it was.

  • CrawleeCrawler collects the URLs it skipped rather than counting them; DocsCrawler forwards them.
  • IndexingStatus gains loginPagesSkipped and skippedLoginUrls, and the completion description names
    up to three of them with the remedy, instead of reading "Indexing complete".
  • Both kinds of absence are reported together, so pages that could not be fetched and pages that
    asked for a password no longer hide each other.
  • The progress notification counts them alongside skipped and failed pages.
  • Pin every hop: the crawler's getter, DocsCrawler's forwarding line, the workflow's stats call, the
    tracker's merge, the description, and the notification. Three of those were deletable with the