fix(crawler): persist metadataOnly flag across resume (#369) - #378
Merged
Conversation
Fixes #369. LinkList#add(url, { metadataOnly: true }) only tracked the flag in memory, so LinkList#resume() re-added every restored URL with no options — a URL queued for a lightweight metadata-only scrape before an interruption silently became a full-scrape target once the crawl resumed. This surfaced most visibly when --dedupe-cap rejections and issue #350's auto-retry loop combined: pages never listed by the operator ended up fully scraped and written to content_items. Persists the decision as content_items.is_metadata_only, written by replaceAnchorEdges (which recomputes !recursive || anchor.isExternal, the same formula processAnchors uses) via resolveContentItemId's new isMetadataOnly option. getCrawlingState() now returns the pending subset flagged this way, and Crawler#resume() / LinkList#resume() thread it through to restore the in-memory flag exactly as it was before the interruption. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI on #378 caught a real regression: append.e2e.ts's "appended root is recorded as a fully-internal page" test failed because a URL that was external (or non-recursive) in an earlier session can carry a stale content_items.is_metadata_only=1 even after --append promotes it into mergedRoots. Crawler#resume() runs before the new roots are added via LinkList#add(), so the dedup in add() let the stale flag silently win over the root's full-scrape intent — the same class of bug #369 fixes, just via a different path (append promoting a URL to root instead of resume restoring pending state). Re-parses mergedRoots to withoutHashAndAuth and excludes them from the metadataOnlyUrls passed to Crawler#resume(), mirroring the existing root-exclusion in #crawlUntilPendingClears and CrawlerOrchestrator.resume. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #369.
--list/--list-file（非再帰）クロールでは、非再帰モードで発見された内部アンカーはLinkList#add(url, { metadataOnly: true })で「タイトルのみ取得」としてキューに積まれる。この
metadataOnlyフラグはメモリ上の Set にしか存在せず、--dedupe-capの拒否とauto-retry（#350）が組み合わさって resume が走ると、
LinkList#resume()が復元後のURL を metadataOnly 情報なしで再登録してしまい、リストに含まれない内部ページが
is_target=1（フルスクレイプ）としてcontent_itemsに書き込まれてしまっていた。content_items.is_metadata_only列を新設して永続化。書き込みはreplaceAnchorEdgesから
resolveContentItemId経由の1箇所のみ（!options.recursive || anchor.isExternalという
processAnchorsと同じ式を再計算）getCrawlingState()がpendingMetadataOnly（is_metadata_only=1の pending URL）を返すようになり、
Crawler#resume()/LinkList#resume()がそれを使ってメモリ上のmetadataOnly フラグを復元
is_metadata_only=1になるケースに対して、resume 直前に
list/config.rootsを使って除外する防御を追加Test plan
yarn buildyarn lintyarn test（5000件超、既存 + 新規）create-entity-tables.spec.ts: 新規列のデフォルト値migrate-content-items-is-metadata-only.spec.ts: マイグレーションの冪等性・旧アーカイブ対応resolve-content-item-id.spec.ts: INSERT/UPDATE時の書き込み・ラチェット挙動database.spec.ts:pendingMetadataOnlyの導出（recursive/anchor組み合わせ）link-list.spec.ts:resume()でのフラグ復元crawler.spec.ts: resume後の実際のスクレイプ挙動（title-only vs full）の回帰テストcrawler-orchestrator.spec.ts: 6経路（auto-retry, append, inventory, recrawl,retryFailed, static resume）すべてでの
pendingMetadataOnly配線 + root除外archive.spec.ts:Archive.setPage()レベルでのrecursive配線の統合テストスコープ外（既知の事項として記録）
isMetadataOnly()のキー正規化不一致（withoutHashAndAuth保存 vswithoutHash参照）is_metadata_onlyを書き込み時点でroot URLごと完全に除外する再設計（現状は resume 直前の読み取り側除外で対応。ARCHITECTURE.md に既知のギャップとして記載済み）
🤖 Generated with Claude Code