feat: URL spidering — crawl linked pages with configurable depth and page limits (#315)

[RobertLD]

Summary

Implements opt-in URL spidering for the POST /api/v1/documents/url endpoint and the submit-document MCP tool. A single seed URL can now crawl and index an entire documentation site or wiki section in one call.

Closes #315.

Changes

New files

• src/core/link-extractor.ts — indexOf-based <a href> extraction (no regex, ReDoS-safe). Resolves relative URLs, strips fragments, deduplicates, filters non-http/https schemes.
• src/core/spider.ts — BFS crawl engine as an async generator. Yields one SpiderResult per page and returns SpiderStats when done.
• tests/unit/link-extractor.test.ts — 25 tests (relative resolution, dedup, fragment stripping, scheme filtering, edge cases)
• tests/unit/spider.test.ts — 20 tests (BFS order, depth/page limits, domain + path + pattern filtering, cycle detection, robots.txt, partial failure recovery, abortReason)

Modified files

• src/core/url-fetcher.ts — adds fetchRaw() export returning the raw body + contentType + finalUrl before HTML conversion, used by the spider for link extraction
• src/api/routes.ts — POST /api/v1/documents/url accepts spider options; returns { documents, pagesIndexed, pagesCrawled, pagesSkipped, errors, abortReason }
• src/mcp/server.ts — submit-document tool gains spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns parameters

Parameters

Parameter	Default	Hard cap	Description
spider	false	—	Enable link following
maxPages	25	200	Max pages indexed per run
maxDepth	2	5	Max hops from seed (0 = seed only)
sameDomain	true	—	Only follow same-domain links
pathPrefix	—	—	Only follow links under this path
excludePatterns	[]	—	Glob patterns for URLs to skip

Safety

• All URLs pass through the existing SSRF validation in fetchRaw() (DNS resolution, private IP blocking, scheme allowlist)
• Hard limits (200 pages, depth 5, 10-minute total timeout) cannot be overridden
• robots.txt fetched once per origin; User-agent: * and User-agent: libscope rules honoured
• 1-second inter-request delay by default
• Individual page fetch failures skip that page and continue crawling

Test plan

• npm run typecheck — zero errors
• npm test — 1059 tests passing (52 files)
• Link extractor: 25 tests covering all edge cases
• Spider: 20 tests covering BFS, limits, filtering, robots.txt, failures

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
libscope	Ignored	Preview	Mar 5, 2026 2:11am

[Copilot]

Pull request overview

Adds opt-in URL spidering to the URL indexing flow so a single seed URL can crawl and index linked pages (within configurable depth/page limits), reusing existing SSRF protections and integrating with both the REST API and MCP tooling.

Changes:

• Introduces a link extractor and BFS spider engine to crawl and yield pages from a seed URL.
• Exposes spidering options via POST /api/v1/documents/url and the submit-document MCP tool.
• Adds fetchRaw() to the URL fetcher to provide raw HTML/text for link extraction, plus new unit test coverage for the spider and extractor.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
src/core/link-extractor.ts	New indexOf-based <a href> extraction, normalization, and scheme filtering.
src/core/spider.ts	New BFS spider async generator with limits, filtering, and robots.txt handling.
src/core/url-fetcher.ts	Adds fetchRaw() export returning raw body/contentType/finalUrl for spider use.
src/api/routes.ts	Adds spider options and spider-mode response fields to POST /api/v1/documents/url.
src/mcp/server.ts	Adds spider parameters and spider-mode behavior to submit-document tool.
tests/unit/link-extractor.test.ts	New unit tests for link extraction and normalization edge cases.
tests/unit/spider.test.ts	New unit tests covering BFS order, limits, filtering, robots, and failure handling.

[Copilot]

The API accepts maxPages/maxDepth as any number and passes them straight through. Non-integers, NaN, or negative values currently lead to surprising behavior (e.g., the crawl loop won’t run at all if maxPages becomes NaN/negative). Consider validating/coercing these inputs at the route level (e.g., ensure finite integers and clamp to sensible min/max) before building spiderOptions.

Suggested change

	const spiderOptions: SpiderOptions = {
	fetchOptions,
	...(typeof b["maxPages"] === "number" && { maxPages: b["maxPages"] }),
	...(typeof b["maxDepth"] === "number" && { maxDepth: b["maxDepth"] }),
	const rawMaxPages = b["maxPages"];
	const rawMaxDepth = b["maxDepth"];
	let maxPages: number | undefined;
	if (rawMaxPages !== undefined) {
	if (typeof rawMaxPages !== "number" || !Number.isFinite(rawMaxPages)) {
	sendError(res, 400, "VALIDATION_ERROR", "Field 'maxPages' must be a finite number");
	return;
	}
	const normalized = Math.trunc(rawMaxPages);
	if (normalized <= 0) {
	sendError(res, 400, "VALIDATION_ERROR", "Field 'maxPages' must be a positive integer");
	return;
	}
	// Clamp to a sensible upper bound to avoid runaway crawls
	const MAX_SPIDER_PAGES = 1000;
	maxPages = Math.min(normalized, MAX_SPIDER_PAGES);
	}
	let maxDepth: number | undefined;
	if (rawMaxDepth !== undefined) {
	if (typeof rawMaxDepth !== "number" || !Number.isFinite(rawMaxDepth)) {
	sendError(res, 400, "VALIDATION_ERROR", "Field 'maxDepth' must be a finite number");
	return;
	}
	const normalized = Math.trunc(rawMaxDepth);
	if (normalized <= 0) {
	sendError(res, 400, "VALIDATION_ERROR", "Field 'maxDepth' must be a positive integer");
	return;
	}
	// Clamp to a sensible upper bound for crawl depth
	const MAX_SPIDER_DEPTH = 10;
	maxDepth = Math.min(normalized, MAX_SPIDER_DEPTH);
	}
	const spiderOptions: SpiderOptions = {
	fetchOptions,
	...(maxPages !== undefined && { maxPages }),
	...(maxDepth !== undefined && { maxDepth }),

[Copilot]

The spider tracks visited and yields/indexes using the pre-redirect URL (url), but link extraction uses raw.finalUrl. If a page redirects, this can lead to (1) indexing the wrong canonical URL and (2) fetching the final URL again later because visited contains only the pre-redirect URL. Consider normalizing to raw.finalUrl after fetch (use it for visited, for yielding, and as the base for further link extraction).

Suggested change

	// Convert to markdown
	const isHtml = raw.contentType.includes("text/html");
	const content = isHtml ? htmlToMarkdown(raw.body) : raw.body;
	const title = isHtml ? extractTitle(raw.body, url) : extractTextTitle(raw.body, url);
	stats.pagesIndexed++;
	yield { url, title, content, depth };
	// Extract and enqueue child links if we haven't hit maxDepth
	if (depth < maxDepth) {
	if (isHtml) {
	const links = extractLinks(raw.body, raw.finalUrl || url);
	for (const link of links) {
	if (!visited.has(link)) {
	queue.push({ url: link, depth: depth + 1 });
	}
	}
	log.debug({ url, linksFound: links.length }, "Spider: extracted links");
	const finalUrl = raw.finalUrl || url;
	visited.add(finalUrl);
	// Convert to markdown
	const isHtml = raw.contentType.includes("text/html");
	const content = isHtml ? htmlToMarkdown(raw.body) : raw.body;
	const title = isHtml
	? extractTitle(raw.body, finalUrl)
	: extractTextTitle(raw.body, finalUrl);
	stats.pagesIndexed++;
	yield { url: finalUrl, title, content, depth };
	// Extract and enqueue child links if we haven't hit maxDepth
	if (depth < maxDepth) {
	if (isHtml) {
	const links = extractLinks(raw.body, finalUrl);
	for (const link of links) {
	if (!visited.has(link)) {
	queue.push({ url: link, depth: depth + 1 });
	}
	}
	log.debug({ url: finalUrl, linksFound: links.length }, "Spider: extracted links");

[Copilot]

SpiderStats.pagesIndexed is produced by the spider itself, but the spider doesn’t actually perform indexing (it fetches + converts + yields). This name is likely to mislead callers into thinking it represents successfully indexed documents. Consider renaming to something like pagesFetched/pagesYielded (and let API/MCP layers report their own "indexed" counts).

[Copilot]

This PR adds new request parameters/response fields for POST /api/v1/documents/url, but src/api/openapi.ts doesn’t appear to document them (no references to "spider" there). Please update the OpenAPI spec so clients can discover spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns, and the new response fields.

[Copilot]

The conditional spread pattern ...(condition && { ... }) will create a false | {...} type and can fail strict TypeScript checks (and it isn’t the pattern used elsewhere in this repo). Consider switching to condition ? { ... } : {} spreads or building the options object via explicit if assignments.

Suggested change

	...(params.maxPages !== undefined && { maxPages: params.maxPages }),
	...(params.maxDepth !== undefined && { maxDepth: params.maxDepth }),
	...(params.sameDomain !== undefined && { sameDomain: params.sameDomain }),
	...(params.pathPrefix !== undefined && { pathPrefix: params.pathPrefix }),
	...(params.excludePatterns !== undefined && { excludePatterns: params.excludePatterns }),
	};
	};
	if (params.maxPages !== undefined) {
	spiderOptions.maxPages = params.maxPages;
	}
	if (params.maxDepth !== undefined) {
	spiderOptions.maxDepth = params.maxDepth;
	}
	if (params.sameDomain !== undefined) {
	spiderOptions.sameDomain = params.sameDomain;
	}
	if (params.pathPrefix !== undefined) {
	spiderOptions.pathPrefix = params.pathPrefix;
	}
	if (params.excludePatterns !== undefined) {
	spiderOptions.excludePatterns = params.excludePatterns;
	}

[Copilot]

extractHref() searches for the substring "href" anywhere in the tag, which can incorrectly treat attributes like data-href / aria-href as real links (since "href" is a suffix and is followed by =). This can cause the spider to crawl URLs that aren’t actual anchors. Consider requiring an attribute boundary (e.g., preceding char is whitespace/< and the next non-whitespace char after href is =), or explicitly matching href as a full attribute name.

[Copilot]

Robots.txt is fetched only once for the seed origin (fetchRobotsTxt(seedOrigin, ...)). When crawling subdomains (allowed by isSameDomain) or when sameDomain=false, this ends up applying the seed origin’s rules to other origins and ignores their own robots.txt entirely, which contradicts the stated behavior and can violate robots policies. Consider maintaining a per-origin cache (Map<origin, disallowSet>) and fetching/parsing robots.txt for each new origin encountered.

[Copilot]

In fetchRobotsTxt(), the object spread order allows caller-provided fetchOptions.timeout to override the intended 10s robots.txt timeout ({ timeout: 10_000, ...fetchOptions }). If the goal is a hard cap for robots.txt fetches, spread fetchOptions first and then set timeout (or explicitly clamp it).

Suggested change

	const raw = await fetchRaw(robotsUrl, { timeout: 10_000, ...fetchOptions });
	const effectiveTimeout =
	fetchOptions?.timeout !== undefined && Number.isFinite(fetchOptions.timeout)
	? Math.min(fetchOptions.timeout, 10_000)
	: 10_000;
	const raw = await fetchRaw(robotsUrl, { ...fetchOptions, timeout: effectiveTimeout });

[Copilot]

The conditional spread pattern ...(condition && { ... }) will produce a false | {...} type and is not used elsewhere in this codebase (which typically uses condition ? {...} : {} for spreads). In strict TS this can fail type-checking (Spread types may only be created from object types). Consider switching to ternaries or building spiderOptions via imperative if statements.

Suggested change

	...(typeof b["maxPages"] === "number" && { maxPages: b["maxPages"] }),
	...(typeof b["maxDepth"] === "number" && { maxDepth: b["maxDepth"] }),
	...(typeof b["sameDomain"] === "boolean" && { sameDomain: b["sameDomain"] }),
	...(typeof b["pathPrefix"] === "string" && { pathPrefix: b["pathPrefix"] }),
	...(Array.isArray(b["excludePatterns"]) && {
	excludePatterns: (b["excludePatterns"] as unknown[]).filter(
	(p): p is string => typeof p === "string",
	),
	}),
	...(typeof b["maxPages"] === "number" ? { maxPages: b["maxPages"] } : {}),
	...(typeof b["maxDepth"] === "number" ? { maxDepth: b["maxDepth"] } : {}),
	...(typeof b["sameDomain"] === "boolean" ? { sameDomain: b["sameDomain"] } : {}),
	...(typeof b["pathPrefix"] === "string" ? { pathPrefix: b["pathPrefix"] } : {}),
	...(Array.isArray(b["excludePatterns"])
	? {
	excludePatterns: (b["excludePatterns"] as unknown[]).filter(
	(p): p is string => typeof p === "string",
	),
	}
	: {}),

[Copilot]

In spider mode, fetch/init errors are caught and returned as HTTP 502, but single-URL mode relies on the outer handler (which currently doesn’t map FetchError to a 4xx/5xx explicitly). This creates inconsistent error semantics for the same endpoint. Consider removing the inner try/catch and letting the top-level handler decide, or adding consistent FetchError handling at the top level (and reusing it for both spider and non-spider paths).

Suggested change

	try {
	const gen = spiderUrl(url, spiderOptions);
	let result = await gen.next();
	while (!result.done) {
	const page = result.value;
	try {
	const doc = await indexDocument(db, provider, {
	content: page.content,
	title: page.title,
	sourceType: "manual",
	url: page.url,
	topicId,
	});
	indexedDocs.push({ id: doc.id, title: page.title, url: page.url });
	} catch (indexErr) {
	const msg = indexErr instanceof Error ? indexErr.message : String(indexErr);
	errors.push({ url: page.url, error: msg });
	}
	result = await gen.next();
	}
	// result.value is SpiderStats when done (generator is exhausted)
	if (result.done && result.value) {
	stats = result.value;
	stats.errors = errors;
	}
	} catch (err) {
	const msg = err instanceof Error ? err.message : String(err);
	sendError(res, 502, "FETCH_ERROR", msg);
	return;
	}
	const gen = spiderUrl(url, spiderOptions);
	let result = await gen.next();
	while (!result.done) {
	const page = result.value;
	try {
	const doc = await indexDocument(db, provider, {
	content: page.content,
	title: page.title,
	sourceType: "manual",
	url: page.url,
	topicId,
	});
	indexedDocs.push({ id: doc.id, title: page.title, url: page.url });
	} catch (indexErr) {
	const msg = indexErr instanceof Error ? indexErr.message : String(indexErr);
	errors.push({ url: page.url, error: msg });
	}
	result = await gen.next();
	}
	// result.value is SpiderStats when done (generator is exhausted)
	if (result.done && result.value) {
	stats = result.value;
	stats.errors = errors;
	}

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

[Copilot]

API response naming/semantics look inconsistent with the PR description: the PR text says the endpoint returns pagesIndexed, but the implementation returns pagesFetched, and sets it to indexedDocs.length (indexed pages), not the spider’s fetched/yielded count. Consider aligning the response fields with the documented contract (e.g. pagesIndexed for indexedDocs length, and optionally include pagesFetched from SpiderStats.pagesFetched if you need both).

[Copilot]

spiderUrl treats an invalid seedUrl as a FetchError (throw new FetchError("Invalid seed URL...")). In the API handler, FetchError is mapped to HTTP 502, so malformed input will surface as a server error instead of a 400 validation error. Consider throwing ValidationError here (or another error that routes maps to 400) for URL parsing failures.

[Copilot]

Redirects can bypass crawl filters: filtering (sameDomain/pathPrefix/excludePatterns/robots) happens before fetch, but after fetchRaw you switch to canonicalUrl = raw.finalUrl and yield it without re-checking. If an in-scope URL redirects to an out-of-scope domain/path, it will still be fetched/yielded and used as the base for link extraction. Re-apply the filtering/robots checks to canonicalUrl (and update stats) before yielding/enqueueing children.

Suggested change

	// Normalize to the final URL after any redirects.
	// This ensures the visited set, yielded URL, and link-extraction base are all consistent.
	const canonicalUrl = raw.finalUrl || url;
	if (canonicalUrl !== url) {
	visited.add(canonicalUrl);
	}
	// Convert to markdown
	const isHtml = raw.contentType.includes("text/html");
	const content = isHtml ? htmlToMarkdown(raw.body) : raw.body;
	const title = isHtml
	? extractTitle(raw.body, canonicalUrl)
	: extractTextTitle(raw.body, canonicalUrl);
	stats.pagesFetched++;
	yield { url: canonicalUrl, title, content, depth };
	// Extract and enqueue child links if we haven't hit maxDepth
	if (depth < maxDepth) {
	if (isHtml) {
	const links = extractLinks(raw.body, canonicalUrl);
	// Convert to markdown
	const isHtml = raw.contentType.includes("text/html");
	const content = isHtml ? htmlToMarkdown(raw.body) : raw.body;
	const title = isHtml
	? extractTitle(raw.body, url)
	: extractTextTitle(raw.body, url);
	stats.pagesFetched++;
	yield { url, title, content, depth };
	// Extract and enqueue child links if we haven't hit maxDepth
	if (depth < maxDepth) {
	if (isHtml) {
	const links = extractLinks(raw.body, url);

[Copilot]

extractLinks finds the end of the <a ...> opening tag via html.indexOf(">", tagStart). This will terminate early if a > appears inside a quoted attribute value (valid HTML), causing extractHref to parse an incomplete tag and potentially miss/extract incorrect links. The tag-end scan should skip over quoted sections (similar to stripTags in spider.ts) instead of using a raw indexOf.

[Copilot]

extractHref only treats space/tab as whitespace around the = and before the value. Newlines/CR are also valid attribute whitespace in HTML, so href\n=... or href=\n"..." won't be parsed. Consider using the same whitespace set used elsewhere in this file (space/tab/\n/\r) for the loops that skip whitespace.

[Copilot]

In spider mode, stats.errors = errors overwrites the spider's own fetch/robots/etc error list returned from spiderUrl. This can silently drop crawl errors. Consider keeping these as separate arrays (e.g. crawlErrors vs indexErrors) or merging them rather than replacing the generator’s stats.

Suggested change

	stats = result.value;
	stats.errors = errors;
	const generatorStats = result.value;
	const combinedErrors =
	Array.isArray(generatorStats.errors) && generatorStats.errors.length > 0
	? [...generatorStats.errors, ...errors]
	: errors;
	stats = {
	...generatorStats,
	errors: combinedErrors,
	};