chore: merge development into main by RobertLD · Pull Request #349 · RobertLD/libscope

RobertLD · 2026-03-05T02:40:52Z

Merge development branch into main to bring in:

fix: merge-gate now allows release-please branches (fix: allow release-please PRs to pass merge gate and trigger CI #348)
docs: comprehensive documentation update for v1.3.0 (docs: comprehensive documentation update for v1.3.0 #347)
fix: concurrent pack installation race condition (feat: concurrent pack installation and -v verbose shorthand (issue #330) #339)
And other recent work

After this merges, release-please will update PR #238 with the full changelog and CI will work correctly for the release.

* chore: add development branch workflow - Add merge-gate.yml: enforces only 'development' can merge into main - Update CI/CodeQL/Docker workflows to run on both main and development - Update dependabot.yml: target-branch set to development for all ecosystems - Update copilot-instructions.md: document branch workflow convention - Rulesets configured: Main (requires merge-gate + squash-only), Development (requires CI status checks + PR) - Default branch set to development - All open PRs retargeted to development Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip Vercel preview deployments on non-main branches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger check refresh --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: comprehensive audit fixes — security, performance, resilience, API hardening Addresses findings from issue #314: - SSRF protection for webhook URLs (CRITICAL) - Scrub secrets from exports - Stored XSS prevention on document URL - O(n²) and N+1 fixes in bulk operations - Rate limit cache eviction improvement - SSE backpressure handling - Replace raw Error() with typed errors - Fetch timeouts on all network calls - Input validation on API parameters - Search query length limit - Silent catch block logging - DNS rebinding check fix - N+1 in Slack user resolution - Pagination on webhook/search list endpoints Closes #314 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: fix SSE listener leak, preserve caller signals, add SSRF validation to webhook test, chunk SQL params, use dynamic import in test - SSE backpressure: create single disconnect promise, race against drain (no listener accumulation) - http-utils.ts/onenote.ts: use AbortSignal.any() to combine caller signal with timeout - Webhook test endpoint: validate URL with validateWebhookUrlSsrf before fetch - bulk.ts: chunk IN clause to 999 params max (SQLite limit) - webhooks.test.ts: dynamic import after vi.mock() for deterministic DNS mocking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: consolidate and fix CI/CD workflows - Merge lint + typecheck into single job (saves one npm ci) - Add concurrency groups to ci, docker, codeql (cancel stale runs) - Add dependency-review-action on PRs (block vulnerable deps) - Add workflow_call trigger to ci.yml for reusability - Remove duplicate npm publish from release.yml (release-please owns it) - Fix SDK paths: sdk-go/ → sdk/go/, sdk-python/ → sdk/python/ - Fix Dependabot paths to match actual SDK directories - Add github-actions ecosystem to Dependabot (keep actions up to date) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb48187. * feat: add --from option to pack create for folder/URL sources Adds createPackFromSource() that builds packs directly from local folders, files, or URLs without requiring database interaction. CLI: libscope pack create --name my-pack --from ~/docs/ [--extensions .md,.html] [--exclude pattern] [--no-recursive] Features: - Walks directories recursively using registered parsers - Fetches URLs via fetchAndConvert - Supports extension filtering, exclude patterns, progress callback - Multiple --from sources supported - Output format identical to DB export (pack install works unchanged) Closes #328 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: fix prettier formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add gzip support for pack files (.json.gz) Pack files can now be compressed with gzip for smaller distribution: - writePackFile/readPackFile auto-detect gzip by extension or magic bytes - installPack accepts both .json and .json.gz files - createPackFromSource defaults to .json.gz output (source packs can be large) - createPack (DB export) still defaults to .json - Auto-detects gzip by magic bytes even if extension is .json 5 new tests covering gzip write, install, magic byte detection, and round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add progress logging and fix dedup handling in pack install - Log each document as it's indexed so large installs show progress - Change pack install to use dedup: 'skip' for graceful duplicate handling - Make title+content_length dedup check respect the dedup mode setting (previously it always threw ValidationError regardless of dedup mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: auto-generate tags during pack creation and apply on install - Export tokenize() and add suggestTagsFromText() in tags.ts for DB-free tag generation - createPackFromSource() now auto-generates tags per document via TF-IDF - installPack() applies doc.tags via addTagsToDocument() after indexing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb48187. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bumps [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) from 9.1.2 to 10.1.8. - [Release notes](https://github.com/prettier/eslint-config-prettier/releases) - [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md) - [Commits](https://github.com/prettier/eslint-config-prettier/commits/v10.1.8) --- updated-dependencies: - dependency-name: eslint-config-prettier dependency-version: 10.1.8 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps the minor-and-patch group with 1 update: [lint-staged](https://github.com/lint-staged/lint-staged). Updates `lint-staged` from 16.3.1 to 16.3.2 - [Release notes](https://github.com/lint-staged/lint-staged/releases) - [Changelog](https://github.com/lint-staged/lint-staged/blob/main/CHANGELOG.md) - [Commits](lint-staged/lint-staged@v16.3.1...v16.3.2) --- updated-dependencies: - dependency-name: lint-staged dependency-version: 16.3.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps the actions group with 5 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `6` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `7` | | [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` | | [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` | Updates `actions/checkout` from 4 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v6) Updates `actions/setup-node` from 4 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v6) Updates `actions/upload-artifact` from 4 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v7) Updates `actions/setup-python` from 5 to 6 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) Updates `actions/setup-go` from 5 to 6 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 22.19.13 to 25.3.3. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.3.3 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) from 11.10.0 to 12.6.2. - [Release notes](https://github.com/WiseLibs/better-sqlite3/releases) - [Commits](WiseLibs/better-sqlite3@v11.10.0...v12.6.2) --- updated-dependencies: - dependency-name: better-sqlite3 dependency-version: 12.6.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [eslint](https://github.com/eslint/eslint) from 9.39.3 to 10.0.2. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v9.39.3...v10.0.2) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert DeRienzo <rderienzo@voloridge.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add passthrough LLM mode for ask-question tool Adds llm.provider = "passthrough" so the ask-question MCP tool returns retrieved context chunks directly to the calling LLM instead of requiring a separate OpenAI/Ollama provider. This is the natural design for MCP tools where the client already has an LLM (e.g. Claude Code). - config.ts: add "passthrough" to llm.provider union type and env var handling - rag.ts: add isPassthroughMode() helper and getContextForQuestion() which retrieves and formats context without an LLM call - mcp/server.ts: ask-question checks passthrough first and returns context directly; falls through to existing LLM path otherwise Enable via config: { "llm": { "provider": "passthrough" } } Enable via env var: LIBSCOPE_LLM_PROVIDER=passthrough Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: format config.ts and include passthrough in provider override - Reformat long if-condition to satisfy prettier (printWidth: 100) - Fix logic bug: passthrough provider was checked in outer condition but not spread into overrides.llm.provider Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address 9 audit findings from issue #332 Security - middleware: use timingSafeEqual for API key comparison (#2) - url-fetcher, http-utils: replace process-wide NODE_TLS_REJECT_UNAUTHORIZED mutation with per-request undici Agent to eliminate TLS race condition (#1) Bugs - indexing: re-throw unexpected embedding errors so transaction rolls back instead of silently committing chunks with no vector (#3) - search: replace correlated minRating subquery with avg_r.avg_rating from the pre-joined aggregate in FTS and LIKE search paths (#4) Performance - bulk: replace O(n²) docs.find() loops with pre-built Map; replace per-document getDocumentTags() calls with a single getDocumentTagsBatch() query (#5) - config: add 30-second TTL cache to loadConfig() so disk reads are not repeated on every request (#6) Code quality - routes: check res.write() return value to handle SSE backpressure (#7) - reindex: delegate to schema.createVectorTable() instead of duplicating the vec0 DDL inline (#8) - obsidian: replace hand-rolled parseSimpleYaml() with js-yaml, normalise Date objects back to ISO-8601 strings (#9) Docs - agents.md: expand architecture tree to include src/api/ and src/connectors/; add Security Patterns section with correct undici examples - CONTRIBUTING.md: fix check-suite command (npm test → npm run test:coverage) and correct coverage threshold (80% → actual 75%/74%) Tests - bulk.test: add dateFrom/dateTo filter coverage - config.test: add cache-hit test; call invalidateConfigCache() before env-var tests so TTL cache doesn't return stale results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused warnIfTlsBypassMissing function Dead code after conflict resolution chose the per-request undici Agent approach (which doesn't need a warning about NODE_TLS_REJECT_UNAUTHORIZED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update tests for config cache and retry semantics - Add invalidateConfigCache() before loadConfig() in 4 env-override tests that were failing because the 30s TTL cache introduced in the config module was returning stale results from the previous test's cache entry - Update http-utils retry assertion: maxRetries=2 means 1 initial + 2 retries = 3 total calls (loop is attempt <= maxRetries) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

… (#336) - Add `src/cli/reporter.ts`: PrettyReporter (ANSI colors + \r progress bar), SilentReporter (no-op for verbose/JSON mode), `isVerbose()` and `createReporter()` factory. `LIBSCOPE_VERBOSE=1` env var alternative. - Update `setupLogging` to default to "silent" in CLI mode (pretty reporter handles user-facing output). Verbose/`--log-level` flags still route to structured JSON pino logs. Fix duplicate `initLogger` calls in onenote connect/disconnect commands to use `setupLogging` consistently. - Update `installPack` in `packs.ts` to support batch embedding and progress reporting: - New `InstallOptions` interface with `batchSize`, `resumeFrom`, `onProgress` fields - Batch documents: chunk all → single `provider.embedBatch` call per batch → single SQLite transaction per batch (avoids N embedding calls) - `resumeFrom` skips the first N documents (enables partial install resume after failure) - `InstallResult` now includes `errors` count - Add `--batch-size` and `--resume-from` CLI options to `pack install` - Tests: `tests/unit/reporter.test.ts` (17 tests covering PrettyReporter, SilentReporter, isVerbose, env var detection); extended `tests/unit/packs.test.ts` with 7 new tests for progress callbacks, batch efficiency, resumeFrom, embedBatch failure handling. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…dates (#337) Bumps the npm_and_yarn group with 2 updates in the / directory: [@hono/node-server](https://github.com/honojs/node-server) and [hono](https://github.com/honojs/hono). Updates `@hono/node-server` from 1.19.9 to 1.19.10 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](honojs/node-server@v1.19.9...v1.19.10) Updates `hono` from 4.12.3 to 4.12.5 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](honojs/hono@v4.12.3...v4.12.5) --- updated-dependencies: - dependency-name: "@hono/node-server" dependency-version: 1.19.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: hono dependency-version: 4.12.5 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (closes #340) **SSRF (CWE-918 — CodeQL alert #28)** Replace the two-step validate-then-fetch approach in url-fetcher.ts with IP-pinned requests using node:http / node:https directly. validateUrl() resolves DNS and checks for private IPs, then the validated IP is passed straight to the TCP connection (hostname: pinnedIp, servername: original hostname for TLS SNI). There is now zero TOCTOU window between validation and the actual network request. The redundant post-fetch DNS rebinding check and the env-var-based TLS bypass wrapper are removed; rejectUnauthorized is now passed directly to the request options. An internal _setRequestImpl hook is exported for unit test injection so tests can stub responses without touching node:http / node:https. Tests are updated accordingly. **ReDoS (CWE-1333 — CodeQL alert #24)** Five regexes in confluence.ts used the pattern [^>]*ac:name="X"[^>]* — two [^>]* quantifiers around a fixed literal. For input that contains a large attribute blob without the target ac:name value, the engine must try all O(n²) splits before concluding no match (catastrophic backtracking). Fix: rewrite the leading [^>]* as (?:(?!ac:name="X")[^>])* — a negative lookahead prevents the quantifier from overlapping with the literal, making backtracking structurally impossible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace remaining polynomial regexes in confluence.ts with indexOf helpers The conflict resolution left the original `/<ac:structured-macro [^>]*>([\s\S]*?) <\/ac:structured-macro>/gi` patterns in place for code blocks and info/tip panels (those were not part of the original security fix diff). These have the same O(n²) backtracking problem: with k opening tags and no closing tags, [\s\S]*? scans O(n - pos) chars per attempt, totalling O(n²). Replace the entire convertConfluenceStorage function with the indexOf-based approach (replaceStructuredMacros / replaceTagPairs / extractTagContent helpers) that eliminates all regex-based tag parsing. Also add removeSelfClosingMacros to handle the self-closing TOC case without regex, since the previous self-closing fix still used a [^>]*ac:name="toc"[^>]* pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…) (#339) * feat: concurrent pack installation and -v verbose shorthand (issue #330) Add concurrent batch embedding to installPack for significant performance improvement on large packs, plus CLI ergonomics improvements. Key changes: - `InstallOptions.concurrency` (default: 4): controls how many embedBatch calls run simultaneously; embedding is I/O-bound so parallelism directly reduces wall-clock installation time - Refactor installPack to pre-chunk all documents upfront, then use a semaphore-based scheduler to run up to `concurrency` embedBatch calls concurrently while inserting completed batches in-order (SQLite requires serialised writes); progress callbacks fire after each batch as before - `pack install --concurrency <n>` CLI flag exposes the new option - `-v` shorthand for `--verbose` on the global program options - Fix transaction install-count tracking: count committed docs accurately without relying on subtract-on-failure arithmetic - Add 6 new tests covering concurrency=1 sequential, concurrency=4 parallel, multiple embedBatch calls per install, concurrency limit enforcement, incremental progress reporting, and partial-failure error counting https://claude.ai/code/session_019hzhbEgV1ysnGmFVXBWzkZ * fix: address all 4 Copilot review comments on PR #339 - Validate batchSize, concurrency, resumeFrom at the start of installPack and throw ValidationError for invalid values (comments 3 & 4). Concurrency <= 0 would silently hang the semaphore indefinitely. - Add CLI lower-bound guard: --concurrency < 1 exits with a user-facing error before ever calling installPack (comment 3). - Lazy chunking: pre-chunking all documents upfront held chunks for the entire pack in memory simultaneously. Batches now store only the raw documents; resolveBatch() chunks on demand right before embedBatch is called, so only one batch's worth of chunks is in memory at a time (comment 2). - Wrap provider.embedBatch() in try/catch so synchronous throws are converted to rejected Promises rather than escaping scheduleNext() and leaving the outer Promise permanently pending (comment 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

- Guard JSON.parse in rowToWebhook with try/catch, default to [] - Guard JSON.parse in rowToSavedSearch with try/catch, default to null - Guard JSON.parse in loadDbConnectorConfig with try/catch, throw ConfigError - Push dateFrom/dateTo filters into SQL WHERE in listDocuments (before LIMIT) - Validate negative limit in resolveSelector, throw ValidationError - Replace manual substring extension parsing with path.extname() in packs.ts - Verified reporter.ts is already tracked on development (no action needed) - Added tests for all fixes (corrupted JSON, SQL-level date filtering, negative limit) Closes #342 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…page limits (#315) (#343) * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) Adds opt-in spidering to URL indexing. A single seed URL can now crawl and index an entire documentation site or wiki section in one call. New files: - src/core/link-extractor.ts: indexOf-based <a href> extraction, relative URL resolution, fragment stripping, dedup, scheme filtering. No regex. - src/core/spider.ts: BFS crawl engine with sameDomain, pathPrefix, excludePatterns (glob), maxPages (hard cap 200), maxDepth (hard cap 5), 10-min total timeout, robots.txt (User-agent: * and libscope), and 1s inter-request delay. Yields SpiderResult per page; returns SpiderStats. - tests/unit/link-extractor.test.ts: 25 tests covering relative resolution, dedup, fragment stripping, scheme filtering, attribute order, edge cases. - tests/unit/spider.test.ts: 20 tests covering BFS order, depth/page limits, domain + path + pattern filtering, cycle detection, robots.txt, partial failure recovery, stats, and abortReason reporting. Modified: - src/core/url-fetcher.ts: adds fetchRaw() export returning raw body + contentType + finalUrl before HTML-to-markdown conversion, so the spider can extract links from HTML before conversion. - src/api/routes.ts: POST /api/v1/documents/url now accepts spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns. Returns { documents, pagesIndexed, pagesCrawled, pagesSkipped, errors, abortReason }. - src/mcp/server.ts: submit-document tool gains spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns parameters. Safety: all fetched URLs pass through the existing SSRF validation in fetchRaw() (DNS resolution, private IP blocking, scheme allowlist). Hard limits (200 pages, depth 5, 10min) cannot be overridden by callers. robots.txt is fetched once per origin and Disallow rules are honoured. Individual page failures do not abort the crawl. Closes #315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve CI lint errors in spider implementation - Remove unnecessary type assertions (routes.ts, mcp/server.ts) — TypeScript already narrows SpiderResult/SpiderStats from the generator - Add explicit return type annotation on mock fetchRaw to satisfy no-unsafe-return rule in spider.test.ts - Replace .resolves.not.toThrow() with a direct assertion — vitest .resolves requires a Promise, not an async function Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address CodeQL security findings in spider/link-extractor link-extractor.ts (CodeQL #30 — incomplete URL scheme check): Replace the enumerated scheme blocklist (javascript:, vbscript:, etc.) with a strict http/https allowlist check on the resolved URL protocol. An allowlist is exhaustive by definition; a blocklist will always miss obscure schemes like vbscript:, blob:, or future additions. spider.ts (CodeQL #31 — incomplete multi-character sanitization): Replace the regex-based tag stripper /<[^>]+>/g in extractTitle() with an indexOf-based stripTags() function. The regex stops at the first > which can be inside a quoted attribute value (e.g. <img alt="a>b">), potentially leaving partial tag content in the extracted title. The new implementation walks quoted attribute values explicitly so no tag content leaks through regardless of its internal structure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address all Copilot review comments on spider PR (#343) - link-extractor: add word-boundary check in extractHref to prevent matching data-href, aria-href (false positives on non-href attributes) - spider: rename pagesIndexed → pagesFetched throughout (SpiderStats interface already used pagesFetched; sync implementation + tests) - spider: per-origin robots.txt cache (Map<origin, Set>) fetched lazily as new origins are encountered during crawl (was seed-only before) - spider: normalize to raw.finalUrl after redirects — visited set, yielded URL, and link-extraction base all use the canonical URL - routes: validate maxPages/maxDepth are finite positive integers - routes: change conditional spread &&-patterns to ternaries - routes: remove inner try/catch for spider fetch errors; add FetchError to top-level handler (consistent with single-URL mode → 502) - mcp/server: replace conditional spreads with explicit if assignments - mcp/server: validate spider=true requires url (throws ValidationError) - openapi: document spider request fields in IndexFromUrlRequest schema, add SpiderResponse schema, update 201 response to oneOf Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix prettier formatting in spider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

- README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs: comprehensive documentation update for v1.3.0 - README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: allow release-please PRs to pass merge gate and trigger CI Two issues prevented PR #238 from getting CI runs: 1. merge-gate blocked release-please PRs — the gate only allowed 'development' as the source branch, but release-please uses 'release-please--branches--main--components--libscope'. Updated to allow any branch matching 'release-please--*'. 2. CI never ran on the PR — GitHub does not trigger workflows when GITHUB_TOKEN creates a PR (intentional security restriction to prevent infinite loops). Fixed by passing a PAT via secrets.GH_TOKEN to the release-please action so its PR creation triggers CI. Note: requires a 'GH_TOKEN' secret in repo settings — a classic PAT with repo and workflow scopes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel · 2026-03-05T02:40:56Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
libscope	Ignored		Mar 5, 2026 2:40am

Copilot

Pull request overview

This PR merges development into main to bring the latest feature, fix, and documentation changes into the release branch, including CI/merge-gate updates, new spidering/indexing capabilities, and pack creation/installation improvements.

Changes:

Adds URL spidering support (core + MCP + REST API) and supporting utilities (raw fetch + link extraction).
Expands pack functionality (create from sources, gzip support, concurrent install) and improves bulk selector/date filtering & config caching.
Updates docs and GitHub workflows (merge gate, release-please CI triggering, expanded CI triggers).

Reviewed changes

Copilot reviewed 65 out of 66 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit/webhooks.test.ts	Updates async flushing expectations; adds corrupted JSON coverage for webhook events.
tests/unit/update-document.test.ts	Uses fake timers for deterministic `updated_at` testing.
tests/unit/spider.test.ts	Adds coverage for new spider engine behavior and limits.
tests/unit/saved-searches.test.ts	Adds corrupted JSON coverage for saved search filters.
tests/unit/reporter.test.ts	Adds tests for new CLI reporter behavior.
tests/unit/parsers.test.ts	Adds HTML parser coverage and supported extensions assertions.
tests/unit/packs.test.ts	Adds extensive coverage for pack-from-source, gzip, batching, and concurrency.
tests/unit/mcp-server.test.ts	Adds MCP error helper tests and integration-style tool logic checks.
tests/unit/link-extractor.test.ts	Adds coverage for the new HTML link extractor.
tests/unit/http-utils.test.ts	Updates retry-count expectation.
tests/unit/connectors-config.test.ts	Adds corrupted JSON coverage for connector config loading.
tests/unit/config.test.ts	Adds/uses config cache invalidation and caching behavior assertions.
tests/unit/bulk.test.ts	Adds date filter tests and updates negative-limit behavior.
src/providers/local.ts	Replaces `any` with typed pipeline interface for local embeddings.
src/mcp/server.ts	Adds spidering options to submit-document; adds passthrough RAG mode handling; factors error helpers out.
src/mcp/errors.ts	Introduces shared MCP errorResponse/withErrorHandling helpers.
src/core/webhooks.ts	Hardens webhook row JSON parsing and narrows SELECT columns.
src/core/versioning.ts	Adds logging on corrupted version metadata JSON.
src/core/url-fetcher.ts	Adds per-request TLS dispatcher support and new `fetchRaw()` API.
src/core/tags.ts	Adds batch tag fetch helper; exports tokenizer; adds DB-less tag suggestion helper.
src/core/spider.ts	Adds BFS spider engine with robots.txt, limits, and filtering.
src/core/search.ts	Optimizes minRating filter to use pre-joined avg rating subquery.
src/core/saved-searches.ts	Hardens filters JSON parsing with logging and fallback.
src/core/reindex.ts	Delegates vector table creation to schema helper.
src/core/rag.ts	Adds passthrough mode + context-only retrieval API.
src/core/parsers/index.ts	Registers new HTML parser.
src/core/parsers/html.ts	Adds HTML-to-markdown parser with basic tag ignoring.
src/core/packs.ts	Adds gzip support, pack-from-source, concurrent install batching, and streaming chunking.
src/core/link-extractor.ts	Adds fast non-regex `<a href>` extraction and normalization.
src/core/indexing.ts	Adds dedup behavior modes and makes embedding insert failures transactional (non-vec errors).
src/core/export.ts	Improves debug logging when optional tables are missing.
src/core/documents.ts	Adds date filters to listDocuments and makes update timestamps fake-timer friendly.
src/core/bulk.ts	Pushes date filtering into SQL; uses batch tag lookup helper.
src/core/analytics.ts	Improves debug logging when analytics table is absent.
src/connectors/slack.ts	Improves debug logging when optional tables are absent.
src/connectors/obsidian.ts	Replaces ad-hoc YAML parsing with `js-yaml` + date normalization.
src/connectors/index.ts	Adds JSON corruption handling for connector config loads; improves debug logging for optional tables.
src/connectors/http-utils.ts	Switches to per-request undici dispatcher for TLS bypass; adjusts retry loop semantics.
src/connectors/confluence.ts	Removes hard page cap and replaces self-closing macro stripping regex with indexOf-based parsing.
src/config.ts	Adds passthrough provider option and introduces a 30s config cache w/ invalidation.
src/cli/reporter.ts	Adds pretty CLI reporter with progress rendering and silent mode.
src/cli/index.ts	Adds `-v` shorthand; integrates reporter and new pack options; changes default CLI logging behavior.
src/api/routes.ts	Adds REST URL spidering mode and maps FetchError to 502.
src/api/openapi.ts	Documents URL spidering request/response in OpenAPI.
package.json	Updates dependencies (better-sqlite3) and adds undici.
docs/reference/rest-api.md	Expands and reorganizes REST API reference; adds examples.
docs/reference/mcp-tools.md	Expands MCP tool documentation list and parameters.
docs/reference/configuration.md	Updates config docs for passthrough and additional keys.
docs/reference/cli.md	Updates supported `config set` keys list.
docs/guide/mcp-setup.md	Expands tool list and categories.
docs/guide/getting-started.md	Updates Node requirement and adds dashboard/API guidance.
docs/guide/configuration.md	Documents passthrough provider behavior.
agents.md	Updates architecture map + security patterns guidance.
README.md	Updates MCP tools and REST endpoints list; adds webhooks section; updates license reference.
.github/workflows/sdk-python.yml	Updates workflow action versions.
.github/workflows/sdk-go.yml	Updates workflow action versions.
.github/workflows/release.yml	Updates workflow action versions.
.github/workflows/release-python.yml	Updates workflow action versions.
.github/workflows/release-please.yml	Uses PAT for release-please PRs; updates workflow action versions.
.github/workflows/merge-gate.yml	Adds merge gate enforcing source branch policy for PRs to main.
.github/workflows/docker.yml	Expands triggers to development; updates workflow action versions.
.github/workflows/codeql.yml	Expands triggers to development; updates workflow action versions.
.github/workflows/ci.yml	Expands triggers to development; updates workflow action versions and artifact upload action.
.github/dependabot.yml	Targets dependabot PRs to `development`.
.github/copilot-instructions.md	Documents branch workflow requirement (PRs target development).

Copilot · 2026-03-05T02:46:58Z

src/core/tags.ts

+export function getDocumentTagsBatch(
+  db: Database.Database,
+  documentIds: string[],
+): Map<string, Tag[]> {
+  if (documentIds.length === 0) return new Map();
+  const placeholders = documentIds.map(() => "?").join(", ");
+  const rows = db
+    .prepare(
+      `SELECT dt.document_id, t.id, t.name, t.created_at
+       FROM tags t
+       JOIN document_tags dt ON dt.tag_id = t.id
+       WHERE dt.document_id IN (${placeholders})
+       ORDER BY t.name`,
+    )
+    .all(...documentIds) as Array<{


getDocumentTagsBatch() builds a single IN (...) query with one placeholder per documentId. Since resolveSelector() allows up to MAX_BATCH_SIZE = 1000, this can exceed SQLite's parameter limit (commonly 999) and throw at runtime for large batches. Consider chunking documentIds into <=999-sized groups (and merging results), or lowering the effective limit to stay within the parameter cap.

Copilot · 2026-03-05T02:46:58Z

src/core/packs.ts

+              try {
+                const vecBuffer = Buffer.from(new Float32Array(embedding).buffer);
+                insertEmbedding.run(chunkId, vecBuffer);
+              } catch (err) {
+                const message = err instanceof Error ? err.message : String(err);
+                if (!message.includes("no such table")) {
+                  log.warn({ chunkId, err }, "Failed to insert vector embedding");
+                }
+              }


Within the batch transaction, embedding insert failures are swallowed for any error other than "no such table". That can commit documents/chunks without embeddings (making them invisible to semantic search) and hide data corruption/constraint issues. Recommend re-throwing non-"no such table" errors (to roll back the transaction) and counting the whole batch as failed, similar to indexDocument()’s behavior.

Copilot · 2026-03-05T02:46:58Z

src/core/packs.ts

+    if (stat.isDirectory()) {
+      allFiles.push(...collectFiles(resolved, resolved, recursive, extensions, excludePatterns));
+    } else if (stat.isFile()) {
+      allFiles.push(resolved);
+    } else {


createPackFromSource() applies the extensions filter when walking directories, but a --from source that is an explicit file path is always added to allFiles regardless of its extension. This means --extensions can be bypassed by passing a file directly. Consider enforcing the extension allowlist (or at least getParserForFile) when stat.isFile() is true before adding it.

Copilot · 2026-03-05T02:46:59Z

src/core/packs.ts

+  for (const entry of entries) {
+    const fullPath = pathJoin(dir, entry);
+    const rel = relative(rootDir, fullPath);
+
+    // Check exclude patterns
+    if (excludePatterns.some((p) => matchesExcludePattern(rel, p))) {
+      continue;


collectFiles() computes rel via path.relative(), which uses platform-specific separators (e.g. \ on Windows). Since matchesExcludePattern() and examples like assets/** assume /, excludes may not work on Windows. Normalize rel to POSIX-style (e.g. replace \ with /) before matching patterns to keep behavior consistent across platforms.

Copilot · 2026-03-05T02:46:59Z

src/connectors/http-utils.ts

+  for (let attempt = 0; attempt <= maxRetries; attempt++) {
+    const fetchOptions = {
+      ...(options ?? {}),
+      ...(dispatcher ? { dispatcher: dispatcher as unknown } : {}),
+    } as RequestInit;
+    const response = await fetch(url, fetchOptions);

-      const response = await fetch(url, {
-        ...options,
-        signal: combinedSignal,
-      });
-
-      if (response.status === 429 || (response.status >= 500 && response.status < 600)) {
-        if (attempt >= maxRetries - 1) {
-          const body = await response.text().catch(() => "");
-          throw new FetchError(`HTTP ${response.status} after ${maxRetries} attempts: ${body}`);
-        }
+    if (response.status === 429 || (response.status >= 500 && response.status < 600)) {
+      if (attempt >= maxRetries) {
+        const body = await response.text().catch(() => "");
+        throw new FetchError(`HTTP ${response.status} after ${maxRetries + 1} attempts: ${body}`);
+      }

-        let delayMs = baseDelay * 2 ** attempt;
-        if (response.status === 429) {
-          const retryAfter = response.headers.get("Retry-After");
-          if (retryAfter) {
-            const parsed = Number(retryAfter);
-            if (!Number.isNaN(parsed)) {
-              delayMs = parsed * 1000;
-            }
+      let delayMs = baseDelay * 2 ** attempt;
+      if (response.status === 429) {
+        const retryAfter = response.headers.get("Retry-After");
+        if (retryAfter) {
+          const parsed = Number(retryAfter);
+          if (!Number.isNaN(parsed)) {
+            delayMs = parsed * 1000;
          }
        }
-
-        log.warn(
-          { status: response.status, attempt: attempt + 1, delayMs },
-          "Retrying after transient error",
-        );
-        await new Promise((resolve) => setTimeout(resolve, delayMs));
-        continue;
      }

-      return response;
+      log.warn(
+        { status: response.status, attempt: attempt + 1, delayMs },
+        "Retrying after transient error",
+      );
+      await new Promise((resolve) => setTimeout(resolve, delayMs));
+      continue;


fetchWithRetry() no longer enforces any timeout or composes options.signal with a per-attempt timeout. Since fetch() has no default timeout, a hung connection can block a connector sync indefinitely (and retries won’t help). Consider restoring a per-attempt timeout (e.g. AbortSignal.timeout(...)) and combining it with options.signal so callers can still cancel.

RobertLD and others added 20 commits March 3, 2026 13:25

Claude/fix issue 331 s1qzu (#338)

3784409

Copilot AI review requested due to automatic review settings March 5, 2026 02:40

Copilot started reviewing on behalf of RobertLD March 5, 2026 02:41 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

RobertLD closed this Mar 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: merge development into main#349

chore: merge development into main#349
RobertLD wants to merge 20 commits intomainfrom
development

RobertLD commented Mar 5, 2026

Uh oh!

vercel bot commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

RobertLD commented Mar 5, 2026

Uh oh!

vercel bot commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants