chore: bring main up to date with development for v1.3.0#353
Merged
Conversation
* chore: add development branch workflow - Add merge-gate.yml: enforces only 'development' can merge into main - Update CI/CodeQL/Docker workflows to run on both main and development - Update dependabot.yml: target-branch set to development for all ecosystems - Update copilot-instructions.md: document branch workflow convention - Rulesets configured: Main (requires merge-gate + squash-only), Development (requires CI status checks + PR) - Default branch set to development - All open PRs retargeted to development Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip Vercel preview deployments on non-main branches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger check refresh --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: comprehensive audit fixes — security, performance, resilience, API hardening Addresses findings from issue #314: - SSRF protection for webhook URLs (CRITICAL) - Scrub secrets from exports - Stored XSS prevention on document URL - O(n²) and N+1 fixes in bulk operations - Rate limit cache eviction improvement - SSE backpressure handling - Replace raw Error() with typed errors - Fetch timeouts on all network calls - Input validation on API parameters - Search query length limit - Silent catch block logging - DNS rebinding check fix - N+1 in Slack user resolution - Pagination on webhook/search list endpoints Closes #314 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: fix SSE listener leak, preserve caller signals, add SSRF validation to webhook test, chunk SQL params, use dynamic import in test - SSE backpressure: create single disconnect promise, race against drain (no listener accumulation) - http-utils.ts/onenote.ts: use AbortSignal.any() to combine caller signal with timeout - Webhook test endpoint: validate URL with validateWebhookUrlSsrf before fetch - bulk.ts: chunk IN clause to 999 params max (SQLite limit) - webhooks.test.ts: dynamic import after vi.mock() for deterministic DNS mocking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: consolidate and fix CI/CD workflows - Merge lint + typecheck into single job (saves one npm ci) - Add concurrency groups to ci, docker, codeql (cancel stale runs) - Add dependency-review-action on PRs (block vulnerable deps) - Add workflow_call trigger to ci.yml for reusability - Remove duplicate npm publish from release.yml (release-please owns it) - Fix SDK paths: sdk-go/ → sdk/go/, sdk-python/ → sdk/python/ - Fix Dependabot paths to match actual SDK directories - Add github-actions ecosystem to Dependabot (keep actions up to date) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb48187. * feat: add --from option to pack create for folder/URL sources Adds createPackFromSource() that builds packs directly from local folders, files, or URLs without requiring database interaction. CLI: libscope pack create --name my-pack --from ~/docs/ [--extensions .md,.html] [--exclude pattern] [--no-recursive] Features: - Walks directories recursively using registered parsers - Fetches URLs via fetchAndConvert - Supports extension filtering, exclude patterns, progress callback - Multiple --from sources supported - Output format identical to DB export (pack install works unchanged) Closes #328 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: fix prettier formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add gzip support for pack files (.json.gz) Pack files can now be compressed with gzip for smaller distribution: - writePackFile/readPackFile auto-detect gzip by extension or magic bytes - installPack accepts both .json and .json.gz files - createPackFromSource defaults to .json.gz output (source packs can be large) - createPack (DB export) still defaults to .json - Auto-detects gzip by magic bytes even if extension is .json 5 new tests covering gzip write, install, magic byte detection, and round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add progress logging and fix dedup handling in pack install - Log each document as it's indexed so large installs show progress - Change pack install to use dedup: 'skip' for graceful duplicate handling - Make title+content_length dedup check respect the dedup mode setting (previously it always threw ValidationError regardless of dedup mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: auto-generate tags during pack creation and apply on install - Export tokenize() and add suggestTagsFromText() in tags.ts for DB-free tag generation - createPackFromSource() now auto-generates tags per document via TF-IDF - installPack() applies doc.tags via addTagsToDocument() after indexing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb48187. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bumps [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) from 9.1.2 to 10.1.8. - [Release notes](https://github.com/prettier/eslint-config-prettier/releases) - [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md) - [Commits](https://github.com/prettier/eslint-config-prettier/commits/v10.1.8) --- updated-dependencies: - dependency-name: eslint-config-prettier dependency-version: 10.1.8 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the minor-and-patch group with 1 update: [lint-staged](https://github.com/lint-staged/lint-staged). Updates `lint-staged` from 16.3.1 to 16.3.2 - [Release notes](https://github.com/lint-staged/lint-staged/releases) - [Changelog](https://github.com/lint-staged/lint-staged/blob/main/CHANGELOG.md) - [Commits](lint-staged/lint-staged@v16.3.1...v16.3.2) --- updated-dependencies: - dependency-name: lint-staged dependency-version: 16.3.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the actions group with 5 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `6` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `7` | | [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` | | [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` | Updates `actions/checkout` from 4 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v6) Updates `actions/setup-node` from 4 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v6) Updates `actions/upload-artifact` from 4 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v7) Updates `actions/setup-python` from 5 to 6 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) Updates `actions/setup-go` from 5 to 6 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 22.19.13 to 25.3.3. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.3.3 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) from 11.10.0 to 12.6.2. - [Release notes](https://github.com/WiseLibs/better-sqlite3/releases) - [Commits](WiseLibs/better-sqlite3@v11.10.0...v12.6.2) --- updated-dependencies: - dependency-name: better-sqlite3 dependency-version: 12.6.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [eslint](https://github.com/eslint/eslint) from 9.39.3 to 10.0.2. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v9.39.3...v10.0.2) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert DeRienzo <rderienzo@voloridge.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: add passthrough LLM mode for ask-question tool
Adds llm.provider = "passthrough" so the ask-question MCP tool returns
retrieved context chunks directly to the calling LLM instead of requiring
a separate OpenAI/Ollama provider. This is the natural design for MCP tools
where the client already has an LLM (e.g. Claude Code).
- config.ts: add "passthrough" to llm.provider union type and env var handling
- rag.ts: add isPassthroughMode() helper and getContextForQuestion() which
retrieves and formats context without an LLM call
- mcp/server.ts: ask-question checks passthrough first and returns context
directly; falls through to existing LLM path otherwise
Enable via config: { "llm": { "provider": "passthrough" } }
Enable via env var: LIBSCOPE_LLM_PROVIDER=passthrough
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: format config.ts and include passthrough in provider override
- Reformat long if-condition to satisfy prettier (printWidth: 100)
- Fix logic bug: passthrough provider was checked in outer condition but
not spread into overrides.llm.provider
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: address 9 audit findings from issue #332 Security - middleware: use timingSafeEqual for API key comparison (#2) - url-fetcher, http-utils: replace process-wide NODE_TLS_REJECT_UNAUTHORIZED mutation with per-request undici Agent to eliminate TLS race condition (#1) Bugs - indexing: re-throw unexpected embedding errors so transaction rolls back instead of silently committing chunks with no vector (#3) - search: replace correlated minRating subquery with avg_r.avg_rating from the pre-joined aggregate in FTS and LIKE search paths (#4) Performance - bulk: replace O(n²) docs.find() loops with pre-built Map; replace per-document getDocumentTags() calls with a single getDocumentTagsBatch() query (#5) - config: add 30-second TTL cache to loadConfig() so disk reads are not repeated on every request (#6) Code quality - routes: check res.write() return value to handle SSE backpressure (#7) - reindex: delegate to schema.createVectorTable() instead of duplicating the vec0 DDL inline (#8) - obsidian: replace hand-rolled parseSimpleYaml() with js-yaml, normalise Date objects back to ISO-8601 strings (#9) Docs - agents.md: expand architecture tree to include src/api/ and src/connectors/; add Security Patterns section with correct undici examples - CONTRIBUTING.md: fix check-suite command (npm test → npm run test:coverage) and correct coverage threshold (80% → actual 75%/74%) Tests - bulk.test: add dateFrom/dateTo filter coverage - config.test: add cache-hit test; call invalidateConfigCache() before env-var tests so TTL cache doesn't return stale results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused warnIfTlsBypassMissing function Dead code after conflict resolution chose the per-request undici Agent approach (which doesn't need a warning about NODE_TLS_REJECT_UNAUTHORIZED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update tests for config cache and retry semantics - Add invalidateConfigCache() before loadConfig() in 4 env-override tests that were failing because the 30s TTL cache introduced in the config module was returning stale results from the previous test's cache entry - Update http-utils retry assertion: maxRetries=2 means 1 initial + 2 retries = 3 total calls (loop is attempt <= maxRetries) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
… (#336) - Add `src/cli/reporter.ts`: PrettyReporter (ANSI colors + \r progress bar), SilentReporter (no-op for verbose/JSON mode), `isVerbose()` and `createReporter()` factory. `LIBSCOPE_VERBOSE=1` env var alternative. - Update `setupLogging` to default to "silent" in CLI mode (pretty reporter handles user-facing output). Verbose/`--log-level` flags still route to structured JSON pino logs. Fix duplicate `initLogger` calls in onenote connect/disconnect commands to use `setupLogging` consistently. - Update `installPack` in `packs.ts` to support batch embedding and progress reporting: - New `InstallOptions` interface with `batchSize`, `resumeFrom`, `onProgress` fields - Batch documents: chunk all → single `provider.embedBatch` call per batch → single SQLite transaction per batch (avoids N embedding calls) - `resumeFrom` skips the first N documents (enables partial install resume after failure) - `InstallResult` now includes `errors` count - Add `--batch-size` and `--resume-from` CLI options to `pack install` - Tests: `tests/unit/reporter.test.ts` (17 tests covering PrettyReporter, SilentReporter, isVerbose, env var detection); extended `tests/unit/packs.test.ts` with 7 new tests for progress callbacks, batch efficiency, resumeFrom, embedBatch failure handling. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…dates (#337) Bumps the npm_and_yarn group with 2 updates in the / directory: [@hono/node-server](https://github.com/honojs/node-server) and [hono](https://github.com/honojs/hono). Updates `@hono/node-server` from 1.19.9 to 1.19.10 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](honojs/node-server@v1.19.9...v1.19.10) Updates `hono` from 4.12.3 to 4.12.5 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](honojs/hono@v4.12.3...v4.12.5) --- updated-dependencies: - dependency-name: "@hono/node-server" dependency-version: 1.19.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: hono dependency-version: 4.12.5 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (closes #340) **SSRF (CWE-918 — CodeQL alert #28)** Replace the two-step validate-then-fetch approach in url-fetcher.ts with IP-pinned requests using node:http / node:https directly. validateUrl() resolves DNS and checks for private IPs, then the validated IP is passed straight to the TCP connection (hostname: pinnedIp, servername: original hostname for TLS SNI). There is now zero TOCTOU window between validation and the actual network request. The redundant post-fetch DNS rebinding check and the env-var-based TLS bypass wrapper are removed; rejectUnauthorized is now passed directly to the request options. An internal _setRequestImpl hook is exported for unit test injection so tests can stub responses without touching node:http / node:https. Tests are updated accordingly. **ReDoS (CWE-1333 — CodeQL alert #24)** Five regexes in confluence.ts used the pattern [^>]*ac:name="X"[^>]* — two [^>]* quantifiers around a fixed literal. For input that contains a large attribute blob without the target ac:name value, the engine must try all O(n²) splits before concluding no match (catastrophic backtracking). Fix: rewrite the leading [^>]* as (?:(?!ac:name="X")[^>])* — a negative lookahead prevents the quantifier from overlapping with the literal, making backtracking structurally impossible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace remaining polynomial regexes in confluence.ts with indexOf helpers The conflict resolution left the original `/<ac:structured-macro [^>]*>([\s\S]*?) <\/ac:structured-macro>/gi` patterns in place for code blocks and info/tip panels (those were not part of the original security fix diff). These have the same O(n²) backtracking problem: with k opening tags and no closing tags, [\s\S]*? scans O(n - pos) chars per attempt, totalling O(n²). Replace the entire convertConfluenceStorage function with the indexOf-based approach (replaceStructuredMacros / replaceTagPairs / extractTagContent helpers) that eliminates all regex-based tag parsing. Also add removeSelfClosingMacros to handle the self-closing TOC case without regex, since the previous self-closing fix still used a [^>]*ac:name="toc"[^>]* pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…) (#339) * feat: concurrent pack installation and -v verbose shorthand (issue #330) Add concurrent batch embedding to installPack for significant performance improvement on large packs, plus CLI ergonomics improvements. Key changes: - `InstallOptions.concurrency` (default: 4): controls how many embedBatch calls run simultaneously; embedding is I/O-bound so parallelism directly reduces wall-clock installation time - Refactor installPack to pre-chunk all documents upfront, then use a semaphore-based scheduler to run up to `concurrency` embedBatch calls concurrently while inserting completed batches in-order (SQLite requires serialised writes); progress callbacks fire after each batch as before - `pack install --concurrency <n>` CLI flag exposes the new option - `-v` shorthand for `--verbose` on the global program options - Fix transaction install-count tracking: count committed docs accurately without relying on subtract-on-failure arithmetic - Add 6 new tests covering concurrency=1 sequential, concurrency=4 parallel, multiple embedBatch calls per install, concurrency limit enforcement, incremental progress reporting, and partial-failure error counting https://claude.ai/code/session_019hzhbEgV1ysnGmFVXBWzkZ * fix: address all 4 Copilot review comments on PR #339 - Validate batchSize, concurrency, resumeFrom at the start of installPack and throw ValidationError for invalid values (comments 3 & 4). Concurrency <= 0 would silently hang the semaphore indefinitely. - Add CLI lower-bound guard: --concurrency < 1 exits with a user-facing error before ever calling installPack (comment 3). - Lazy chunking: pre-chunking all documents upfront held chunks for the entire pack in memory simultaneously. Batches now store only the raw documents; resolveBatch() chunks on demand right before embedBatch is called, so only one batch's worth of chunks is in memory at a time (comment 2). - Wrap provider.embedBatch() in try/catch so synchronous throws are converted to rejected Promises rather than escaping scheduleNext() and leaving the outer Promise permanently pending (comment 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
- Guard JSON.parse in rowToWebhook with try/catch, default to [] - Guard JSON.parse in rowToSavedSearch with try/catch, default to null - Guard JSON.parse in loadDbConnectorConfig with try/catch, throw ConfigError - Push dateFrom/dateTo filters into SQL WHERE in listDocuments (before LIMIT) - Validate negative limit in resolveSelector, throw ValidationError - Replace manual substring extension parsing with path.extname() in packs.ts - Verified reporter.ts is already tracked on development (no action needed) - Added tests for all fixes (corrupted JSON, SQL-level date filtering, negative limit) Closes #342 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…page limits (#315) (#343) * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) Adds opt-in spidering to URL indexing. A single seed URL can now crawl and index an entire documentation site or wiki section in one call. New files: - src/core/link-extractor.ts: indexOf-based <a href> extraction, relative URL resolution, fragment stripping, dedup, scheme filtering. No regex. - src/core/spider.ts: BFS crawl engine with sameDomain, pathPrefix, excludePatterns (glob), maxPages (hard cap 200), maxDepth (hard cap 5), 10-min total timeout, robots.txt (User-agent: * and libscope), and 1s inter-request delay. Yields SpiderResult per page; returns SpiderStats. - tests/unit/link-extractor.test.ts: 25 tests covering relative resolution, dedup, fragment stripping, scheme filtering, attribute order, edge cases. - tests/unit/spider.test.ts: 20 tests covering BFS order, depth/page limits, domain + path + pattern filtering, cycle detection, robots.txt, partial failure recovery, stats, and abortReason reporting. Modified: - src/core/url-fetcher.ts: adds fetchRaw() export returning raw body + contentType + finalUrl before HTML-to-markdown conversion, so the spider can extract links from HTML before conversion. - src/api/routes.ts: POST /api/v1/documents/url now accepts spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns. Returns { documents, pagesIndexed, pagesCrawled, pagesSkipped, errors, abortReason }. - src/mcp/server.ts: submit-document tool gains spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns parameters. Safety: all fetched URLs pass through the existing SSRF validation in fetchRaw() (DNS resolution, private IP blocking, scheme allowlist). Hard limits (200 pages, depth 5, 10min) cannot be overridden by callers. robots.txt is fetched once per origin and Disallow rules are honoured. Individual page failures do not abort the crawl. Closes #315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve CI lint errors in spider implementation - Remove unnecessary type assertions (routes.ts, mcp/server.ts) — TypeScript already narrows SpiderResult/SpiderStats from the generator - Add explicit return type annotation on mock fetchRaw to satisfy no-unsafe-return rule in spider.test.ts - Replace .resolves.not.toThrow() with a direct assertion — vitest .resolves requires a Promise, not an async function Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address CodeQL security findings in spider/link-extractor link-extractor.ts (CodeQL #30 — incomplete URL scheme check): Replace the enumerated scheme blocklist (javascript:, vbscript:, etc.) with a strict http/https allowlist check on the resolved URL protocol. An allowlist is exhaustive by definition; a blocklist will always miss obscure schemes like vbscript:, blob:, or future additions. spider.ts (CodeQL #31 — incomplete multi-character sanitization): Replace the regex-based tag stripper /<[^>]+>/g in extractTitle() with an indexOf-based stripTags() function. The regex stops at the first > which can be inside a quoted attribute value (e.g. <img alt="a>b">), potentially leaving partial tag content in the extracted title. The new implementation walks quoted attribute values explicitly so no tag content leaks through regardless of its internal structure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address all Copilot review comments on spider PR (#343) - link-extractor: add word-boundary check in extractHref to prevent matching data-href, aria-href (false positives on non-href attributes) - spider: rename pagesIndexed → pagesFetched throughout (SpiderStats interface already used pagesFetched; sync implementation + tests) - spider: per-origin robots.txt cache (Map<origin, Set>) fetched lazily as new origins are encountered during crawl (was seed-only before) - spider: normalize to raw.finalUrl after redirects — visited set, yielded URL, and link-extraction base all use the canonical URL - routes: validate maxPages/maxDepth are finite positive integers - routes: change conditional spread &&-patterns to ternaries - routes: remove inner try/catch for spider fetch errors; add FetchError to top-level handler (consistent with single-URL mode → 502) - mcp/server: replace conditional spreads with explicit if assignments - mcp/server: validate spider=true requires url (throws ValidationError) - openapi: document spider request fields in IndexFromUrlRequest schema, add SpiderResponse schema, update 201 response to oneOf Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix prettier formatting in spider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: comprehensive documentation update for v1.3.0 - README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: allow release-please PRs to pass merge gate and trigger CI Two issues prevented PR #238 from getting CI runs: 1. merge-gate blocked release-please PRs — the gate only allowed 'development' as the source branch, but release-please uses 'release-please--branches--main--components--libscope'. Updated to allow any branch matching 'release-please--*'. 2. CI never ran on the PR — GitHub does not trigger workflows when GITHUB_TOKEN creates a PR (intentional security restriction to prevent infinite loops). Fixed by passing a PAT via secrets.GH_TOKEN to the release-please action so its PR creation triggers CI. Note: requires a 'GH_TOKEN' secret in repo settings — a classic PAT with repo and workflow scopes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
There was a problem hiding this comment.
Pull request overview
This PR merges the development branch into main to prepare for the v1.3.0 release, resolving a pre-existing add/add merge conflict in .github/workflows/merge-gate.yml. It delivers a substantial documentation update reflecting all new features added in v1.3.0, and also fixes the CI pipeline so that release-please-generated PRs can be automatically merged into main.
Changes:
- Updates the
merge-gate.ymlCI rule to allowrelease-please--*branches (in addition todevelopment) to merge intomain, breaking the circular dependency that would otherwise block the automated release PR. - Adds a PAT-based token to
release-please.ymlso the release PR it creates triggers CI workflows (the defaultGITHUB_TOKENcannot). - Expands documentation across multiple files to cover new v1.3.0 features: document update, cross-reference links, tag suggestions, webhooks, saved searches, analytics/connector endpoints, the
passthroughLLM provider, and the web dashboard.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/merge-gate.yml |
Allows release-please--* branches to pass the enforce-source-branch check |
.github/workflows/release-please.yml |
Uses a PAT token so the created release PR triggers CI |
docs/reference/rest-api.md |
Reorganises endpoint table by section and adds new v1.3.0 endpoints |
docs/reference/mcp-tools.md |
Documents new MCP tools: update-document, suggest-tags, link-documents, get-document-links, delete-link |
docs/reference/configuration.md |
Adds passthrough LLM provider and llm.ollamaUrl config key |
docs/reference/cli.md |
Expands the config set supported keys list |
docs/guide/mcp-setup.md |
Updates available-tools overview to cover all 26 tools by category |
docs/guide/getting-started.md |
Adds Web Dashboard, Organize/Annotate, and REST API sections; bumps Node.js minimum to 20 |
docs/guide/configuration.md |
Adds passthrough LLM provider description |
README.md |
Updates MCP tool table, REST API endpoint table, CLI cheat-sheet, webhook documentation, and license from MIT to BUSL 1.1 |
| - **`ask-question`** — RAG Q&A with synthesized answers | ||
| - **`submit-document`** — index new content (by text or URL) | ||
| - **`list-topics`** — browse what's in the knowledge base | ||
| Once connected, your AI assistant gets access to all 26 of LibScope's MCP tools. See the [MCP Tools Reference](/reference/mcp-tools) for full parameter details. |
There was a problem hiding this comment.
The page states "all 26 of LibScope's MCP tools" but only lists 25 tools. The delete-saved-search tool (documented in docs/reference/mcp-tools.md) is not included in the available tools overview. Either add delete-saved-search to the Saved Searches section, or change the count to "25".
| | ------ | -------------------------------- | ---------------------------------- | | ||
| | `GET` | `/api/v1/health` | Health check with document count | | ||
| | `GET` | `/api/v1/stats` | Usage statistics | | ||
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps| |
There was a problem hiding this comment.
There is a missing space before the closing pipe character at the end of the table cell: knowledge gaps| should be knowledge gaps |. While most Markdown renderers tolerate this, it is inconsistent with every other table row in this file.
Suggested change
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps| | |
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps | |
| | `GET/POST` | `/api/v1/webhooks` | List or create webhooks | | ||
| | `DELETE` | `/api/v1/webhooks/:id` | Delete a webhook | | ||
| | `POST` | `/api/v1/webhooks/:id/test` | Send a test ping to a webhook | | ||
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps| |
There was a problem hiding this comment.
Same missing space before the closing pipe on the "Search analytics and knowledge gaps" table cell: knowledge gaps| should be knowledge gaps |. All other rows in this table and in the file have a trailing space before the closing pipe.
Suggested change
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps| | |
| | `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps | |
RobertLD
added a commit
that referenced
this pull request
Mar 6, 2026
) * Prepare for release (#345) * chore: add development branch workflow (#327) * chore: add development branch workflow - Add merge-gate.yml: enforces only 'development' can merge into main - Update CI/CodeQL/Docker workflows to run on both main and development - Update dependabot.yml: target-branch set to development for all ecosystems - Update copilot-instructions.md: document branch workflow convention - Rulesets configured: Main (requires merge-gate + squash-only), Development (requires CI status checks + PR) - Default branch set to development - All open PRs retargeted to development Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip Vercel preview deployments on non-main branches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger check refresh --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: create-pack from local folder or URL sources (#329) * fix: comprehensive audit fixes — security, performance, resilience, API hardening Addresses findings from issue #314: - SSRF protection for webhook URLs (CRITICAL) - Scrub secrets from exports - Stored XSS prevention on document URL - O(n²) and N+1 fixes in bulk operations - Rate limit cache eviction improvement - SSE backpressure handling - Replace raw Error() with typed errors - Fetch timeouts on all network calls - Input validation on API parameters - Search query length limit - Silent catch block logging - DNS rebinding check fix - N+1 in Slack user resolution - Pagination on webhook/search list endpoints Closes #314 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: fix SSE listener leak, preserve caller signals, add SSRF validation to webhook test, chunk SQL params, use dynamic import in test - SSE backpressure: create single disconnect promise, race against drain (no listener accumulation) - http-utils.ts/onenote.ts: use AbortSignal.any() to combine caller signal with timeout - Webhook test endpoint: validate URL with validateWebhookUrlSsrf before fetch - bulk.ts: chunk IN clause to 999 params max (SQLite limit) - webhooks.test.ts: dynamic import after vi.mock() for deterministic DNS mocking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: consolidate and fix CI/CD workflows - Merge lint + typecheck into single job (saves one npm ci) - Add concurrency groups to ci, docker, codeql (cancel stale runs) - Add dependency-review-action on PRs (block vulnerable deps) - Add workflow_call trigger to ci.yml for reusability - Remove duplicate npm publish from release.yml (release-please owns it) - Fix SDK paths: sdk-go/ → sdk/go/, sdk-python/ → sdk/python/ - Fix Dependabot paths to match actual SDK directories - Add github-actions ecosystem to Dependabot (keep actions up to date) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb481870c883f77278291b72245b1ca0b890a78c. * feat: add --from option to pack create for folder/URL sources Adds createPackFromSource() that builds packs directly from local folders, files, or URLs without requiring database interaction. CLI: libscope pack create --name my-pack --from ~/docs/ [--extensions .md,.html] [--exclude pattern] [--no-recursive] Features: - Walks directories recursively using registered parsers - Fetches URLs via fetchAndConvert - Supports extension filtering, exclude patterns, progress callback - Multiple --from sources supported - Output format identical to DB export (pack install works unchanged) Closes #328 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: fix prettier formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add gzip support for pack files (.json.gz) Pack files can now be compressed with gzip for smaller distribution: - writePackFile/readPackFile auto-detect gzip by extension or magic bytes - installPack accepts both .json and .json.gz files - createPackFromSource defaults to .json.gz output (source packs can be large) - createPack (DB export) still defaults to .json - Auto-detects gzip by magic bytes even if extension is .json 5 new tests covering gzip write, install, magic byte detection, and round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add progress logging and fix dedup handling in pack install - Log each document as it's indexed so large installs show progress - Change pack install to use dedup: 'skip' for graceful duplicate handling - Make title+content_length dedup check respect the dedup mode setting (previously it always threw ValidationError regardless of dedup mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: auto-generate tags during pack creation and apply on install - Export tokenize() and add suggestTagsFromText() in tags.ts for DB-free tag generation - createPackFromSource() now auto-generates tags per document via TF-IDF - installPack() applies doc.tags via addTagsToDocument() after indexing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing (#318) * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb481870c883f77278291b72245b1ca0b890a78c. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build(deps-dev): Bump eslint-config-prettier from 9.1.2 to 10.1.8 (#325) Bumps [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) from 9.1.2 to 10.1.8. - [Release notes](https://github.com/prettier/eslint-config-prettier/releases) - [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md) - [Commits](https://github.com/prettier/eslint-config-prettier/commits/v10.1.8) --- updated-dependencies: - dependency-name: eslint-config-prettier dependency-version: 10.1.8 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump lint-staged from 16.3.1 to 16.3.2 Bumps the minor-and-patch group with 1 update: [lint-staged](https://github.com/lint-staged/lint-staged). Updates `lint-staged` from 16.3.1 to 16.3.2 - [Release notes](https://github.com/lint-staged/lint-staged/releases) - [Changelog](https://github.com/lint-staged/lint-staged/blob/main/CHANGELOG.md) - [Commits](https://github.com/lint-staged/lint-staged/compare/v16.3.1...v16.3.2) --- updated-dependencies: - dependency-name: lint-staged dependency-version: 16.3.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Bump the actions group with 5 updates Bumps the actions group with 5 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `6` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `7` | | [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` | | [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` | Updates `actions/checkout` from 4 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) Updates `actions/setup-node` from 4 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v4...v6) Updates `actions/upload-artifact` from 4 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) Updates `actions/setup-python` from 5 to 6 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v6) Updates `actions/setup-go` from 5 to 6 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump @types/node from 22.19.13 to 25.3.3 Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 22.19.13 to 25.3.3. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.3.3 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Bump better-sqlite3 from 11.10.0 to 12.6.2 Bumps [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) from 11.10.0 to 12.6.2. - [Release notes](https://github.com/WiseLibs/better-sqlite3/releases) - [Commits](https://github.com/WiseLibs/better-sqlite3/compare/v11.10.0...v12.6.2) --- updated-dependencies: - dependency-name: better-sqlite3 dependency-version: 12.6.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump eslint from 9.39.3 to 10.0.2 Bumps [eslint](https://github.com/eslint/eslint) from 9.39.3 to 10.0.2. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](https://github.com/eslint/eslint/compare/v9.39.3...v10.0.2) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert DeRienzo <rderienzo@voloridge.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add passthrough LLM mode for ask-question tool (#335) * feat: add passthrough LLM mode for ask-question tool Adds llm.provider = "passthrough" so the ask-question MCP tool returns retrieved context chunks directly to the calling LLM instead of requiring a separate OpenAI/Ollama provider. This is the natural design for MCP tools where the client already has an LLM (e.g. Claude Code). - config.ts: add "passthrough" to llm.provider union type and env var handling - rag.ts: add isPassthroughMode() helper and getContextForQuestion() which retrieves and formats context without an LLM call - mcp/server.ts: ask-question checks passthrough first and returns context directly; falls through to existing LLM path otherwise Enable via config: { "llm": { "provider": "passthrough" } } Enable via env var: LIBSCOPE_LLM_PROVIDER=passthrough Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: format config.ts and include passthrough in provider override - Reformat long if-condition to satisfy prettier (printWidth: 100) - Fix logic bug: passthrough provider was checked in outer condition but not spread into overrides.llm.provider Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address 9 audit findings from issue #332 (#333) * fix: address 9 audit findings from issue #332 Security - middleware: use timingSafeEqual for API key comparison (#2) - url-fetcher, http-utils: replace process-wide NODE_TLS_REJECT_UNAUTHORIZED mutation with per-request undici Agent to eliminate TLS race condition (#1) Bugs - indexing: re-throw unexpected embedding errors so transaction rolls back instead of silently committing chunks with no vector (#3) - search: replace correlated minRating subquery with avg_r.avg_rating from the pre-joined aggregate in FTS and LIKE search paths (#4) Performance - bulk: replace O(n²) docs.find() loops with pre-built Map; replace per-document getDocumentTags() calls with a single getDocumentTagsBatch() query (#5) - config: add 30-second TTL cache to loadConfig() so disk reads are not repeated on every request (#6) Code quality - routes: check res.write() return value to handle SSE backpressure (#7) - reindex: delegate to schema.createVectorTable() instead of duplicating the vec0 DDL inline (#8) - obsidian: replace hand-rolled parseSimpleYaml() with js-yaml, normalise Date objects back to ISO-8601 strings (#9) Docs - agents.md: expand architecture tree to include src/api/ and src/connectors/; add Security Patterns section with correct undici examples - CONTRIBUTING.md: fix check-suite command (npm test → npm run test:coverage) and correct coverage threshold (80% → actual 75%/74%) Tests - bulk.test: add dateFrom/dateTo filter coverage - config.test: add cache-hit test; call invalidateConfigCache() before env-var tests so TTL cache doesn't return stale results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused warnIfTlsBypassMissing function Dead code after conflict resolution chose the per-request undici Agent approach (which doesn't need a warning about NODE_TLS_REJECT_UNAUTHORIZED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update tests for config cache and retry semantics - Add invalidateConfigCache() before loadConfig() in 4 env-override tests that were failing because the 30s TTL cache introduced in the config module was returning stale results from the previous test's cache entry - Update http-utils retry assertion: maxRetries=2 means 1 initial + 2 retries = 3 total calls (loop is attempt <= maxRetries) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: CLI logging improvements and pack installation performance (#330) (#336) - Add `src/cli/reporter.ts`: PrettyReporter (ANSI colors + \r progress bar), SilentReporter (no-op for verbose/JSON mode), `isVerbose()` and `createReporter()` factory. `LIBSCOPE_VERBOSE=1` env var alternative. - Update `setupLogging` to default to "silent" in CLI mode (pretty reporter handles user-facing output). Verbose/`--log-level` flags still route to structured JSON pino logs. Fix duplicate `initLogger` calls in onenote connect/disconnect commands to use `setupLogging` consistently. - Update `installPack` in `packs.ts` to support batch embedding and progress reporting: - New `InstallOptions` interface with `batchSize`, `resumeFrom`, `onProgress` fields - Batch documents: chunk all → single `provider.embedBatch` call per batch → single SQLite transaction per batch (avoids N embedding calls) - `resumeFrom` skips the first N documents (enables partial install resume after failure) - `InstallResult` now includes `errors` count - Add `--batch-size` and `--resume-from` CLI options to `pack install` - Tests: `tests/unit/reporter.test.ts` (17 tests covering PrettyReporter, SilentReporter, isVerbose, env var detection); extended `tests/unit/packs.test.ts` with 7 new tests for progress callbacks, batch efficiency, resumeFrom, embedBatch failure handling. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Claude/fix issue 331 s1qzu (#338) * build(deps): Bump the npm_and_yarn group across 1 directory with 2 updates (#337) Bumps the npm_and_yarn group with 2 updates in the / directory: [@hono/node-server](https://github.com/honojs/node-server) and [hono](https://github.com/honojs/hono). Updates `@hono/node-server` from 1.19.9 to 1.19.10 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](https://github.com/honojs/node-server/compare/v1.19.9...v1.19.10) Updates `hono` from 4.12.3 to 4.12.5 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](https://github.com/honojs/hono/compare/v4.12.3...v4.12.5) --- updated-dependencies: - dependency-name: "@hono/node-server" dependency-version: 1.19.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: hono dependency-version: 4.12.5 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (#341) * fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (closes #340) **SSRF (CWE-918 — CodeQL alert #28)** Replace the two-step validate-then-fetch approach in url-fetcher.ts with IP-pinned requests using node:http / node:https directly. validateUrl() resolves DNS and checks for private IPs, then the validated IP is passed straight to the TCP connection (hostname: pinnedIp, servername: original hostname for TLS SNI). There is now zero TOCTOU window between validation and the actual network request. The redundant post-fetch DNS rebinding check and the env-var-based TLS bypass wrapper are removed; rejectUnauthorized is now passed directly to the request options. An internal _setRequestImpl hook is exported for unit test injection so tests can stub responses without touching node:http / node:https. Tests are updated accordingly. **ReDoS (CWE-1333 — CodeQL alert #24)** Five regexes in confluence.ts used the pattern [^>]*ac:name="X"[^>]* — two [^>]* quantifiers around a fixed literal. For input that contains a large attribute blob without the target ac:name value, the engine must try all O(n²) splits before concluding no match (catastrophic backtracking). Fix: rewrite the leading [^>]* as (?:(?!ac:name="X")[^>])* — a negative lookahead prevents the quantifier from overlapping with the literal, making backtracking structurally impossible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace remaining polynomial regexes in confluence.ts with indexOf helpers The conflict resolution left the original `/<ac:structured-macro [^>]*>([\s\S]*?) <\/ac:structured-macro>/gi` patterns in place for code blocks and info/tip panels (those were not part of the original security fix diff). These have the same O(n²) backtracking problem: with k opening tags and no closing tags, [\s\S]*? scans O(n - pos) chars per attempt, totalling O(n²). Replace the entire convertConfluenceStorage function with the indexOf-based approach (replaceStructuredMacros / replaceTagPairs / extractTagContent helpers) that eliminates all regex-based tag parsing. Also add removeSelfClosingMacros to handle the self-closing TOC case without regex, since the previous self-closing fix still used a [^>]*ac:name="toc"[^>]* pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: concurrent pack installation and -v verbose shorthand (issue #330) (#339) * feat: concurrent pack installation and -v verbose shorthand (issue #330) Add concurrent batch embedding to installPack for significant performance improvement on large packs, plus CLI ergonomics improvements. Key changes: - `InstallOptions.concurrency` (default: 4): controls how many embedBatch calls run simultaneously; embedding is I/O-bound so parallelism directly reduces wall-clock installation time - Refactor installPack to pre-chunk all documents upfront, then use a semaphore-based scheduler to run up to `concurrency` embedBatch calls concurrently while inserting completed batches in-order (SQLite requires serialised writes); progress callbacks fire after each batch as before - `pack install --concurrency <n>` CLI flag exposes the new option - `-v` shorthand for `--verbose` on the global program options - Fix transaction install-count tracking: count committed docs accurately without relying on subtract-on-failure arithmetic - Add 6 new tests covering concurrency=1 sequential, concurrency=4 parallel, multiple embedBatch calls per install, concurrency limit enforcement, incremental progress reporting, and partial-failure error counting https://claude.ai/code/session_019hzhbEgV1ysnGmFVXBWzkZ * fix: address all 4 Copilot review comments on PR #339 - Validate batchSize, concurrency, resumeFrom at the start of installPack and throw ValidationError for invalid values (comments 3 & 4). Concurrency <= 0 would silently hang the semaphore indefinitely. - Add CLI lower-bound guard: --concurrency < 1 exits with a user-facing error before ever calling installPack (comment 3). - Lazy chunking: pre-chunking all documents upfront held chunks for the entire pack in memory simultaneously. Batches now store only the raw documents; resolveBatch() chunks on demand right before embedBatch is called, so only one batch's worth of chunks is in memory at a time (comment 2). - Wrap provider.embedBatch() in try/catch so synchronous throws are converted to rejected Promises rather than escaping scheduleNext() and leaving the outer Promise permanently pending (comment 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * fix: address 7 pre-release bugs from audit (#342) (#344) - Guard JSON.parse in rowToWebhook with try/catch, default to [] - Guard JSON.parse in rowToSavedSearch with try/catch, default to null - Guard JSON.parse in loadDbConnectorConfig with try/catch, throw ConfigError - Push dateFrom/dateTo filters into SQL WHERE in listDocuments (before LIMIT) - Validate negative limit in resolveSelector, throw ValidationError - Replace manual substring extension parsing with path.extname() in packs.ts - Verified reporter.ts is already tracked on development (no action needed) - Added tests for all fixes (corrupted JSON, SQL-level date filtering, negative limit) Closes #342 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) (#343) * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) Adds opt-in spidering to URL indexing. A single seed URL can now crawl and index an entire documentation site or wiki section in one call. New files: - src/core/link-extractor.ts: indexOf-based <a href> extraction, relative URL resolution, fragment stripping, dedup, scheme filtering. No regex. - src/core/spider.ts: BFS crawl engine with sameDomain, pathPrefix, excludePatterns (glob), maxPages (hard cap 200), maxDepth (hard cap 5), 10-min total timeout, robots.txt (User-agent: * and libscope), and 1s inter-request delay. Yields SpiderResult per page; returns SpiderStats. - tests/unit/link-extractor.test.ts: 25 tests covering relative resolution, dedup, fragment stripping, scheme filtering, attribute order, edge cases. - tests/unit/spider.test.ts: 20 tests covering BFS order, depth/page limits, domain + path + pattern filtering, cycle detection, robots.txt, partial failure recovery, stats, and abortReason reporting. Modified: - src/core/url-fetcher.ts: adds fetchRaw() export returning raw body + contentType + finalUrl before HTML-to-markdown conversion, so the spider can extract links from HTML before conversion. - src/api/routes.ts: POST /api/v1/documents/url now accepts spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns. Returns { documents, pagesIndexed, pagesCrawled, pagesSkipped, errors, abortReason }. - src/mcp/server.ts: submit-document tool gains spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns parameters. Safety: all fetched URLs pass through the existing SSRF validation in fetchRaw() (DNS resolution, private IP blocking, scheme allowlist). Hard limits (200 pages, depth 5, 10min) cannot be overridden by callers. robots.txt is fetched once per origin and Disallow rules are honoured. Individual page failures do not abort the crawl. Closes #315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve CI lint errors in spider implementation - Remove unnecessary type assertions (routes.ts, mcp/server.ts) — TypeScript already narrows SpiderResult/SpiderStats from the generator - Add explicit return type annotation on mock fetchRaw to satisfy no-unsafe-return rule in spider.test.ts - Replace .resolves.not.toThrow() with a direct assertion — vitest .resolves requires a Promise, not an async function Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address CodeQL security findings in spider/link-extractor link-extractor.ts (CodeQL #30 — incomplete URL scheme check): Replace the enumerated scheme blocklist (javascript:, vbscript:, etc.) with a strict http/https allowlist check on the resolved URL protocol. An allowlist is exhaustive by definition; a blocklist will always miss obscure schemes like vbscript:, blob:, or future additions. spider.ts (CodeQL #31 — incomplete multi-character sanitization): Replace the regex-based tag stripper /<[^>]+>/g in extractTitle() with an indexOf-based stripTags() function. The regex stops at the first > which can be inside a quoted attribute value (e.g. <img alt="a>b">), potentially leaving partial tag content in the extracted title. The new implementation walks quoted attribute values explicitly so no tag content leaks through regardless of its internal structure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address all Copilot review comments on spider PR (#343) - link-extractor: add word-boundary check in extractHref to prevent matching data-href, aria-href (false positives on non-href attributes) - spider: rename pagesIndexed → pagesFetched throughout (SpiderStats interface already used pagesFetched; sync implementation + tests) - spider: per-origin robots.txt cache (Map<origin, Set>) fetched lazily as new origins are encountered during crawl (was seed-only before) - spider: normalize to raw.finalUrl after redirects — visited set, yielded URL, and link-extraction base all use the canonical URL - routes: validate maxPages/maxDepth are finite positive integers - routes: change conditional spread &&-patterns to ternaries - routes: remove inner try/catch for spider fetch errors; add FetchError to top-level handler (consistent with single-URL mode → 502) - mcp/server: replace conditional spreads with explicit if assignments - mcp/server: validate spider=true requires url (throws ValidationError) - openapi: document spider request fields in IndexFromUrlRequest schema, add SpiderResponse schema, update 201 response to oneOf Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix prettier formatting in spider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: bring main up to date with development for v1.3.0 (#353) * chore: add development branch workflow (#327) * chore: add development branch workflow - Add merge-gate.yml: enforces only 'development' can merge into main - Update CI/CodeQL/Docker workflows to run on both main and development - Update dependabot.yml: target-branch set to development for all ecosystems - Update copilot-instructions.md: document branch workflow convention - Rulesets configured: Main (requires merge-gate + squash-only), Development (requires CI status checks + PR) - Default branch set to development - All open PRs retargeted to development Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip Vercel preview deployments on non-main branches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger check refresh --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: create-pack from local folder or URL sources (#329) * fix: comprehensive audit fixes — security, performance, resilience, API hardening Addresses findings from issue #314: - SSRF protection for webhook URLs (CRITICAL) - Scrub secrets from exports - Stored XSS prevention on document URL - O(n²) and N+1 fixes in bulk operations - Rate limit cache eviction improvement - SSE backpressure handling - Replace raw Error() with typed errors - Fetch timeouts on all network calls - Input validation on API parameters - Search query length limit - Silent catch block logging - DNS rebinding check fix - N+1 in Slack user resolution - Pagination on webhook/search list endpoints Closes #314 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: fix SSE listener leak, preserve caller signals, add SSRF validation to webhook test, chunk SQL params, use dynamic import in test - SSE backpressure: create single disconnect promise, race against drain (no listener accumulation) - http-utils.ts/onenote.ts: use AbortSignal.any() to combine caller signal with timeout - Webhook test endpoint: validate URL with validateWebhookUrlSsrf before fetch - bulk.ts: chunk IN clause to 999 params max (SQLite limit) - webhooks.test.ts: dynamic import after vi.mock() for deterministic DNS mocking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: consolidate and fix CI/CD workflows - Merge lint + typecheck into single job (saves one npm ci) - Add concurrency groups to ci, docker, codeql (cancel stale runs) - Add dependency-review-action on PRs (block vulnerable deps) - Add workflow_call trigger to ci.yml for reusability - Remove duplicate npm publish from release.yml (release-please owns it) - Fix SDK paths: sdk-go/ → sdk/go/, sdk-python/ → sdk/python/ - Fix Dependabot paths to match actual SDK directories - Add github-actions ecosystem to Dependabot (keep actions up to date) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb481870c883f77278291b72245b1ca0b890a78c. * feat: add --from option to pack create for folder/URL sources Adds createPackFromSource() that builds packs directly from local folders, files, or URLs without requiring database interaction. CLI: libscope pack create --name my-pack --from ~/docs/ [--extensions .md,.html] [--exclude pattern] [--no-recursive] Features: - Walks directories recursively using registered parsers - Fetches URLs via fetchAndConvert - Supports extension filtering, exclude patterns, progress callback - Multiple --from sources supported - Output format identical to DB export (pack install works unchanged) Closes #328 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: fix prettier formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add gzip support for pack files (.json.gz) Pack files can now be compressed with gzip for smaller distribution: - writePackFile/readPackFile auto-detect gzip by extension or magic bytes - installPack accepts both .json and .json.gz files - createPackFromSource defaults to .json.gz output (source packs can be large) - createPack (DB export) still defaults to .json - Auto-detects gzip by magic bytes even if extension is .json 5 new tests covering gzip write, install, magic byte detection, and round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add progress logging and fix dedup handling in pack install - Log each document as it's indexed so large installs show progress - Change pack install to use dedup: 'skip' for graceful duplicate handling - Make title+content_length dedup check respect the dedup mode setting (previously it always threw ValidationError regardless of dedup mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: auto-generate tags during pack creation and apply on install - Export tokenize() and add suggestTagsFromText() in tags.ts for DB-free tag generation - createPackFromSource() now auto-generates tags per document via TF-IDF - installPack() applies doc.tags via addTagsToDocument() after indexing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add HTML file parser for .html/.htm document indexing (#318) * feat: add HTML file parser for .html/.htm document indexing Adds HtmlParser using the existing node-html-markdown dependency. Strips <script>, <style>, and <nav> tags before conversion. Registered for .html and .htm extensions. Includes 12 tests covering conversion, tag stripping, edge cases. Closes #317 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address CodeQL and review comments on HTML parser - Replace regex-based tag stripping with node-html-markdown's native ignore option — eliminates all 3 CodeQL alerts (incomplete sanitization, bad HTML filtering regexp) - Wrap translate() in try/catch, throw ValidationError (consistent with other parsers) - Use trimEnd() instead of trim() to preserve leading indentation - Reuse single NHM instance for efficiency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "fix: skip Vercel preview deployments on non-main branches" This reverts commit eb481870c883f77278291b72245b1ca0b890a78c. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build(deps-dev): Bump eslint-config-prettier from 9.1.2 to 10.1.8 (#325) Bumps [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) from 9.1.2 to 10.1.8. - [Release notes](https://github.com/prettier/eslint-config-prettier/releases) - [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md) - [Commits](https://github.com/prettier/eslint-config-prettier/commits/v10.1.8) --- updated-dependencies: - dependency-name: eslint-config-prettier dependency-version: 10.1.8 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump lint-staged from 16.3.1 to 16.3.2 Bumps the minor-and-patch group with 1 update: [lint-staged](https://github.com/lint-staged/lint-staged). Updates `lint-staged` from 16.3.1 to 16.3.2 - [Release notes](https://github.com/lint-staged/lint-staged/releases) - [Changelog](https://github.com/lint-staged/lint-staged/blob/main/CHANGELOG.md) - [Commits](https://github.com/lint-staged/lint-staged/compare/v16.3.1...v16.3.2) --- updated-dependencies: - dependency-name: lint-staged dependency-version: 16.3.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Bump the actions group with 5 updates Bumps the actions group with 5 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `6` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `7` | | [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` | | [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` | Updates `actions/checkout` from 4 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) Updates `actions/setup-node` from 4 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v4...v6) Updates `actions/upload-artifact` from 4 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) Updates `actions/setup-python` from 5 to 6 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v6) Updates `actions/setup-go` from 5 to 6 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump @types/node from 22.19.13 to 25.3.3 Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 22.19.13 to 25.3.3. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.3.3 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Bump better-sqlite3 from 11.10.0 to 12.6.2 Bumps [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) from 11.10.0 to 12.6.2. - [Release notes](https://github.com/WiseLibs/better-sqlite3/releases) - [Commits](https://github.com/WiseLibs/better-sqlite3/compare/v11.10.0...v12.6.2) --- updated-dependencies: - dependency-name: better-sqlite3 dependency-version: 12.6.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): Bump eslint from 9.39.3 to 10.0.2 Bumps [eslint](https://github.com/eslint/eslint) from 9.39.3 to 10.0.2. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](https://github.com/eslint/eslint/compare/v9.39.3...v10.0.2) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert DeRienzo <rderienzo@voloridge.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add passthrough LLM mode for ask-question tool (#335) * feat: add passthrough LLM mode for ask-question tool Adds llm.provider = "passthrough" so the ask-question MCP tool returns retrieved context chunks directly to the calling LLM instead of requiring a separate OpenAI/Ollama provider. This is the natural design for MCP tools where the client already has an LLM (e.g. Claude Code). - config.ts: add "passthrough" to llm.provider union type and env var handling - rag.ts: add isPassthroughMode() helper and getContextForQuestion() which retrieves and formats context without an LLM call - mcp/server.ts: ask-question checks passthrough first and returns context directly; falls through to existing LLM path otherwise Enable via config: { "llm": { "provider": "passthrough" } } Enable via env var: LIBSCOPE_LLM_PROVIDER=passthrough Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: format config.ts and include passthrough in provider override - Reformat long if-condition to satisfy prettier (printWidth: 100) - Fix logic bug: passthrough provider was checked in outer condition but not spread into overrides.llm.provider Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address 9 audit findings from issue #332 (#333) * fix: address 9 audit findings from issue #332 Security - middleware: use timingSafeEqual for API key comparison (#2) - url-fetcher, http-utils: replace process-wide NODE_TLS_REJECT_UNAUTHORIZED mutation with per-request undici Agent to eliminate TLS race condition (#1) Bugs - indexing: re-throw unexpected embedding errors so transaction rolls back instead of silently committing chunks with no vector (#3) - search: replace correlated minRating subquery with avg_r.avg_rating from the pre-joined aggregate in FTS and LIKE search paths (#4) Performance - bulk: replace O(n²) docs.find() loops with pre-built Map; replace per-document getDocumentTags() calls with a single getDocumentTagsBatch() query (#5) - config: add 30-second TTL cache to loadConfig() so disk reads are not repeated on every request (#6) Code quality - routes: check res.write() return value to handle SSE backpressure (#7) - reindex: delegate to schema.createVectorTable() instead of duplicating the vec0 DDL inline (#8) - obsidian: replace hand-rolled parseSimpleYaml() with js-yaml, normalise Date objects back to ISO-8601 strings (#9) Docs - agents.md: expand architecture tree to include src/api/ and src/connectors/; add Security Patterns section with correct undici examples - CONTRIBUTING.md: fix check-suite command (npm test → npm run test:coverage) and correct coverage threshold (80% → actual 75%/74%) Tests - bulk.test: add dateFrom/dateTo filter coverage - config.test: add cache-hit test; call invalidateConfigCache() before env-var tests so TTL cache doesn't return stale results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused warnIfTlsBypassMissing function Dead code after conflict resolution chose the per-request undici Agent approach (which doesn't need a warning about NODE_TLS_REJECT_UNAUTHORIZED). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update tests for config cache and retry semantics - Add invalidateConfigCache() before loadConfig() in 4 env-override tests that were failing because the 30s TTL cache introduced in the config module was returning stale results from the previous test's cache entry - Update http-utils retry assertion: maxRetries=2 means 1 initial + 2 retries = 3 total calls (loop is attempt <= maxRetries) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: CLI logging improvements and pack installation performance (#330) (#336) - Add `src/cli/reporter.ts`: PrettyReporter (ANSI colors + \r progress bar), SilentReporter (no-op for verbose/JSON mode), `isVerbose()` and `createReporter()` factory. `LIBSCOPE_VERBOSE=1` env var alternative. - Update `setupLogging` to default to "silent" in CLI mode (pretty reporter handles user-facing output). Verbose/`--log-level` flags still route to structured JSON pino logs. Fix duplicate `initLogger` calls in onenote connect/disconnect commands to use `setupLogging` consistently. - Update `installPack` in `packs.ts` to support batch embedding and progress reporting: - New `InstallOptions` interface with `batchSize`, `resumeFrom`, `onProgress` fields - Batch documents: chunk all → single `provider.embedBatch` call per batch → single SQLite transaction per batch (avoids N embedding calls) - `resumeFrom` skips the first N documents (enables partial install resume after failure) - `InstallResult` now includes `errors` count - Add `--batch-size` and `--resume-from` CLI options to `pack install` - Tests: `tests/unit/reporter.test.ts` (17 tests covering PrettyReporter, SilentReporter, isVerbose, env var detection); extended `tests/unit/packs.test.ts` with 7 new tests for progress callbacks, batch efficiency, resumeFrom, embedBatch failure handling. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Claude/fix issue 331 s1qzu (#338) * build(deps): Bump the npm_and_yarn group across 1 directory with 2 updates (#337) Bumps the npm_and_yarn group with 2 updates in the / directory: [@hono/node-server](https://github.com/honojs/node-server) and [hono](https://github.com/honojs/hono). Updates `@hono/node-server` from 1.19.9 to 1.19.10 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](https://github.com/honojs/node-server/compare/v1.19.9...v1.19.10) Updates `hono` from 4.12.3 to 4.12.5 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](https://github.com/honojs/hono/compare/v4.12.3...v4.12.5) --- updated-dependencies: - dependency-name: "@hono/node-server" dependency-version: 1.19.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: hono dependency-version: 4.12.5 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (#341) * fix: eliminate SSRF TOCTOU and ReDoS vulnerabilities (closes #340) **SSRF (CWE-918 — CodeQL alert #28)** Replace the two-step validate-then-fetch approach in url-fetcher.ts with IP-pinned requests using node:http / node:https directly. validateUrl() resolves DNS and checks for private IPs, then the validated IP is passed straight to the TCP connection (hostname: pinnedIp, servername: original hostname for TLS SNI). There is now zero TOCTOU window between validation and the actual network request. The redundant post-fetch DNS rebinding check and the env-var-based TLS bypass wrapper are removed; rejectUnauthorized is now passed directly to the request options. An internal _setRequestImpl hook is exported for unit test injection so tests can stub responses without touching node:http / node:https. Tests are updated accordingly. **ReDoS (CWE-1333 — CodeQL alert #24)** Five regexes in confluence.ts used the pattern [^>]*ac:name="X"[^>]* — two [^>]* quantifiers around a fixed literal. For input that contains a large attribute blob without the target ac:name value, the engine must try all O(n²) splits before concluding no match (catastrophic backtracking). Fix: rewrite the leading [^>]* as (?:(?!ac:name="X")[^>])* — a negative lookahead prevents the quantifier from overlapping with the literal, making backtracking structurally impossible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: replace remaining polynomial regexes in confluence.ts with indexOf helpers The conflict resolution left the original `/<ac:structured-macro [^>]*>([\s\S]*?) <\/ac:structured-macro>/gi` patterns in place for code blocks and info/tip panels (those were not part of the original security fix diff). These have the same O(n²) backtracking problem: with k opening tags and no closing tags, [\s\S]*? scans O(n - pos) chars per attempt, totalling O(n²). Replace the entire convertConfluenceStorage function with the indexOf-based approach (replaceStructuredMacros / replaceTagPairs / extractTagContent helpers) that eliminates all regex-based tag parsing. Also add removeSelfClosingMacros to handle the self-closing TOC case without regex, since the previous self-closing fix still used a [^>]*ac:name="toc"[^>]* pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: concurrent pack installation and -v verbose shorthand (issue #330) (#339) * feat: concurrent pack installation and -v verbose shorthand (issue #330) Add concurrent batch embedding to installPack for significant performance improvement on large packs, plus CLI ergonomics improvements. Key changes: - `InstallOptions.concurrency` (default: 4): controls how many embedBatch calls run simultaneously; embedding is I/O-bound so parallelism directly reduces wall-clock installation time - Refactor installPack to pre-chunk all documents upfront, then use a semaphore-based scheduler to run up to `concurrency` embedBatch calls concurrently while inserting completed batches in-order (SQLite requires serialised writes); progress callbacks fire after each batch as before - `pack install --concurrency <n>` CLI flag exposes the new option - `-v` shorthand for `--verbose` on the global program options - Fix transaction install-count tracking: count committed docs accurately without relying on subtract-on-failure arithmetic - Add 6 new tests covering concurrency=1 sequential, concurrency=4 parallel, multiple embedBatch calls per install, concurrency limit enforcement, incremental progress reporting, and partial-failure error counting https://claude.ai/code/session_019hzhbEgV1ysnGmFVXBWzkZ * fix: address all 4 Copilot review comments on PR #339 - Validate batchSize, concurrency, resumeFrom at the start of installPack and throw ValidationError for invalid values (comments 3 & 4). Concurrency <= 0 would silently hang the semaphore indefinitely. - Add CLI lower-bound guard: --concurrency < 1 exits with a user-facing error before ever calling installPack (comment 3). - Lazy chunking: pre-chunking all documents upfront held chunks for the entire pack in memory simultaneously. Batches now store only the raw documents; resolveBatch() chunks on demand right before embedBatch is called, so only one batch's worth of chunks is in memory at a time (comment 2). - Wrap provider.embedBatch() in try/catch so synchronous throws are converted to rejected Promises rather than escaping scheduleNext() and leaving the outer Promise permanently pending (comment 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * fix: address 7 pre-release bugs from audit (#342) (#344) - Guard JSON.parse in rowToWebhook with try/catch, default to [] - Guard JSON.parse in rowToSavedSearch with try/catch, default to null - Guard JSON.parse in loadDbConnectorConfig with try/catch, throw ConfigError - Push dateFrom/dateTo filters into SQL WHERE in listDocuments (before LIMIT) - Validate negative limit in resolveSelector, throw ValidationError - Replace manual substring extension parsing with path.extname() in packs.ts - Verified reporter.ts is already tracked on development (no action needed) - Added tests for all fixes (corrupted JSON, SQL-level date filtering, negative limit) Closes #342 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) (#343) * feat: URL spidering — crawl linked pages with configurable depth and page limits (#315) Adds opt-in spidering to URL indexing. A single seed URL can now crawl and index an entire documentation site or wiki section in one call. New files: - src/core/link-extractor.ts: indexOf-based <a href> extraction, relative URL resolution, fragment stripping, dedup, scheme filtering. No regex. - src/core/spider.ts: BFS crawl engine with sameDomain, pathPrefix, excludePatterns (glob), maxPages (hard cap 200), maxDepth (hard cap 5), 10-min total timeout, robots.txt (User-agent: * and libscope), and 1s inter-request delay. Yields SpiderResult per page; returns SpiderStats. - tests/unit/link-extractor.test.ts: 25 tests covering relative resolution, dedup, fragment stripping, scheme filtering, attribute order, edge cases. - tests/unit/spider.test.ts: 20 tests covering BFS order, depth/page limits, domain + path + pattern filtering, cycle detection, robots.txt, partial failure recovery, stats, and abortReason reporting. Modified: - src/core/url-fetcher.ts: adds fetchRaw() export returning raw body + contentType + finalUrl before HTML-to-markdown conversion, so the spider can extract links from HTML before conversion. - src/api/routes.ts: POST /api/v1/documents/url now accepts spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns. Returns { documents, pagesIndexed, pagesCrawled, pagesSkipped, errors, abortReason }. - src/mcp/server.ts: submit-document tool gains spider, maxPages, maxDepth, sameDomain, pathPrefix, excludePatterns parameters. Safety: all fetched URLs pass through the existing SSRF validation in fetchRaw() (DNS resolution, private IP blocking, scheme allowlist). Hard limits (200 pages, depth 5, 10min) cannot be overridden by callers. robots.txt is fetched once per origin and Disallow rules are honoured. Individual page failures do not abort the crawl. Closes #315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve CI lint errors in spider implementation - Remove unnecessary type assertions (routes.ts, mcp/server.ts) — TypeScript already narrows SpiderResult/SpiderStats from the generator - Add explicit return type annotation on mock fetchRaw to satisfy no-unsafe-return rule in spider.test.ts - Replace .resolves.not.toThrow() with a direct assertion — vitest .resolves requires a Promise, not an async function Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address CodeQL security findings in spider/link-extractor link-extractor.ts (CodeQL #30 — incomplete URL scheme check): Replace the enumerated scheme blocklist (javascript:, vbscript:, etc.) with a strict http/https allowlist check on the resolved URL protocol. An allowlist is exhaustive by definition; a blocklist will always miss obscure schemes like vbscript:, blob:, or future additions. spider.ts (CodeQL #31 — incomplete multi-character sanitization): Replace the regex-based tag stripper /<[^>]+>/g in extractTitle() with an indexOf-based stripTags() function. The regex stops at the first > which can be inside a quoted attribute value (e.g. <img alt="a>b">), potentially leaving partial tag content in the extracted title. The new implementation walks quoted attribute values explicitly so no tag content leaks through regardless of its internal structure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address all Copilot review comments on spider PR (#343) - link-extractor: add word-boundary check in extractHref to prevent matching data-href, aria-href (false positives on non-href attributes) - spider: rename pagesIndexed → pagesFetched throughout (SpiderStats interface already used pagesFetched; sync implementation + tests) - spider: per-origin robots.txt cache (Map<origin, Set>) fetched lazily as new origins are encountered during crawl (was seed-only before) - spider: normalize to raw.finalUrl after redirects — visited set, yielded URL, and link-extraction base all use the canonical URL - routes: validate maxPages/maxDepth are finite positive integers - routes: change conditional spread &&-patterns to ternaries - routes: remove inner try/catch for spider fetch errors; add FetchError to top-level handler (consistent with single-URL mode → 502) - mcp/server: replace conditional spreads with explicit if assignments - mcp/server: validate spider=true requires url (throws ValidationError) - openapi: document spider request fields in IndexFromUrlRequest schema, add SpiderResponse schema, update 201 response to oneOf Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix prettier formatting in spider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: comprehensive documentation update for v1.3.0 (#347) - README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: allow release-please PRs to pass merge gate and trigger CI (#348) * docs: comprehensive documentation update for v1.3.0 - README: fix license (BUSL-1.1, not MIT), expand MCP tools table to all 26 tools, expand REST API table with all endpoints (webhooks, links, analytics, connectors status, suggest-tags, bulk ops), add webhooks section with HMAC signing example, add missing CLI commands (bulk ops, saved searches, document links, docs update) - getting-started: fix Node.js requirement (20, not 18), add sections for web dashboard, organize/annotate features, REST API - mcp-setup: expand available tools section to list all 26 tools grouped by category instead of just 4 - mcp-tools reference: add 5 missing tools — update-document, suggest-tags, link-documents, get-document-links, delete-link - rest-api reference: add all missing endpoints, reorganize by category, add examples for update, bulk retag, webhooks, links, saved searches - configuration guide: document passthrough LLM provider - configuration reference: add passthrough LLM, llm.ollamaUrl key, expand config set examples to cover all settable keys - cli reference: expand config set supported keys list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: allow release-please PRs to pass merge gate and trigger CI Two issues prevented PR #238 from getting CI runs: 1. merge-gate blocked release-please PRs — the gate only allowed 'development' as the source branch, but release-please uses 'release-please--branches--main--components--libscope'. Updated to allow any branch matching 'release-please--*'. 2. CI never ran on the PR — GitHub does not trigger workflows when GITHUB_TOKEN creates a PR (intentional security restriction to prevent infinite loops). Fixed by passing a PAT via secrets.GH_TOKEN to the release-please action so its PR creation triggers CI. Note: requires a 'GH_TOKEN' secret in repo settings — a classic PAT with repo and workflow scopes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * ci: trigger checks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: merge development into main for v1.3.0 release (#354) * chore: add development branch workflow (#327) * chore: add development branch workflow - Add merge-gate.yml: enforces only 'development' can merge into main - Update CI/CodeQL/Docker workflows to run on both main and development - Update dependabot.yml: target-branch set to development for all ecosystems - Update copilot-instructions.md: document branch workflow convention - Rulesets configured: Main (requires merge-gate + squash-only), Development (requires CI status checks + PR) - Default branch set to development - All open PRs retargeted to development Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip Vercel preview deployments on non-main branches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger check refresh --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: create-pack from local folder or URL sources (#329) * fix: comprehensive audit fixes — security, performance, resilience, API hardening Addresses findings from issue #31…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch contains all of development's changes merged with main, with the merge-gate.yml conflict already resolved (keeping the version that allows release-please branches).
Why this branch instead of development directly: The merge-gate.yml file was independently added to both branches (add/add conflict) — git will always conflict when merging development→main. This branch has that conflict pre-resolved.
NOTE: The
enforce-source-branchcheck will fail since source is notdevelopment. You need to bypass it once to break the circular dependency.To merge this PR: Go to Settings → Rules → Main Rules → add yourself (RobertLD) as a bypass actor, merge this PR, then remove yourself as bypass actor.
After this merges, release-please will run and create a clean v1.3.0 release PR that will pass the (now-updated) merge gate.