feat: file content source for search — bounded passive-scope file index - #17
Merged
Conversation
Proposes a bounded passive-scan file index as a second nmg search content source: .nmg-search-scope hot-zone file (semantically opposite to .gitignore), scope learned from Agent grep/read behaviour, lexical-first FTS index, git-status incremental updates, and memory+file result fusion. Also anchors .nmg/ to the repo root in .gitignore so nested .nmg dirs are not ignored. Mirrors Zoekt/CocoIndex patterns at small scale; lexical-first per the grep-vs-vector finding.
Adds a second content source to nmg search: a bounded passive-scan FTS index over the project's own files, so an Agent does not have to manually re-discover a changing project. - src/core/file-index.ts: FileIndex with .nmg-search-scope (path-list, semantically opposite to .gitignore), incremental crawl via content hash, trigram FTS (external-content table), search with excerpts, and addScopePath for scope growth (the Agent is the first crawler). - types.ts: FileHit + MemoryContext.files (separate from memory results). - service.ts: #searchProjectFiles attaches file hits when projectDir is supplied; #fileIndexes lifecycle. - stg.ts: mergeStgLtgContexts passes files through. - search-projection.ts + agent-surface.ts + main.ts: file hits render as a separate file= partition (not mixed into memory ranking). - tests: file-index.test.ts (7 cases: scope parse, resolve, collect, crawl/search/incremental, change/removal, scope growth, excludes). Files are a search index, not memory: content never enters STG/LTG.
The DSH adapter observes grep/read tool results (tools/result event) and feeds non-empty hit paths into the file content source scope via a provided FileIndex service. The Agent's own search behaviour grows .nmg-search-scope, closing the scope-evolution loop of the file content source design. The listener observes only (never intercepts), parses defensively, and degrades to a no-op when the FileIndex service is absent. FileIndex is created per-plugin and closed on dispose.
Accept exec.toolName and exec.input as defensive aliases for exec.name/exec.arguments (different host integrations may use either), with comments noting the confirmed dsh-tools field names (grep/read from dsh-tool-fs-search/dsh-tool-fs).
wefio
changed the title
Old stores created before the opaque-id migration still carry UNIQUE(task_id, sequence), but new writes store sequence=0 (the column is legacy), so the second entry of any channel failed with a UNIQUE constraint error. ensureTaskBoardColumns now rebuilds the table without that constraint when the legacy autoindex is detected, preserving all data and columns. Verified on the real user DB: two entries on the same task now insert cleanly.
collectScopePaths → guard clauses + composed helpers (grepHitPaths/readHitPaths), applying the guard-clause and composed-function refactoring techniques to bring the observer below the complexity threshold.
wefio
added a commit
that referenced
this pull request
Sep 1, 2026
Guides reducing cyclomatic complexity (CodeFactor Complex Method / npm run complexity:gate) using guard clauses, composed functions, lookup tables, and modern TS features, with the worked example of the dsh-nmg scope observer (37 → below threshold). Validated on PR #17.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design proposal + core implementation for adding a file content source to
mg search.
Design (docs/design/file-content-source-design.md)
Bounded passive-scan file index with a learned scope:
Implementation (committed)
Verified: tsc clean, file-index 7/7, agent-surface + session-AG 20/20, end-to-end crawl+search on the real repo works (62 files indexed; \hybridScore\ hits search-ranking/retrieval/qpp).
Pending