Merge PR #9: PathFilter, file_path scoping, progress reporting, and include_tests#10
Merged
Codeturion merged 30 commits intomasterfrom Apr 28, 2026
Merged
Merge PR #9: PathFilter, file_path scoping, progress reporting, and include_tests#10Codeturion merged 30 commits intomasterfrom
Codeturion merged 30 commits intomasterfrom
Conversation
…r into server indexing
…embers, clean up docstring
rglob() cannot skip directories mid-walk — it traverses the entire tree (including all git worktrees) before the path_filter check runs. With 16 worktrees the indexer was doing ~17x the necessary work, causing 1000+s startup times instead of ~2 minutes. Switch all six walk sites (base.py, server.py x2, typescript.py, python_parser.py, go.py, java.py) to os.walk() with in-place dirs[:] pruning. Excluded directories (.worktrees, submodules, _SKIP_DIRS) are now dropped before descent so os.walk never enters them.
…orktree-support feat: file filtering, worktree exclusions, and query-time file_path scoping
Also forward on_progress in TypeScriptParser's parse_directory override. The callback is invoked after each successful parse_file call, passing the Path of the parsed file; skipped files and parse errors do not trigger it.
…ertions - Remove `print(summary, file=sys.stderr)` from `main()` after calling `_index_full()`, since `_index_full` already prints the done line itself. - Strengthen `test_index_full_emits_progress_to_stderr` to also assert that the scanning line and 0% baseline indexing line appear in stderr.
…le level Move on_progress callback to a finally block in all five parsers so it fires whether parse_file succeeds or raises. Previously, unparseable files were counted by _count_files but never triggered on_progress, causing the startup progress display to stall below 100% on repos with problematic files. Also move `import sys` from inside except blocks to module-level imports in typescript.py, python_parser.py, go.py, and java.py.
- Add _DEFAULT_EXCLUDED_DIRS to PathFilter (node_modules, .git, dist, build, vendor, .nx, .yarn, etc.) so vendored dirs are always skipped - Rewrite detect_languages to use os.walk with PathFilter instead of rglob, which crawled into node_modules and hung indefinitely - Consolidate duplicate parse_directory from all 4 parsers into BaseParser with str ops instead of pathlib (saves ~3s on 30K files) - Rewrite _count_braces_and_parens: regex strip + str.count with fast path for lines without strings/comments (saves ~4s on 30K files) - Replace Path.read_text with open()+read(), path.relative_to with os.path.relpath, and _file_to_module with pure string ops - Remove per-parser _SKIP_DIRS (now centralized in PathFilter) - Remove __tests__/__mocks__/test/spec file exclusions from TS parser so test files are indexed for coding reference - Add skip_suffixes, skip_files, _should_skip_dir hooks to BaseParser for per-parser file filtering without duplicating the walk loop Benchmark on 34K-file TS monorepo: 22s -> 15s (32% faster)
Path('~/work/cloud').is_dir() returns False because Python's pathlib
does not expand ~ — causing the server to skip indexing entirely and
sit idle in the MCP listen loop without emitting the done: line.
These are internal planning artifacts and fork install instructions that don't belong in the upstream PR.
- Remove CppParser.parse_directory override, use BaseParser's os.walk with _should_skip_dir for C++-specific dirs (Debug, Release, x64, x86, cmake-build-*) - Remove _SKIP_DIRS from cpp.py (now centralized in PathFilter) - Apply open()+read() and os.path.relpath to cpp.py for consistency - Restore .test.ts/.spec.ts skip suffixes in TypeScript parser
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive path filtering system and significant performance optimizations for the indexing process. Key changes include the addition of a PathFilter class to handle default and user-defined exclusions, the integration of progress reporting during full indexing, and the refactoring of language parsers to use os.walk for faster directory traversal. The database layer was also updated to support scoped searches via a new file_path parameter. Feedback focuses on further performance improvements, such as consolidating multiple directory walks, reducing pathlib overhead in hot loops, and addressing code duplication in SQL construction. Additionally, the file counting logic for progress reporting needs to be synchronized with parser-specific exclusion rules to ensure accuracy.
- Extract _walk_files from parse_directory so _count_files can reuse the same filter logic (skip_suffixes, skip_files, _should_skip_dir) - Collect mtimes in on_progress callback instead of a separate walk - _index_full now does 2 walks (count + parse) instead of 3
- db.py: extract _file_path_condition() helper, dedupe between search() and get_class_members() - filters.py: add is_dir_excluded_git() and _read_git_file_str() for string-path walks; route is_file_excluded() through is_file_excluded_rel - parsers/base.py: walk loop uses is_dir_excluded_git(root, d) instead of Path(os.path.join(...)) - server.py: _index_incremental walk is fully string-based — drops per-dir Path() and per-file relative_to() in favor of prefix slicing
Add include_tests boolean (default false) to search, get_signature, and get_class MCP tools so test files are excluded from results by default but easily pulled in when needed. Test file detection covers common conventions: - Directory patterns: __tests__/, tests/, test/ - Filename patterns: .test., .spec., _test., test_ Also adds .js/.jsx to TypeScript parser file_extensions so plain JavaScript files are indexed alongside TypeScript.
This was referenced Apr 28, 2026
Codeturion
changed the title
The incremental walk previously used a single global os.walk over all_extensions(), which only applied path_filter. Per-parser rules (skip_suffixes, skip_files, _should_skip_dir) were skipped, so _file_mtimes accumulated entries for files the parser ultimately ignored (e.g. .test.ts, .spec.ts, .d.ts, module-info.java). This caused first-reindex-after-full-index to spuriously report files as "added" and inflated the scanned-file count. Switch to per-parser _walk_files (same as _count_files in _index_full) so the two indexing paths see the same file set. Add a regression test.
_index_full pins to a single parser when --language is set, but _index_incremental ignored the flag and called get_parsers_for_project, which auto-detects every language present in the tree. On a polyglot project run with --language=cpp, the incremental walk found .py files the full walk had skipped, so the first reindex falsely reported them as added (and re-parsed them on every restart-then-reindex cycle). Store the CLI language in a module global at startup and apply the same pinning to incremental walks. Add a regression test covering the case where --language=python is set in a project containing both .py and .ts files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges #9 from @michael-howell-island with conflict resolution against #8 (C++ parser).
namespaceandfile_pathparams now coexist inget_class_members()BaseParser.parse_directory(consolidated os.walk with PathFilter).test.ts/.spec.tsskip suffixes in TypeScript parserWhat's included from #9
.codesurfaceignore,--excludeCLI globs, worktree/submodule skippingsearch,get_signature,get_classtools accept optionalfile_pathfilterscanning N files...,indexing: 50%,done:)parse_directory: all parsers now useBaseParser's os.walk loop withskip_suffixes,skip_files,_should_skip_dirhookssearch,get_signature,get_classexclude test files by default; passinclude_tests=trueto opt in. TypeScript parser also picks up.js/.jsx.Review comments addressed
_file_path_condition()helper in db.py (was duplicated betweensearch()andget_class_members())Path()per dir, norelative_to()per file)_count_filescalls each parser's_walk_files)on_progress)Test plan
include_tests)