fix: stop CPU/memory blowup on non-git projects (#713, #937, #841)

[bethzyy]

Fixes #713. Also addresses the non-git half of #937 / #841 (watcher re-index churn).

Non-git folders currently hit two independent resource-exhaustion bugs. They share one root cause — every guard in these paths assumes git answers for the project — so this PR fixes both; happy to split it into two PRs if you prefer (the mcp/discover and watcher changes are fully independent).

1. Memory: auto-index OOM guard never fires on non-git roots (#713)

maybe_auto_index counted files with git ls-files | wc -l:

• In a non-git folder the count is 0, so auto_index_limit can never trigger and auto-index walks/indexes the entire tree — tens of GB of RAM on large code folders (auto_index uses tens of GB of memory in a non-git folder containing code #713).
• On Windows the pipe was broken for everyone: cmd.exe passes single quotes literally and has no wc, so the guard silently returned nothing even for git repos.

Fix (src/mcp/mcp.c, src/discover/discover.{c,h}):

• Count git ls-files lines in C — portable, no wc, double quotes.
• When the git count is 0 (non-git root, or nothing tracked), fall back to a new cbm_discover_count_files(root, limit): a bounded native walk that honors the same skip dirs, suffix filters, and language gate as discovery and stops at limit + 1, so arbitrarily large trees cost O(limit) work. It deliberately does not load gitignore — over-counting only makes the guard more conservative, and the path must stay cheap.

2. CPU: watcher re-indexes forever when a tree stays dirty (#937/#841)

Two stacked defects in src/watcher/watcher.c:

• is_git_repo() used git rev-parse --git-dir, which walks up the directory tree. A plain non-git folder nested under any unrelated ancestor repo is misclassified as git — and since every file in it is untracked in that ancestor, git status never comes back clean.
• check_changes() treated "working tree is dirty" as "changed", so a persistently dirty tree triggered a full re-index on every poll cycle (5–60s), forever. For the nested non-git case this is a guaranteed infinite loop with zero user action; Watcher can repeatedly re-index dirty repos and cause large disk write amplification #937 measured >1 TB/day of disk writes from this class of churn.

Fix:

• A folder with no .git entry of its own and zero tracked files is treated as non-git (watcher skips polling, per the module's design). An empty freshly-git init-ed root keeps its .git; a monorepo sub-package keeps its tracked files — both stay watched.
• Boolean dirty check replaced with a status signature: FNV-1a over git status --porcelain --untracked-files=all -- . output, mixing in the mtime+size of each listed path. A poll re-indexes only when the signature moves:
  • a tree that merely stays dirty hashes identically → no churn;
  • a repeated edit to an already-dirty file changes mtime/size → still caught;
  • -- . scopes status to the watched subtree, so a monorepo package no longer churns on sibling changes;
  • the signature is committed only after a successful re-index, so failed runs retry;
  • submodule status is folded into the hash on POSIX (same coverage as the old submodule foreach check).

Tests

• watcher/discover/gitignore/git_context (173) and mcp (145) suites pass under ASan+UBSan on Linux.
• New: watcher_nested_non_git_skips (baseline logs strategy=none for a non-git folder under a git ancestor), discover_count_files_bounded (count, early exit, edge cases).
• Updated: watcher_continued_dirty previously asserted the buggy every-poll re-index; it now asserts stay-dirty → no re-trigger, further edit → re-trigger.
• E2E with the production binary: a 15-file non-git folder with auto_index_limit=10 logs autoindex.skip reason=too_many_files files=11 limit=10 (files=11 = early exit at limit+1); with limit=100 auto-index completes normally.

scripts/security-allowlist.txt updated for the renamed/added cbm_popen call sites in watcher.c (no new call classes).

🤖 Generated with Claude Code

[DeusData]

Thanks for tackling this resource-safety class. I have triaged it into 0.9.1-rc as high priority. Since it touches discovery, watcher behavior, MCP paths, and the security allowlist, review will focus on non-git guard behavior, watcher churn, the allowlist delta, and confirming there are no new file, process, or network side effects.