Version: graphifyy 0.8.35 · macOS · post-commit hook
Summary
On a repo where the post-commit hook has been running since June, graph.json accumulated
683 nodes (3.9%) whose source_file no longer exists, plus 1,068 edges touching them.
84% came from a single directory removed in one commit six weeks ago. Three separate
issues combine to make this permanent:
1. _rebuild_code full-reconciliation branch filters on _CODE_EXTENSIONS, so docs are never evicted
In watch.py, the changed_paths is None branch reconciles existing nodes against the
current corpus to evict deleted files (#1007), but skips anything that is not a code file:
if Path(sf).suffix.lower() not in _CODE_EXTENSIONS:
continue_CODE_EXTENSIONS does not include .md. For a docs-heavy repo this means markdown nodes
are never evicted by any code path. In our graph, 182 of the 683 stale nodes are .md
and are unreachable by both branches.
This looks unintended: the branch exists specifically to evict vanished sources, and the
extension filter defeats that for exactly the file types many users index.
2. Rebuild already in progress - changes queued discards the change list; nothing is queued
acquire_lock=True takes a non-blocking flock and returns False when held — the
docstring says so. But the log line says "changes queued", and no queue exists. That
commit's changed_paths (including its deletions) is lost permanently.
Our log has 116 occurrences of this line, plus 40 hard failures
(exceeded 600s, Refusing to overwrite, No such file or directory: 'graphify-out').
Every one is a potential permanent leak, because the incremental path never revisits
anything outside the current commit's diff.
Suggest either wording it as "skipped" or actually recording the paths for the next run.
3. No reconciliation backstop
The post-commit hook always passes changed_paths, so the full-reconciliation branch
never runs in normal operation. Combined with (2), a single skipped run leaks forever.
A graphify prune subcommand — drop nodes whose source_file no longer exists, plus their
edges — would make this self-healing regardless of root cause. We wrote a 90-line local
script that does exactly this; happy to open a PR if the shape is wanted.
Repro sketch
- Index a repo containing a directory of
.md files.
git rm -r that directory and commit while another rebuild holds the lock
(or simply let one of the 600s timeouts fire).- Commit anything else. The removed files' nodes are still in
graph.json, and no
subsequent hook-driven rebuild removes them.
Version: graphifyy 0.8.35 · macOS · post-commit hook
Summary
On a repo where the post-commit hook has been running since June,
graph.jsonaccumulated683 nodes (3.9%) whose
source_fileno longer exists, plus 1,068 edges touching them.84% came from a single directory removed in one commit six weeks ago. Three separate
issues combine to make this permanent:
1.
_rebuild_codefull-reconciliation branch filters on_CODE_EXTENSIONS, so docs are never evictedIn
watch.py, thechanged_paths is Nonebranch reconciles existing nodes against thecurrent corpus to evict deleted files (#1007), but skips anything that is not a code file:
_CODE_EXTENSIONSdoes not include.md. For a docs-heavy repo this means markdown nodesare never evicted by any code path. In our graph, 182 of the 683 stale nodes are
.mdand are unreachable by both branches.
This looks unintended: the branch exists specifically to evict vanished sources, and the
extension filter defeats that for exactly the file types many users index.
2.
Rebuild already in progress - changes queueddiscards the change list; nothing is queuedacquire_lock=Truetakes a non-blocking flock and returnsFalsewhen held — thedocstring says so. But the log line says "changes queued", and no queue exists. That
commit's
changed_paths(including its deletions) is lost permanently.Our log has 116 occurrences of this line, plus 40 hard failures
(
exceeded 600s,Refusing to overwrite,No such file or directory: 'graphify-out').Every one is a potential permanent leak, because the incremental path never revisits
anything outside the current commit's diff.
Suggest either wording it as "skipped" or actually recording the paths for the next run.
3. No reconciliation backstop
The post-commit hook always passes
changed_paths, so the full-reconciliation branchnever runs in normal operation. Combined with (2), a single skipped run leaks forever.
A
graphify prunesubcommand — drop nodes whosesource_fileno longer exists, plus theiredges — would make this self-healing regardless of root cause. We wrote a 90-line local
script that does exactly this; happy to open a PR if the shape is wanted.
Repro sketch
.mdfiles.git rm -rthat directory and commit while another rebuild holds the lock(or simply let one of the 600s timeouts fire).
graph.json, and nosubsequent hook-driven rebuild removes them.