feat: incremental ingest with file hashing#29
Merged
Conversation
- SQLiteStore: files_state(file_path, hash) table + indexes on nodes/edges.file_path; new methods: get_file_hash, upsert_file_hash, delete_file_data, get_nodes_by_file, get_all_tracked_files - IngestionPipeline.run(store=): incremental mode — unchanged files are skipped (nodes loaded from store for resolver), changed/new files are re-extracted and saved; stale files removed automatically - CLI: --incremental / -i flag on cgis ingest; warns and falls back to full ingest when output is JSON - Tests: 4 new store tests + 3 new pipeline incremental tests (75 total) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… transaction - Use full_path.as_posix() in pipeline to prevent hash mismatches on Windows (str(Path) returns backslashes; posix paths are consistent). - delete_file_data now also deletes edges by source node, so structural edges with file_path=None (e.g. CONTAINS, DECLARES) are not orphaned. - Replace 3-commits-per-file loop with save_incremental_batch: a single SQLite transaction covers all deletes + inserts + hash upserts for every changed and stale file (150 → 1 transaction for 50 files). - _persist_incremental uses source→file lookup to assign edges without file_path to the correct file bucket. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SQLiteStore: newfiles_state(file_path, hash)table + indexes onnodes/edges.file_path; methods:get_file_hash,upsert_file_hash,delete_file_data,get_nodes_by_file,get_all_tracked_filesIngestionPipeline.run(store=): incremental mode — unchanged files (same MD5) are skipped, existing nodes loaded from store for the resolver; changed/new files re-extracted and saved; stale files (deleted from disk) cleaned up automatically--incremental / -iflag oncgis ingest; graceful warning + fallback if output is JSONUsage
Test plan
make pytest— 75 passedmake lint && make type-check— cleanCloses #22
🤖 Generated with Claude Code