feat(hashing): add file hash caching with SQLite backend (ITL-472)#204
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull request overview
Adds an opt-in, persistent file-hash cache layer to Orcapod’s file hashing pipeline to dramatically reduce repeated hashing costs by memoizing ContentHash results keyed by (resolved path, mtime_ns, size).
Changes:
- Introduces a generic
CacherProtocol[K, V]and file-specific cache keyFileHashKey. - Renames
BasicFileHasher→FileHasher, and addsCachedFileHasherthat composes aFileContentHasherProtocolwith a typed cache backend. - Implements
InMemoryHashCacher(tests) andSqliteHashCacher(persistent WAL/WITHOUT ROWID), plusenable_file_hash_caching()to wire caching into the default context; adds tests and a benchmark script.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_hashing/test_hash_cachers.py | Adds coverage for in-memory and SQLite cache backends plus enable_file_hash_caching() wiring behavior. |
| tests/test_hashing/test_file_hashers.py | Updates tests for FileHasher rename and typed-cache-based CachedFileHasher. |
| tests/test_hashing/test_file_handler.py | Updates handler tests to use FileHasher instead of BasicFileHasher. |
| superpowers/specs/2026-07-02-itl-472-file-hash-caching-design.md | Design spec documenting the caching architecture, schema, and constraints. |
| superpowers/plans/2026-07-02-itl-472-file-hash-caching.md | Implementation plan and checklist for delivering ITL-472. |
| src/orcapod/protocols/hashing_protocols.py | Adds generic CacherProtocol[K, V] to formalize typed cache backends. |
| src/orcapod/hashing/semantic_hashing/builtin_handlers.py | Updates fallback hasher to FileHasher(sha256) and related docstrings. |
| src/orcapod/hashing/hash_cachers.py | Adds InMemoryHashCacher and SqliteHashCacher implementations. |
| src/orcapod/hashing/file_hashers.py | Adds FileHashKey, renames BasicFileHasher → FileHasher, and refactors CachedFileHasher to use typed caching. |
| src/orcapod/hashing/init.py | Updates public hashing exports to include new caching-related symbols and removes BasicFileHasher. |
| src/orcapod/contexts/data/v0.1.json | Updates default context config to reference FileHasher. |
| src/orcapod/contexts/init.py | Adds enable_file_hash_caching() to wrap the default FileHandler with a SQLite-backed cache layer. |
| bench/bench_file_hasher_cache.py | Adds a benchmark script to measure cached vs uncached file hashing performance. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ``~/.orcapod/file_hash_cache.db`` or the | ||
| ``ORCAPOD_HASH_CACHE_DB`` environment variable. | ||
| """ | ||
| from pathlib import Path # noqa: F811 |
There was a problem hiding this comment.
Fixed — removed the from pathlib import Path import and the # noqa: F811 suppression from inside enable_file_hash_caching(). The annotation "Path | None" is already a quoted string so no runtime import is needed.
| while isinstance(base_hasher, CachedFileHasher): | ||
| base_hasher = base_hasher.file_hasher |
There was a problem hiding this comment.
Fixed — the unwrap loop now calls cacher.close() on each discarded CachedFileHasher layer if the cacher exposes a close() method:
while isinstance(base_hasher, CachedFileHasher):
cacher = base_hasher.cacher
if hasattr(cacher, "close"):
cacher.close()
base_hasher = base_hasher.file_hasherThis closes any thread-local SQLite connections held by SqliteHashCacher instances before they are discarded, preventing file descriptor leaks.
Review round 1 — all comments addressedAll 4 Copilot review comments resolved in commit 98f3554.
432 hashing tests still pass after these changes. |
Review round 1 — all 4 comments addressedAll changes in commit
All 432 hashing tests pass after these changes. |
| from orcapod.protocols import hashing_protocols as hp | ||
| from orcapod.protocols import semantic_types_protocols as sp | ||
|
|
||
| _logger = _logging.getLogger(__name__) |
There was a problem hiding this comment.
Good catch — no reason for the underscore prefix. Renamed to plain logging / logger = logging.getLogger(__name__) to match the convention used everywhere else in the codebase (e.g. builtin_handlers.py).
Review round 2 — addressedSingle commit
|
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… and CachedFileHasher Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Key; update CachedFileHasher to CacherProtocol
…ryHashCacher, SqliteHashCacher
…er into default context
- Close existing cachers (if closeable) when unwrapping in enable_file_hash_caching() - Remove unused 'threading' import from test_hash_cachers.py - Remove unused 'logging'/'logger' from hash_cachers.py - Remove unused 'from pathlib import Path' and noqa comment from enable_file_hash_caching() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
35e08a0 to
31b98de
Compare
…) in SqliteHashCacher schema
Review round 3 — addressedSingle commit
All 18 hash cacher tests pass after this change. |
Summary
CacherProtocol[K, V]— a generic two-method (get/put) protocol for typed cachesBasicFileHasher→FileHasher; addsFileHashKey(frozen dataclass withpath: UPath,mtime_ns,size) as the typed cache lookup keyCachedFileHasher— a decorator that wraps anyFileContentHasherProtocolwith aCacherProtocol[FileHashKey, ContentHash], reducing repeat hashing of the same file to a sub-millisecond DB lookupInMemoryHashCacher(dict-backed, for tests) andSqliteHashCacher(WAL mode,WITHOUT ROWID, thread-local connections, BLOB storage viaContentHash.to_prefixed_digest())enable_file_hash_caching(db_path=None)toorcapod.contexts— one call at startup wiresCachedFileHasher+SqliteHashCacherinto the default context; warns and unwraps on double-callCloses ITL-472
Test Plan
uv run pytest tests/test_hashing/test_hash_cachers.py— allInMemoryHashCacher,SqliteHashCacher, andenable_file_hash_caching()tests passuv run pytest tests/test_hashing/test_file_hashers.py tests/test_hashing/test_file_handler.py— all file hasher tests pass with renamedFileHasheruv run pytest tests/ -q— full suite: 3992 passed, 56 skipped, 6 xfaileduv run python bench/bench_file_hasher_cache.py— sub-millisecond cache hit confirmed🤖 Generated with Claude Code