feat(hashing): add populate_hash_cache() and orcapod warm-cache CLI (ITL-489)

[kurodo3]

Summary

• Adds populate_hash_cache() utility in src/orcapod/hashing/cache_population.py that recursively walks a directory, skips files below a configurable size threshold (default 500 MB), checks the SQLite hash cache first, and hashes+caches qualifying files using the existing FileHasher and SqliteHashCacher from ITL-472.
• Adds CachePopulationStats frozen dataclass with full throughput accounting: hashed, already_cached, skipped_small, errors, total_bytes_hashed, total_duration, avg_hashing_speed.
• Adds orcapod warm-cache PATH [--min-size FLOAT] [--db-path PATH] [--algorithm TEXT] [--buffer-size INT] [--workers INT] CLI subcommand via typer>=0.12.
• Registers orcapod as an installable entry point in [project.scripts].
• Exports CachePopulationStats and populate_hash_cache from orcapod.hashing.
• Concurrent hashing via ThreadPoolExecutor — max_workers: int = 4 (default) dispatches per-file stat/cache-lookup/hash/write to worker threads; main thread owns all DFS traversal and DB-file exclusion. SqliteHashCacher is safe to share across threads (uses threading.local() connections internally); FileHasher is stateless. Pass max_workers=1 to restore serial behaviour.
• 19 unit tests for the core function (including 3 concurrency tests), 6 CLI tests (including --workers smoke test).

Test plan

• uv run pytest tests/test_hashing/test_cache_population.py -v — all 19 tests pass
• uv run pytest tests/test_cli/test_warm_cache.py -v — all 6 tests pass
• uv run pytest tests/ -x -q — full suite passes
• uv run orcapod warm-cache --help — shows PATH + all options including --workers INT with default 4
• uv run orcapod warm-cache /tmp --min-size 0 --db-path /tmp/test.db --workers 4 — runs cleanly, prints summary

Fixes ITL-489

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.78014% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/orcapod/hashing/cache_population.py	87.73%	13 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Adds a new hash-cache “warming” utility and CLI so large files can be pre-hashed and stored in the existing SQLite cache (with optional concurrent hashing to improve throughput on NAS/HDD workloads).

Changes:

• Introduces populate_hash_cache() + CachePopulationStats for recursive directory hashing/caching with throughput stats and max_workers concurrency.
• Adds orcapod warm-cache Typer CLI (entry point + --workers) wrapping the cache-population utility.
• Adds unit/CLI test coverage plus design/plan documentation and updates dependencies/lockfile.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
uv.lock	Locks the newly added typer dependency.
pyproject.toml	Adds typer>=0.12 and registers the orcapod CLI entry point.
src/orcapod/hashing/cache_population.py	New cache population implementation with concurrent hashing and stats accounting.
src/orcapod/hashing/init.py	Exports CachePopulationStats and populate_hash_cache from orcapod.hashing.
src/orcapod/cli/init.py	Adds the Typer app root and registers the warm-cache command.
src/orcapod/cli/warm_cache.py	Implements orcapod warm-cache including --workers and summary output.
tests/test_hashing/test_cache_population.py	Adds comprehensive unit tests, including concurrency behavior and export checks.
tests/test_cli/init.py	Marks CLI tests as a package.
tests/test_cli/test_warm_cache.py	Adds CLI tests for help text, basic run, cache hits, defaults, and --workers.
superpowers/specs/2026-07-03-itl-concurrent-warm-cache-design.md	Design doc for concurrent hashing changes (max_workers / --workers).
superpowers/specs/2026-07-03-itl-489-warm-cache-design.md	Design doc for warm-cache utility + CLI.
superpowers/plans/2026-07-03-itl-concurrent-warm-cache.md	Implementation plan/checklist for concurrency changes.
superpowers/plans/2026-07-03-itl-489-warm-cache.md	Implementation plan/checklist for warm-cache feature set.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kurodo3]

Addressing review comment on sidecar path construction (#208 (comment)):

Fixed in 66e192e. Replaced all three with_suffix(suffix + "...") calls with plain string concatenation:

_db_str = str(_db_resolved)
_excluded: frozenset[Path] = frozenset([
    _db_resolved,
    Path(_db_str + "-wal"),
    Path(_db_str + "-shm"),
    Path(_db_str + "-journal"),
])

This handles DB paths with no file extension (where with_suffix("-wal") would raise ValueError) and also adds the previously missing rollback-journal sidecar (-journal). A new test test_db_path_without_extension in TestTraversal confirms a no-extension DB path works end-to-end.

---

Addressing review comment on max_workers validation (#208 (comment)):

Fixed in 66e192e. populate_hash_cache() now raises ValueError eagerly before creating the executor:

if max_workers < 1:
    raise ValueError(f"max_workers must be >= 1, got {max_workers!r}")

The CLI (warm_cache()) also validates before proceeding, printing Error: --workers must be at least 1 to stderr and exiting with code 1. Two tests were added: test_max_workers_zero_raises (API) and test_workers_zero_exits_nonzero (CLI).

---

Summary of all changes in this review round:

All fixes are in a single commit: 66e192ec

Issue	Fix	Tests added
with_suffix() ValueError on extension-less DB paths	String concatenation for sidecar paths	test_db_path_without_extension
Missing -journal sidecar in exclusion set	Added Path(str(db) + "-journal") to _excluded	(covered by above)
max_workers < 1 raises unhelpful error from executor	Early ValueError in populate_hash_cache() + CLI guard	test_max_workers_zero_raises, test_workers_zero_exits_nonzero

Total tests: 33 (up from 30), both source modules still at 100% line coverage.

[eywalker]

Add an issue to follow up and allow for other units to be specified e.g. ("1 GB", "20 B", "30MiB")

[kurodo3]

Created Linear issue PLT-1913 to track this: https://linear.app/enigma-metamorphic/issue/PLT-1913/warm-cache-support-human-readable-size-strings-for-min-size-option

The issue covers supporting strings like "1 GB", "500 MB", "30 MiB" alongside bare floats (which will keep their MB interpretation for backward compatibility).

[eywalker]

Shouldn't this be UPath? The root can be specified as UPath

[kurodo3]

Fixed in 64a7ffc. Changed root = Path(path) to root = UPath(path) and added from upath import UPath to the imports. The CLI now uses UPath consistently for path handling.

[eywalker]

this should be explicitly path: PathLIke | UPath to accept workingon UPath and internally we should consistently use UPath handling.

[kurodo3]

Fixed in 64a7ffc. Changed path: PathLike to path: PathLike | UPath. While UPath is technically already covered by os.PathLike (which UPath implements), making it explicit matches the existing pattern on the db_path parameter and makes the intent clear at a glance. Docstring updated accordingly.

[eywalker]

As we are accepting UPath here, shouldn't the type hint clearly mention that? I don't think os.PathLIke maps to UPath, does it?

[kurodo3]

Fixed together with the comment on line 93 (64a7ffc). The path parameter is now PathLike | UPath — explicit about UPath support, matching the db_path: UPath | PathLike | None pattern already used elsewhere in the same signature.

[kurodo3]

Review round summary (eywalker, 64a7ffc)

Comment	Action
Allow human-readable size strings ("1 GB", "30MiB") for --min-size	Created PLT-1913 to track this as a follow-up. No code change in this PR.
root = Path(path) in warm_cache.py should use UPath	Changed to UPath(path). Added from upath import UPath import.
path: PathLike in populate_hash_cache() should include UPath explicitly	Changed to path: PathLike | UPath. Docstring updated.
Same UPath type-hint question re: db_path context	Resolved by the same PathLike | UPath fix above.

All 33 tests continue to pass.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

[kurodo3]

Response to Copilot review (round 2) — commit 112422e

Comment 1: Unbounded futures list → O(num_files) memory

Fixed. Replaced the unbounded futures list with a bounded pending set driven by wait(FIRST_COMPLETED). When len(pending) >= max_workers * 4, the traversal loop blocks until at least one future completes before submitting the next file. This keeps peak memory O(max_workers) rather than O(num_files) and provides natural backpressure when hashing is slower than traversal. The remaining futures are drained with as_completed once the DFS stack is exhausted.

Comment 2: Size filtering inside worker → futures created for every tiny file

Fixed. Moved resolve(), stat(), and the size check onto the traversal thread. Files below min_size_bytes now increment skipped_small directly and are never submitted to the executor — no Future object is created for them. As a bonus, qualifying files no longer pay a second stat() inside _hash_one: the os.stat_result obtained on the traversal thread is passed directly to the worker, which uses it to build the cache key (mtime_ns + size) without any duplicate syscall. The _hash_one signature changed accordingly (takes resolved: Path, file_stat: os.stat_result instead of entry: UPath, min_size_bytes: int) and its return type no longer includes "skipped_small".

All 33 targeted tests pass (22 unit + 11 CLI).

[kurodo3]

Fixed in af2005e. _hash_one now takes resolved: UPath instead of Path. The _excluded frozenset is also typed as frozenset[UPath] (wrapping each member in UPath(...)), so entry.resolve() — which natively returns a UPath — can be compared directly without the Path() cast that was previously required to reconcile types. UPath is now consistent throughout: stack, entries, resolved paths, exclusion set, and the worker signature.