Add more granular collection step checkpoints

[ppinchuk]

Now if collection times our during web crawl, the search engine search does not have to be re-done.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.41667% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.60%. Comparing base (31d7bec) to head (c5b27ce).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
compass/pipeline/collection/base.py	74.35%	5 Missing and 5 partials ⚠️
compass/pipeline/collection/dedupe.py	75.00%	1 Missing and 1 partial ⚠️
compass/pipeline/jurisdiction.py	86.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #517      +/-   ##
==========================================
+ Coverage   63.54%   63.60%   +0.06%     
==========================================
  Files          78       78              
  Lines        7367     7425      +58     
  Branches      741      754      +13     
==========================================
+ Hits         4681     4723      +42     
- Misses       2533     2545      +12     
- Partials      153      157       +4     

Flag	Coverage Δ
unittests	63.60% <85.41%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds step-level checkpointing to the collection workflow so that jurisdictions can resume collection from the last completed step (e.g., avoiding re-running search-engine discovery if a later website crawl times out). It updates persistence metadata to track per-step completion and adjusts unit/integration tests accordingly.

Changes:

• Add per-step shard writes and resume logic in DocumentCollection (load existing shard, skip completed steps, write after each step).
• Update persistence/manifest structures to record completed_step_document_counts per jurisdiction and completed_step_document_totals in the manifest.
• Expand unit/integration coverage for checkpoint resumption, restored artifacts, and updated persistence fields.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/python/unit/pipeline/test_pipeline_jurisdiction.py	Adds coverage that collection delegates to the workflow and that loading a shard restores jurisdiction_website.
tests/python/unit/pipeline/test_pipeline_collection.py	New unit tests covering per-step checkpoint writes, skipping completed steps, resuming with persisted docs, and legacy shard behavior.
tests/python/unit/pipeline/test_pipeline_collection_persistence.py	Updates persistence tests for new step-count fields and adds coverage for preserving restored docs/step counts and omitting “checkpoint-only” jurisdictions from manifest.
tests/python/unit/pipeline/test_pipeline_collection_dedupe.py	Updates dedupe API usage and adds a test ensuring restored artifacts are preserved while merging provenance.
tests/python/integration/test_integrated_pipeline_orchestration.py	Extends integration assertions to validate completed-step counts/totals and resumed collection behavior.
compass/pipeline/jurisdiction.py	Refactors collection to rely on workflow checkpointing, adds shard read/write helpers, and moves relative-path handling to a cached property.
compass/pipeline/coordinator.py	Removes explicit relative_to propagation now that relative-path handling is encapsulated in the jurisdiction workflow.
compass/pipeline/collection/persistence.py	Adds step completion metadata fields and introduces _store_docs_as_needed to avoid rewriting already-persisted artifacts.
compass/pipeline/collection/dedupe.py	Updates dedupe keying and supports adding restored docs without requiring a step/jurisdiction name.
compass/pipeline/collection/base.py	Adds resume/skip behavior and per-step checkpoint writes within the collection execution flow.