Skip to content

Coordinate local H5 workers with catalog requests#986

Draft
anth-volk wants to merge 3 commits into
codex/h5-migration-pr8-worker-servicefrom
codex/h5-migration-pr9-coordinator
Draft

Coordinate local H5 workers with catalog requests#986
anth-volk wants to merge 3 commits into
codex/h5-migration-pr8-worker-servicefrom
codex/h5-migration-pr9-coordinator

Conversation

@anth-volk
Copy link
Copy Markdown
Collaborator

Fixes #985

Depends on #973. This PR is based on codex/h5-migration-pr8-worker-service so the diff only contains the PR 9 coordinator work.

Summary

  • Move regional local H5 request enumeration onto USAreaCatalog and typed AreaBuildRequest payloads.
  • Query the staged target database for congressional district targets and use the catalog to define the regional release shape.
  • Partition typed weighted requests and send workers --requests-json payloads while keeping the legacy work_items_override path available.
  • Normalize worker responses before coordinator error handling so protocol errors, worker failures, transport failures, and nonfatal issues are distinct.

Testing

  • make lint
  • uv run --no-sync pytest tests/unit/build_outputs/test_area_catalog.py tests/unit/build_outputs/test_partitioning.py tests/unit/build_outputs/test_worker_responses.py tests/unit/test_modal_local_area.py -q
  • uv run --no-sync --with pyyaml python scripts/run_quality_guards.py
  • uv run --no-sync --with pyyaml --with pytest pytest tests/unit/test_pipeline_docs_extractor.py tests/unit/test_pipeline_doc_guards.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant