Skip to content

feat: decouple ingestion pipeline jobs - #223

Merged
mdiapenabc merged 3 commits into
devfrom
feat/decouple-ingestion-jobs
Apr 27, 2026
Merged

feat: decouple ingestion pipeline jobs#223
mdiapenabc merged 3 commits into
devfrom
feat/decouple-ingestion-jobs

Conversation

@mdiapenabc

Copy link
Copy Markdown
Collaborator

Description

Splits the monolithic data ingestion pipeline into three independently runnable jobs. Previously, INGEST_DATA chained into eligibility, which chained into auto-batching and ICM sync-back. Each step now runs as a standalone job with its own entrypoint, so they can be triggered independently from OpenShift or manually.

What Changes

  • INGEST_DATA now stops at staging tables: fetches ICM + MIS data in parallel, no eligibility chain
  • RUN_ELIGIBILITY runs independently: upserts master contacts table, then syncs flagged contacts back to ICM
  • AUTO_BATCH is a new independent job: inds all eligible / not_eligible_in_pay contacts and adds them to the pending batch, then syncs back to ICM
  • New entrypoint: backend/src/jobs/entrypoints/auto-batch.ts (runnable as job:auto-batch)
  • Fixed eligibility threshold to use last RUN_ELIGIBILITY success timestamp instead of INGEST_DATA
  • Removed dead autoBatchContacts() method and autoBatched field from EligibilityService (logic now lives exclusively in AutoBatchService)

@mdiapenabc mdiapenabc changed the title decouple ingestion pipeline jobs feat: decouple ingestion pipeline jobs Apr 27, 2026
@mdiapenabc
mdiapenabc merged commit 10ca911 into dev Apr 27, 2026
13 of 15 checks passed
@mdiapenabc
mdiapenabc deleted the feat/decouple-ingestion-jobs branch April 27, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants