Skip to content

[C7] Verify ETL-inserted rows reach the enrichment consumer via CDC #896

@jakebromberg

Description

@jakebromberg

Problem

jobs/flowsheet-etl/job.ts (bulk + incremental) inserts rows into flowsheet but does not trigger metadata enrichment for them. They rely on the daily cron to catch up. Per Epic C #1 (the metadata_status column), this means ETL-inserted rows have metadata_status DEFAULT 'pending' and the consumer should pick them up via CDC.

Today the ETL also doesn't broadcast liveFs events because it inserts via raw db.insert(flowsheet) which bypasses the BS internal-route's updateLastModified + broadcast. So even after Epic C, the ETL-inserted rows wouldn't reach the consumer via CDC unless we either (a) emit CDC events properly (already happens via PG triggers — confirm), or (b) the consumer doesn't depend on the broadcast and only on the PG trigger.

Verification + adjustment

  • Verify: ETL db.insert(flowsheet).values(...) triggers the same CDC PG trigger (shared/database/src/migrations/0045_cdc_notify_triggers.sql) that webhook/controller inserts do. (Yes, the trigger is on the table, not the path — so all writes notify. But verify.)
  • Verify: the consumer in C2, listening on CDC, receives ETL-inserted rows the same as controller-inserted ones.
  • If ETL needs additional signaling (e.g., to trigger the SSE update event), wire it.

Out of scope

  • Performance: bulk ETL inserts millions of rows on the rebuild path. The consumer can't enrich those at 50 req/min; the cron is the safety net there.

Acceptance

  • ETL-inserted rows enter the same metadata_status='pending' lifecycle as other writes.
  • Integration test: run a small ETL sync; verify the consumer picks up the new rows.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requeststatus:blockedCannot start until a dependency closes

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions