Skip to content

⭐ Grid Operations

Jump to bottom
Terrence Daniels edited this page Sep 17, 2026 · 1 revision

A new service tracking the field-operations side of a utility — real outages and work orders — fed by a tech filing one directly through a real HTTP form, and by a genuine technical hook back into the existing Kafka pipeline: correlating meters that stop transmitting into likely outages.

Composition

  • WorkOrder — HazardType (one of DownedWire/BlownTransformer/BrokenPole/VegetationContact/EquipmentFailure/VehicleAccident/WeatherDamage/AnimalContact/SuspectedOutage), nullable MeterId/AccountId/StreetName/StreetNumber (a tech may file one against a general area, not always a specific meter), Status (Reported→Dispatched→EnRoute→OnSite→Repaired→Restored→Closed, plus Ignored from Reported only), nullable OutageId
  • Outage — StreetName, StreetNumberRangeStart/StreetNumberRangeEnd (the correlated span), Status (Suspected/Confirmed/Restored)
  • WorkOrderStatusTransitioner / WorkOrderStatusUpdater — status-transition validation lives in a plain processor class, never on the entity, matching this repo's existing anemic-POCO-plus-processor-class shape (InvoiceGenerator, ReadingProcessor)
  • DetectedAnomaly / OutageCorrelator / WorkOrderCorrelationProcessor — the correlation logic: OutageCorrelator is pure decision logic (no DB access, directly unit-testable), WorkOrderCorrelationProcessor does the actual database reads/writes based on its decision
  • UsageAnomalyConsumer — a BackgroundService subscribed to usage.anomaly.detected, resolving each anomaly's real street address via a live HTTP call to Account/Customer Service's GET /meters/{meterId} (scoped to fetch only StreetName/StreetNumber — AccountId already arrives on the Kafka event itself)
  • REST endpoints: POST /work-orders (the tech-fillable form), GET /work-orders, GET /work-orders/{id}, PATCH /work-orders/{id}/status, GET /outages, GET /outages/{id}

Detection is split across two services

The low-level signal — "this meter stopped transmitting" — is detected by Usage Aggregation, which publishes usage.anomaly.detected. Grid Operations owns the higher-level correlation: grouping quiet meters by street-number proximity and a time window into a shared Outage, keeping exactly one surviving WorkOrder per outage rather than one per affected meter. Full reasoning, including the "meter went quiet" vs. "usage dropped" signal choice and the street-number-only correlation limitation, is in docs/architecture.md.

Two real bugs found via live verification, both closed

  • #29: stopping meter-simulator with 20 real meters active and letting them all go quiet together produced 5 separate 2-meter outages plus several orphaned standalone drafts sitting inside those outages' ranges, instead of one shared work order. Root cause: the correlator only ever paired a new anomaly with the single nearest existing draft. Fixed by redesigning OutageCorrelator to extend an existing outage's range (absorbing newly-in-range orphaned drafts) or merge a whole cluster of nearby drafts into one new outage, either way keeping exactly one WorkOrder.
  • #30: on a fresh AppHost start, real leftover readings from a previous dev session got flagged as brand-new anomalies immediately. Fixed with UsageAnomalyOptions.MaxAnomalyLookbackSeconds — a reading only signals a new anomaly if it's recent enough to mean the meter just went quiet, not simply old data.

Verified live, not just built

A real AppHost run with meter-simulator intentionally stopped to force real quiet meters, confirmed against real gridoperationsdb data via psql — both the single-meter-draft case and the multi-meter correlated-outage case. The fixed correlator and the cold-start lookback bound were both re-verified live the same way after their respective fixes.

Tested

32 xUnit tests: 20 cover WorkOrderStatusTransitioner's status-transition rules (every allowed transition, several disallowed ones, and which statuses count as terminal) via [Theory]/InlineData; 3 cover WorkOrderStatusUpdater against EF Core's InMemory provider (a valid transition persists, an invalid one is rejected and leaves the status unchanged); 9 cover OutageCorrelator's pure decision logic — the single-quiet-meter case, idempotency (an existing draft or a covered street number returns AlreadyCovered), merging nearby drafts into a shared outage, and extending an existing outage's range while absorbing an orphaned draft (the exact scenario from #29).

Clone this wiki locally