Repository navigation
Architecture
System design, tradeoffs, and improvement roadmap for the storm data pipeline.
Three services, two Kafka topics, one database. Data flows left to right through a collector → ETL → API pipeline. Clients query the GraphQL API on the far right.
The system runs on a local Kubernetes cluster (minikube) with resources spread across two namespaces.
kafka -- Strimzi operator and all Kafka-related custom resources (Kafka cluster, KafkaNodePool, KafkaTopic). Strimzi requires its operator and managed resources in the same namespace for RBAC scoping.
storm-data -- All application workloads (collector, ETL, API, mock server, dashboard), PostgreSQL, Prometheus, and Kafka UI. Services reference the Kafka broker via its cross-namespace DNS name: kafka-kafka-bootstrap.kafka.svc.cluster.local:9092.
| Resource | Kind | Namespace | Description |
|---|---|---|---|
kafka |
Kafka (Strimzi CR) | kafka | Single-node KRaft-mode Kafka 4.1.1 cluster |
broker |
KafkaNodePool (Strimzi CR) | kafka | Broker pool: 1 replica, broker+controller roles, 1Gi PVC |
raw-weather-reports |
KafkaTopic (Strimzi CR) | kafka | 1 partition, 1 replica |
transformed-weather-data |
KafkaTopic (Strimzi CR) | kafka | 1 partition, 1 replica |
postgres |
StatefulSet | storm-data | PostgreSQL 16, 1 replica, 1Gi PVC, headless Service |
postgres-credentials |
Secret | storm-data | POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB |
collector |
Deployment | storm-data | 1 replica, ConfigMap for Kafka/URL config |
etl |
Deployment | storm-data | 1 replica, ConfigMap for Kafka topics and batch config |
api |
Deployment | storm-data | 1 replica, ConfigMap + Secret (DATABASE_URL) |
mock-server |
Deployment | storm-data | 1 replica, local image (imagePullPolicy: Never) |
dashboard |
Deployment | storm-data | nginx serving HTML from ConfigMap volume |
prometheus |
Deployment | storm-data | Scrapes collector, ETL, and API /metrics endpoints |
kafka-ui |
Deployment | storm-data | Web UI for topic inspection |
Each Deployment has a corresponding ClusterIP Service for in-cluster DNS resolution. PostgreSQL uses a headless Service (clusterIP: None) for stable pod DNS (postgres-0.postgres.storm-data.svc).
Kafka is managed by the Strimzi operator rather than raw StatefulSets. The operator watches for Kafka, KafkaNodePool, and KafkaTopic custom resources and reconciles the actual Kafka broker pods, storage, and topic configuration.
This replaces the kafka-init container from the Docker Compose setup. Previously, an init container ran kafka-topics.sh --create on startup to ensure topics existed. Now, topics are declared as KafkaTopic CRs in k8s/kafka/, and Strimzi's entity operator creates and manages them. Adding a topic means adding a YAML file, not modifying a startup script.
The Strimzi-managed Kafka runs in KRaft mode (no ZooKeeper) with a single combined broker/controller node. The bootstrap service is exposed at kafka-kafka-bootstrap.kafka.svc.cluster.local:9092 -- the naming convention is {cluster-name}-kafka-bootstrap.
Application manifests are packaged as a Helm chart in helm/storm-data/. Each service has a single template file containing its Deployment, Service, ConfigMap, and Secret resources.
-
values.yaml-- Default configuration shared across environments. -
values-dev.yaml-- Dev overrides: local mock-server image withimagePullPolicy: Never, loaded into minikube's Docker daemon viaeval $(minikube docker-env). -
values-ci.yaml-- CI overrides: pins published Docker Hub image tags for the three application services.
Kafka resources (in k8s/kafka/) are applied separately to the kafka namespace via the Makefile, outside the Helm chart. The dashboard HTML ConfigMap is also created by the Makefile before the Helm install, rather than being bundled in the chart.
Environment-specific changes are expressed as Helm values file overrides (helm upgrade --install -f values-dev.yaml) rather than Kustomize patches.
Each service is an independent Git repository with its own CI, dependencies, and release cycle.
Why: Services are written in different languages (TypeScript, Go) with different toolchains. Independent repos allow independent deployments and avoid coupling release schedules. Each repo has its own Docker Compose for local development of that service in isolation.
Tradeoff: Cross-service changes (like adding a new field to the Kafka message schema) require coordinated PRs across repos. The storm-data-system repo mitigates this by providing a unified E2E test that validates the full pipeline.
Alternative considered: A monorepo with shared tooling. Would simplify cross-cutting changes but introduces coupling between TypeScript and Go toolchains.
Services communicate exclusively through Kafka topics. There are no synchronous service-to-service calls.
Why: Decouples producers from consumers. The collector can publish without knowing (or waiting for) downstream processing. The ETL can be restarted, scaled, or replaced without affecting the collector. Kafka provides durability, replay capability, and natural backpressure.
Tradeoff: Added infrastructure complexity. Kafka requires careful configuration (replication, retention, partitioning) and consumes significant memory (~512MB--1GB). For the current data volume (hundreds of records per day), this is overprovisioned.
Alternative considered: Direct HTTP calls between services, or a simpler queue like Redis Streams. Would reduce infrastructure but sacrifice decoupling and replay capability.
The ETL and API consumers use manual offset commits. Offsets are committed only after successful processing and writing.
Why: Prevents data loss. If a consumer crashes mid-processing, the message is redelivered on restart.
Tradeoff: Duplicate processing is possible. The API handles this with INSERT ... ON CONFLICT (id) DO NOTHING using deterministic IDs (SHA-256 hash of event_type+state+lat+lon+time+magnitude). The ETL's transform is stateless and idempotent, so duplicates produce identical output.
Event IDs are SHA-256 hashes of event_type|state|lat|lon|time|magnitude. The same raw event always produces the same ID.
Why: Enables idempotent writes at every stage. The API's upsert (ON CONFLICT DO NOTHING) naturally deduplicates. No distributed ID generation needed.
Tradeoff: Two genuinely different events with identical event type, state, coordinates, timestamp, and magnitude would collide. In practice, NOAA data has sufficient uniqueness in these fields. Adding the Comments field to the hash would reduce collision risk at the cost of sensitivity to comment edits.
The collector fetches NOAA CSVs on a schedule (daily by default) rather than streaming in real time.
Why: NOAA publishes daily CSV files, not a real-time stream. A cron schedule matches the data source's cadence. The collector also runs once immediately on startup for faster feedback during development.
Tradeoff: Data freshness is limited to the cron interval. For storm data used in analysis (not real-time alerting), daily collection is sufficient.
The ETL service defines BatchExtractor, Transformer, and BatchLoader interfaces. Kafka adapters implement these interfaces, and the pipeline orchestrates them.
Why: Domain logic (parsing, enrichment, severity classification) is completely isolated from infrastructure. Tests can substitute in-memory implementations. The Kafka consumer could be swapped for a file reader or HTTP endpoint without touching business logic.
Tradeoff: More files and indirection for what is essentially a single pipeline. For a small service, this is arguably over-engineered. The payoff comes when adding new enrichment steps or swapping infrastructure.
The API defines the GraphQL schema in .graphqls files and generates Go code from it.
Why: The schema is the contract. Frontend developers can read the schema without understanding Go. Code generation eliminates boilerplate and ensures the implementation matches the schema.
Tradeoff: Schema changes require running go generate, and the generated code can be verbose. Runtime reflection-based approaches (like graphql-go) are more flexible but lose compile-time safety.
The ETL skips malformed messages (logs a warning, commits the offset, continues). The API does the same.
Why: A single bad message should not block the entire pipeline. Logging the error provides visibility for investigation.
Tradeoff: Silently skipping messages can hide data quality issues. A dead letter queue (DLQ) would be better -- see improvements below.
For the complete data model and message shapes at each stage, see the ETL Enrichment rules and the API Architecture database schema.
Ingest volume is low (hundreds of records per day during storm season). The collector and ETL are massively overprovisioned and don't need to scale. The improvements below focus on the surfaces that matter: API query performance, storage scalability, and developer experience.
Schema registry -- Introduce Avro or Protobuf schemas with a Confluent Schema Registry (or Buf). Currently the Kafka message format is an implicit JSON contract between three repos. A schema registry would catch breaking changes at publish time rather than at consumer parse time.
OpenTelemetry tracing -- Add distributed tracing across all three services. Each Kafka message would carry a trace context header, enabling end-to-end latency visualization from ingest to query response.
Alerting on data lag -- The API already exposes dataLagMinutes via GraphQL. Add Prometheus alerting rules for when data lag exceeds a threshold (e.g., 2 hours during storm season).
API response caching -- Add an in-memory or Redis cache in front of PostgreSQL for frequently-queried time ranges and aggregations. Storm data is append-only, so cache invalidation only needs to happen on new ingestion batches. This is the first scaling lever for the read-heavy API workload.
Read replicas -- Add PostgreSQL read replicas to scale query throughput. The API's read-heavy workload (GraphQL queries) is a natural fit for read replicas, while the single writer (Kafka consumer) continues targeting the primary.
PostgreSQL partitioning -- Partition the storm_reports table by event_time (monthly or yearly). At current ingest rates the dataset grows slowly and existing indexes handle query performance well. Partitioning becomes relevant if retention spans many years or query patterns shift toward large time-range scans.
GraphQL subscriptions -- Add WebSocket-based subscriptions so clients can receive updates when new storm reports are ingested. gqlgen supports subscriptions natively. The current data cadence is daily batch collection, so subscriptions would primarily benefit workflows that poll for fresh data after each ingest cycle rather than true real-time streaming.
Vector tile serving -- The current query layer (dynamic filters, bounding box pre-filter, attribute filtering) is structurally similar to a vector tile server's per-tile spatial query. Migrating to PostGIS (see Haversine scaling note above) would enable ST_AsMVT for native Mapbox Vector Tile generation directly from SQL:
SELECT ST_AsMVT(tile, 'storms')
FROM (
SELECT ST_AsMVTGeom(geom, ST_TileEnvelope(z, x, y)) AS geom,
event_type, severity, magnitude
FROM storm_reports
WHERE geom && ST_TileEnvelope(z, x, y)
AND event_type = ANY($1)
AND event_time BETWEEN $2 AND $3
) AS tileThe existing querybuilder.go filter logic would translate to tile-coordinate bounding boxes (z/x/y → lat/lon envelope) with the same attribute filters. Client-side rendering (Mapbox GL JS or MapLibre) would replace server-side serialization, offloading geometry rendering to the browser's GPU. For user-defined filter criteria, the GraphQL API could accept filter definitions and return tile endpoint URLs, keeping the schema-first contract while enabling dynamic map layers.
In production, vector tile serving would likely be a dedicated service — tile rendering has different scaling and caching characteristics (bursty tile requests on pan/zoom, aggressive CDN cacheability) than the GraphQL data API. The spatial query patterns and filter logic from the API would transfer directly, with shared components (config, observability, health endpoints) pulled from storm-data-shared.
ML enrichment -- Exploratory. The current rule-based severity classification serves existing use cases well. A machine learning step in the ETL pipeline (damage estimation, storm path prediction, or severity classification beyond thresholds) could add value but is not driven by a current user need. Worth revisiting if the dataset or user base grows to warrant it.
- Collector Architecture -- CSV fetching, retry strategy, and Kafka publishing
- ETL Architecture -- hexagonal design, enrichment pipeline, and offset strategy
- API Architecture -- GraphQL resolvers, store layer, and query protection
- Shared Architecture -- package design and interface contracts
- Development -- running the stack, E2E tests, and conventions