Development

Development

Prerequisites

• Go 1.25+
• Docker and Docker Compose (for integration tests and local Kafka)
• golangci-lint (for linting)
• pre-commit (optional, for git hooks)

Setup

git clone <repo-url>
cd storm-data-etl
cp .env.example .env

Install pre-commit hooks (optional):

pre-commit install

Build

make build

Output binary: bin/etl

Testing

Unit Tests

make test-unit

Runs unit tests with the race detector enabled (-race -count=1).

Coverage

make test-cover

Generates coverage.out and opens an HTML coverage report in the browser.

Integration Tests

Integration tests use testcontainers-go to spin up Kafka and verify end-to-end message flow.

make test-integration

These tests require Docker to be running and may take 1-2 minutes to start the containers.

Test Data

Sample storm report JSON files live in data/mock/. These are used by the TestStormTransformer_WithMockJSONData test to verify transformation against realistic data for all three event types (hail, tornado, wind).

Linting

make lint

Uses golangci-lint with the configuration in .golangci.yml. Enabled linters include:

• bodyclose, errorlint, gosec, noctx -- security and correctness
• gocritic, revive -- style
• misspell, unparam, errname, testifylint -- hygiene
• prealloc, unconvert -- performance
• exhaustive -- completeness
• gocyclo -- complexity (threshold: 15)

Formatting

make fmt

Runs gofmt and goimports across the project.

Pre-commit Hooks

The .pre-commit-config.yaml configures hooks that run on every commit:

• Trailing whitespace removal
• End-of-file newline
• YAML and JSON validation
• Merge conflict markers
• Large file detection
• Private key detection (gitleaks)
• gofmt and goimports
• golangci-lint
• yamllint
• markdownlint

CI Pipeline

The .github/workflows/ci.yml workflow runs on pushes and pull requests to main:

Job	What It Does
test-unit	make test-unit (unit tests with race detector)
lint	make lint (golangci-lint with the project config)
build	make build (compile check)
sonarcloud	Unit tests with coverage + SonarCloud scan

A separate release.yml workflow (triggered by CI success on main) handles versioning, GitHub releases, and Docker image publishing.

Project Conventions

• Interfaces defined by consumers: The BatchExtractor, Transformer, and BatchLoader interfaces are defined in the pipeline package (which uses them), not in the adapter packages that implement them. The Kafka adapters satisfy these interfaces directly.
• Domain logic is pure: The domain package has no infrastructure imports. Transformation functions are stateless and operate on domain types.
• Testable time: The clock package variable (via clockwork) allows tests to control time without relying on time.Sleep or wall-clock assertions.
• Structured logging: All logging uses log/slog with key-value pairs. The pipeline logs include Kafka metadata (topic, partition, offset) for traceability.
• Adapter constructor injection: All adapters (Kafka, HTTP) accept *slog.Logger via their constructors for consistent, testable logging.

Related

• System Development -- multi-repo workflow, CI conventions, and cross-service patterns
• Shared Development -- shared library development and versioning
• Architecture -- environment variables, hexagonal design, batch processing, and offset strategy
• Enrichment -- severity classification, location parsing, and enrichment rules