Async Bulk Export Service

A Go backend service for exporting large relational datasets to CSV asynchronously. Built around a bounded worker pool, backpressure, and graceful shutdown, with PostgreSQL for both the data and job tracking.

The problem

Generating a large export inside an HTTP request doesn't scale. A query across millions of rows can take longer than a client is willing to wait, and it blocks the server from handling other requests in the meantime. This service separates "accept the request" from "do the work": exports run in the background through a worker pool, and the client polls for status.

Architecture

Client
  |  POST /exports {seller_id, date_from, date_to}
  v
HTTP API -- writes job (pending) --> PostgreSQL
  |
  |  pushes job onto bounded channel
  v
Worker Pool (fixed-size goroutines)
  |
  |-- picks up job
  |-- runs export query
  |-- streams result to CSV
  \-- updates job status in PostgreSQL

Client
  |  GET /exports/:id            poll status
  |  GET /exports/:id/download   fetch finished file
  |  GET /stats                  jobs completed/failed since startup

Stack

• Go: HTTP API, worker pool, concurrency
• PostgreSQL: relational data and job state
• Docker Compose: local Postgres environment

Design decisions

• Bounded worker pool. A fixed number of goroutines process jobs, not one goroutine per request. Keeps DB connections and memory usage predictable under load.
• Backpressure. The job queue has a fixed capacity. Once full, new requests get a 503 immediately instead of queuing indefinitely.
• Graceful shutdown. On shutdown, the service stops accepting new jobs but lets in-flight exports finish before exiting.
• Fail fast on no data. If a seller has no matching orders, the job fails with a clear message instead of producing an empty file.
• Durable job state. Job status lives in Postgres, not just memory, so it survives restarts.

Data

The database is seeded with synthetic but relationally realistic data: sellers, customers, products, orders, and order items, at a scale of a few million rows. Seller order volume is deliberately skewed, a small number of sellers hold most of the order history, so query performance is a real problem to solve, not a hypothetical one.

What's documented in detail

• docs/query_optimization.md: the export query went from ~160ms unindexed, to ~228ms with a naive index (worse), to ~108ms with a covering index. Includes the write-side tradeoff.
• docs/race_condition.md: a real data race in a shared counter, caught with go test -race, fixed with sync/atomic.
• docs/load_testing.md: backpressure and throughput under burst, extreme, and realistic load.

Local setup

# 1. Copy environment config and fill in values
cp .env.example .env

# 2. Start PostgreSQL (applies migrations automatically on first run)
make up

# 3. Seed the database with sample data
make seed

# 4. Run the server
make run

Commands

Command	Description
make up	Start PostgreSQL (detached)
make down	Stop PostgreSQL (keeps data)
make seed	Run the seed script
make reseed	Wipe data, restart Postgres, and reseed from scratch
make run	Run the server
make logs	Tail logs from all running services
make psql	Open a psql shell into the running Postgres container

API

POST   /exports              create an export job
GET    /exports/:id          check job status
GET    /exports/:id/download download the finished CSV
GET    /stats                jobs completed/failed since startup

Project structure

export-service/
├── cmd/server/       entrypoint
├── internal/
│   ├── api/           HTTP handlers and routing
│   ├── worker/        worker pool and export logic
│   ├── jobs/           job model and Postgres queries
│   ├── queue/           bounded job queue
│   ├── stats/            in-memory operational counters
│   ├── db/                 database connection
│   └── config/               environment/config loading
├── migrations/          numbered SQL schema migrations
├── seed/                 data seeding script
└── docs/                  detailed write-ups (linked above)