Marathon

Run big data changes on a live database without writing a throwaway script.

Sooner or later you have to backfill a column across millions of rows, repair some bad data, re-encrypt a field, or copy a table into a new shape. The usual answer is a one-off script that hammers the database, has no way to resume when it dies at 60%, can't be paused, and gives you no idea how far along it is.

Marathon does that job properly: it walks the table in small batches, checkpoints after every one, throttles itself so production stays healthy, and gives you a dashboard to watch and control the whole thing. Kill it halfway and it picks up where it left off.

It's a single Go binary (built on GoFr) plus a small React dashboard. Postgres and MySQL are supported. You run it yourself — your data never leaves your network.

Quick start

You need Docker and Go 1.26+.

make up      # starts Postgres, Redis, and a demo DB seeded with 1M rows
             # (the demo data is deploy/sandbox/seed-target.sql — sandbox only)
make run     # starts Marathon on :8000

Build the dashboard once and it's served straight from the binary:

cd web && npm install && npm run build
# open http://localhost:8000/app

From there: create an operation, dry-run it to preview the change, then start it and watch it go. Or drive it from the API:

# preview what would change — writes nothing
curl -X POST localhost:8000/tasks/1/dry-run

# start a run, then pause / resume / kill it
curl -X POST  localhost:8000/tasks/1/runs
curl -X PATCH localhost:8000/runs/1 -d '{"action":"pause"}'

How you point it at your data

Two ways, chosen per operation:

• SQL — give Marathon a connection string and a range UPDATE/DELETE. It runs the statement on your database, batch by batch. Use a least-privilege user that can only read the cursor column and write the target table.
• HTTP callback — Marathon reads each batch and POSTs the rows to an endpoint you host. Your service does the work (in any language) and reports back per row. Rows you reject get quarantined; the run keeps going.

The dashboard's "Connect your system" page walks through both with examples.

What you get

• Checkpointed batches, so a crash resumes from the last committed point
• A rate limit that keeps load off production (and an adaptive throttle in fleet mode)
• Dry-run with a before/after sample
• A quarantine for rows that fail, with retry
• Live progress over websockets: percent, rows/sec, ETA
• An audit log of every run and every action
• Optional API-key roles (viewer / operator / admin)
• Horizontal scale-out: many stateless workers share one run and cover for each other when one dies

How it works

Short version: a stateless control plane decides what should run and hands out work; stateless workers do the batches; Postgres holds the durable state and Redis coordinates the workers. The same binary runs on a laptop or as a fleet on Kubernetes.

The full picture, with diagrams, is in docs/ARCHITECTURE.md.

Running as a fleet

make up
make fleet-server    # control plane
make fleet-worker    # a worker — run as many as you want

Kill a worker mid-run and the control plane notices the missing heartbeat, requeues its unfinished slice, and another worker finishes it.

Tests

make test        # unit tests, no external services
make up && make itest    # integration tests against real Postgres, MySQL, Redis

The integration tests prove the things that are easy to get wrong: resume after a crash re-applies exactly one batch, three workers cover every row exactly once, and a downstream rejection quarantines the right row while the run still finishes.

Config

Set via environment or configs/.env:

Variable	What it does
HTTP_PORT	API + dashboard port (default 8000)
DB_*	Marathon's own control-store connection
REDIS_ADDR	Redis address for fleet coordination
MARATHON_KEYS	key:role,... to require API keys; unset = open
MARATHON_WEB_DIR	where the built dashboard lives (default ./web/dist)

Status

The core works and is tested end to end. Still on the list: a run-history page in the UI, a saved-connections vault, browser e2e tests, and a published image + CI.

License

Apache-2.0.