rust-job-queue-api-worker-system

Copyright 2026 infinityabundance

Try it now (no install required)

Click the Open in Colab badge above.

1. Open the Colab notebook
2. Click ► Run all

---

A compact, production-shaped Rust job queue: Axum HTTP API, Tokio workers, Postgres SELECT ... FOR UPDATE SKIP LOCKED dequeue, retries with decorrelated jitter, idempotency, cooperative cancellation, OpenAPI, Prometheus metrics, Docker Compose, and reproducible tests/benchmarks.

Single Cargo package, two feature flags, two service binaries, and a one-shot migration runner.

For hiring reviewers

If you have 90 seconds:

1. Inspect src/queue.rs for the Postgres FOR UPDATE SKIP LOCKED dequeue path.
2. Inspect tests/concurrency_two_workers.rs for the 200-job / 8-worker no-duplicate-processing test.
3. Read docs/tradeoffs.md for scope decisions and production omissions.
4. Visit /docs after docker compose up to see the OpenAPI / Swagger UI.
5. Run cargo test --all-features --locked if Docker is available.

What this demonstrates

Area	Evidence in this repo
Concurrent worker correctness	A 200-job / 8-worker integration test validates no duplicate processing in that bounded SKIP LOCKED scenario, using a separate transactional processed_log table.
API design	Axum routes, typed DTOs, idempotency-key handling, status filters, OpenAPI JSON, and Swagger UI.
Reliability semantics	Retry scheduling, stale-lock recovery, cooperative cancellation, DB constraints, and documented trade-offs.
Operations	Docker Compose, migration binary, structured tracing, API metrics, worker metrics, health check, and runbook.
Release hygiene	Feature-gating probes, MSRV check, docs build with warnings denied, RustSec audit, and cargo publish --dry-run in CI.

Evidence

• cargo test --all-features --locked: 46 tests in the current suite (12 lib unit tests, 33 integration tests, 1 doctest). Integration tests require Docker because they use testcontainers Postgres.
• CI workflow: ci.yml.
• Architecture notes: docs/architecture.md.
• API reference: docs/api.md.
• Operational runbook: docs/runbook.md.
• Trade-offs and omissions: docs/tradeoffs.md.
• Colab smoke/repro notebook: notebooks/rust_job_queue_api_worker_system_colab.ipynb.

Quickstart

docker compose up                       # boots postgres + migrations + api + worker
open http://localhost:8080/docs         # Swagger UI
curl http://localhost:8080/metrics      # API Prometheus metrics
curl http://localhost:9091/metrics      # worker Prometheus metrics
cargo test --all-features --locked      # full test suite; needs Docker

Environment variables are documented in .env.example.

Endpoints

Method	Path	Port	Notes
POST	/jobs	8080	Enqueue. Supports Idempotency-Key header.
GET	/jobs/{id}	8080	Fetch one.
GET	/jobs	8080	Paginated list, filterable by status and kind.
POST	/jobs/{id}/cancel	8080	200 cancelled, 202 running and pending, 409 terminal.
GET	/health	8080	DB-level liveness.
GET	/metrics	8080	API Prometheus metrics.
GET	/metrics	9091	Worker Prometheus metrics on the worker process.
GET	/docs	8080	Swagger UI.
GET	/api-docs/openapi.json	8080	OpenAPI 3 spec.

Full endpoint reference: docs/api.md.

Architecture in one paragraph

The core modules (domain, payload, db, queue, retry, ids, error) are always compiled; api and worker are gated behind features of the same names. The API and worker binaries use required-features, so they cannot be built without their dependency sets. A third binary, job-queue-migrate, runs embedded migrations and exits; Docker Compose uses it before starting the API and worker.

The queue lives in one Postgres table (jobs) with a partial index on (run_at) filtered by status IN ('queued','retrying'). Retries are scheduled by setting run_at into the future; the same dequeue query picks them up when run_at <= now(). Cancellation is atomic for queued/retrying jobs and cooperative for running jobs.

Full design, including state and sequence diagrams: docs/architecture.md.

Feature flags

Feature set	Default	Pulls in
core	always	Domain types, payload validation, queue SQL, retry math, migrations, utoipa schema derives.
api	yes	axum, tower, tower-http, utoipa-swagger-ui, metrics-exporter-prometheus.
worker	yes	tokio-util and metrics-exporter-prometheus.

Consumers who only want the queue types and SQL helpers can use:

cargo add rust-job-queue-api-worker-system --no-default-features

CI checks core-only, API-only, and worker-only builds.

Tests

cargo test --all-features --locked

The suite currently has 46 tests across lib unit tests, one doctest, and seven integration binaries:

• queue_ops (19): SKIP LOCKED dequeue, retry math, cancel paths, recovery sweep, DB-layer idempotency.
• api_jobs (5): create/get/list, 404, 422, /health, OpenAPI rendering.
• idempotency (4): two-concurrent same-key, 10-concurrent same-key race, different keys, header precedence over body.
• retry_and_cancel (2): fail-once-then-succeed, always-fail-to-permanent.
• worker_dequeue (1): enqueue -> worker processes -> succeeded.
• cancel_running (1): cancellation observed at a safe point.
• concurrency_two_workers (1): 200 jobs across 8 workers, no duplicate processing observed by the test harness.
• lib unit tests (12): payload validation, retry jitter bounds, deterministic simulator hash.
• doctest (1): crate-level example in src/lib.rs.

All integration tests use testcontainers; each test gets an isolated ephemeral database.

Benchmarks

Two release-supported benches are reproducible against a fresh Postgres container in Docker.

# Single-worker queue-overhead microbench (Criterion).
cargo bench --bench dequeue --features worker

# Concurrent throughput across {1,2,4,8,16} workers and 3 Postgres configs.
./bench/run.sh

Measured results are in bench/RESULTS.md. The numbers are scoped to the recorded hardware and methodology. They are not presented as production throughput claims.

The exploratory head-to-head comparison benchmark is intentionally excluded from the v0.1 crates.io package until its methodology and teardown behavior are corrected.

Example: embed the worker in your own application

DATABASE_URL=postgres://localhost/jobs \
  cargo run --example embed_worker --features worker

See examples/embed_worker.rs for a small example of plugging a custom Executor into WorkerRuntime.

CI

ci.yml runs:

1. cargo fmt --check, clippy with warnings denied, feature-decoupling checks, full tests, release build, docs with warnings denied, cargo package --list --locked, and cargo publish --dry-run --locked.
2. cargo +1.88.0 check --all-features --locked for the declared MSRV.
3. RustSec audit.

Trade-offs and intentional omissions

This artifact is deliberately small. The decisions and "what would change in production" notes live in docs/tradeoffs.md. Start there if you are evaluating engineering judgment.

Tech stack

Rust (MSRV 1.88), Tokio 1, Axum 0.8, sqlx 0.8, Postgres 16, tower-http 0.6, tracing 0.1, utoipa 5 + utoipa-swagger-ui 9, metrics + metrics-exporter-prometheus 0.18, thiserror 2, anyhow 1, testcontainers 0.27, Criterion 0.5.

License

Apache-2.0