Checkpoint-aware, fault-tolerant job orchestration for AI training workloads.
Veriflow is a Kubernetes-based AI workload orchestrator that implements a control-plane + scheduler architecture for running training-style jobs with runtime awareness, failure recovery, and checkpoint-aware retry.
It treats AI workloads as distributed systems problems — where scheduling, correctness, and failure handling are first-class concerns — rather than simple API execution.
Training a large model overnight and waking up to find it crashed at hour 6 — with no checkpoint, no retry, and no idea why — is a rite of passage in ML engineering.
Most job runners treat AI workloads like simple scripts: run it, and if it fails, restart from zero. But training jobs are long-running, stateful, and expensive. They need:
- Checkpoint-aware recovery — resume from where it failed, not from scratch
- Concurrency-safe scheduling — multiple schedulers, zero duplicate runs
- Runtime observability — know why a job failed, not just that it failed
- GPU-aware placement — match jobs to the right hardware, every time
Veriflow treats AI training as what it actually is: a distributed systems problem.
Modern AI systems are not just model pipelines — they are distributed systems.
Veriflow models:
- job lifecycle as a state machine
- execution as Kubernetes workloads
- runtime signals as control-plane inputs
- recovery as checkpoint-aware retry
- Idempotent job submission (
Idempotency-Key) - Concurrency-safe job claiming (
FOR UPDATE SKIP LOCKED) - Queue + priority-based scheduling
- GPU-aware placement decisions
- Retry with backoff (
next_run_at) - Timeout handling for long-running jobs
- Event-sourced lifecycle tracking (jobs, runs, events)
- Kubernetes-based execution (
batch/v1.Job) - Runtime-aware reconciliation (training progress, checkpoints)
- Checkpoint-aware retry and resume
Client
│ POST /v1/jobs (Idempotency-Key)
▼
job-api (Go)
│ writes jobs/spec to Postgres
▼
Postgres (jobs, runs, events)
▲
│ claim (FOR UPDATE SKIP LOCKED)
│ create run attempt
│ dispatch → Kubernetes Job
│ reconcile runtime + K8s state
▼
scheduler (Go) ───────────► Kubernetes Job / Pod
This mirrors real-world AI infra where:
- control plane = state + scheduling
- data plane = execution (K8s jobs)
A training job now produces:
JOB_SUBMITTED
JOB_SCHEDULED
RUN_CREATED
PLACEMENT_SELECTED
DISPATCH_REQUESTED
POD_RUNNING
TRAINING_PROGRESS
CHECKPOINT_SAVED
RUN_FAILED
RETRY_TRIGGERED
RUN_CREATED (attempt 2)
TRAINING_RESUMED
...
JOB_SUCCEEDED
Veriflow was tested with two concurrent scheduler instances (sched-a, sched-b) operating on the same Postgres-backed queue.
- Dispatch distribution:
- sched-a: 3 dispatches
- sched-b: 3 dispatches
- No duplicate dispatch of the same run observed
This demonstrates that FOR UPDATE SKIP LOCKED enables safe concurrent job claiming without duplicate execution, while allowing multiple schedulers to share work.
- Submitted 20 concurrent jobs
- Scheduler continued:
- claiming jobs
- dispatching workloads
- emitting runtime events
- No crashes or deadlocks observed
Veriflow maintains stable control-plane behavior under burst submission.
- Jobs emit:
TRAINING_PROGRESSCHECKPOINT_SAVED
- Failures detected via container exit codes
- Events show consistent failure propagation:
RUN_FAILED → (retry OR terminal failure)
For retry-enabled jobs:
-
checkpoint persisted:
latest_checkpoint_uri = /artifacts/ckpt-2 -
retry scheduled:
RETRY_TRIGGERED -
resumed execution:
TRAINING_RESUMED -
final success:
JOB_SUCCEEDED
Veriflow resumes failed training runs from the latest checkpoint instead of restarting from scratch.
- Multiple jobs failed concurrently
- Scheduler:
- continued processing
- emitted consistent events
- avoided duplicate claims
System remains consistent under failure bursts.
- verified best-fit placement selects tighter-fit nodes when multiple nodes satisfy a request
- verified exact GPU type filtering (
requested_gpu_type = A100) - verified minimum per-GPU memory filtering (
min_gpu_memory_mb) - verified explicit deferred placement reasons for unsatisfied constraints
- 40+ RUN_CREATED
- 30+ CHECKPOINT_SAVED
- 20+ RUN_FAILED
- successful retry + resume observed
- basic job submission
- Postgres persistence
- job claiming
- run lifecycle
- dispatch as K8s Jobs
- reconcile job state
- next_run_at
- multi-attempt runs
- event-sourced lifecycle
- logs + status tracking
- runtime status ingestion
- training progress tracking
- checkpoint persistence
- checkpoint-aware retry + resume
Veriflow implements constraint-based GPU scheduling with device-level resource accounting and explainable placement decisions.
Jobs can specify GPU requirements via the API:
gpuCount— number of GPUs requiredgpuType(optional) — exact GPU type (e.g., A100, L4)minGpuMemoryMb(optional) — minimum per-GPU memory requirement
Example:
{
"gpuCount": 2,
"gpuType": "A100",
"minGpuMemoryMb": 30000
}- added rotating queue order for fairer multi-queue scheduling
- enforced queue-level GPU quotas
- deferred over-quota jobs with explicit
queue_gpu_quota_exceededreason - preserved device-level best-fit GPU placement and explicit constraint failure reasons
This runs a full end-to-end workflow locally:
./scripts/demo.shThis script will:
- submit a job
- Kubernetes execution
- log output
- DB lifecycle + event timeline
make up
make api
make sched
make demo-success
make eventscurl -X POST http://localhost:8080/v1/jobs \
-H "Content-Type: application/json" \
-H "Idempotency-Key: test-1" \
-d '{
"image": "busybox",
"jobType": "training",
"gpuCount": 1,
"datasetUri": "s3://data/sample"
}'See RUNBOOK.md for full operational steps.
- Control-plane design for distributed systems
- Concurrency-safe scheduling using DB primitives
- Kubernetes-based workload orchestration
- Runtime-aware system reconciliation
- Failure detection and retry semantics
- Checkpoint-aware recovery
- AI infrastructure patterns (training workloads)
AI systems should be treated as fault-tolerant distributed systems, not just model execution pipelines.
MIT