'onepool train job.yaml --pool' hosts a session; every 'onepool join'
worker participates automatically:
- Coordinator (job submitter) owns canonical LoRA weights + Nesterov
outer optimizer; trains its own shard while workers train theirs
- per-round flow: H local AdamW steps -> workers ship float32
pseudo-gradients (no pickle on the wire) -> sample-weighted average
-> outer step -> fresh weights broadcast
- data sharded by member; reshards when membership changes
- fault handling: late worker skipped for the round and resynced;
disconnected worker dropped; mid-job joiner enrolled at the next
round boundary with current weights
- workers without the training stack decline gracefully (TRAIN_ERR)
- dashboard: live job panel with SVG loss curve
- --nodes N gate to wait for workers before starting
Verified: 2 processes, distilgpt2, loss 6.05 -> 4.80 across 2 rounds
with sync over TLS; both sides exit clean.