From 2460337151c85f0fead7c2195c876fbcab3e478a Mon Sep 17 00:00:00 2001 From: sergeyb Date: Wed, 29 Apr 2026 19:23:37 +0000 Subject: [PATCH] docs(rfc): Add orchestrator workflow diagram and controller summary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the queue-driven controller pipeline from gateway entry through batching, scoring, build, merge, and conclude — with an ASCII diagram, a per-controller summary table, and a short overview of the two cycles in the pipeline (CI feedback and merge → speculate). --- doc/rfc/index.md | 1 + doc/rfc/workflow.md | 82 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) create mode 100644 doc/rfc/workflow.md diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 3f1597d2..a2160392 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -5,3 +5,4 @@ Design documents and technical proposals for SubmitQueue. ## Index - [SQL-Based Distributed Queue](sql-queue-rfc.md) - MySQL-based distributed message queue with partition leasing and at-least-once delivery +- [Orchestrator Workflow](workflow.md) - Queue-driven controller pipeline from gateway entry through batching, scoring, build, merge, and conclude diff --git a/doc/rfc/workflow.md b/doc/rfc/workflow.md new file mode 100644 index 00000000..356a7ae4 --- /dev/null +++ b/doc/rfc/workflow.md @@ -0,0 +1,82 @@ +# Orchestrator Workflow + +The orchestrator processes land requests through a queue-driven pipeline of small, single-purpose controllers. The gateway accepts a request over RPC and hands it off asynchronously; from there each controller consumes one topic, advances the request or batch, and publishes to the next topic. Most hops carry only an ID — the controller fetches the entity from storage — while a few entry points (`start`, `buildsignal`, `log`) carry the full payload because there is no row to fetch yet. + +The pipeline has two cycles: `speculate → build → buildsignal → speculate` (CI feedback loop) and `merge → speculate` (advance the next batch). `conclude` is the only stage that transitions a request to a terminal state; `log` is an append-only sink that any controller can publish to via `core/request.PublishLog`. + +## Diagram + +``` + ┌──────────────────────────────────┐ + │ gateway:Land (RPC entry) │ + │ Accept, mint ID, hand off async │ + └────────────────┬─────────────────┘ + │ LandRequest + ▼ + ┌──────────────────────┐ ┌──────────────────────────────────┐ + │ log (terminal sink) │◄───│ start │ + │ Append RequestLog │ │ Persist Request, emit Started │ + └──────────────────────┘ └────────────────┬─────────────────┘ + ▲ │ RequestID + │ ▼ + │ ┌──────────────────────────────────┐ + │ │ validate │ + │ │ Check mergeability + change meta │ + │ └────────────────┬─────────────────┘ + │ │ RequestID + │ ▼ + │ ┌──────────────────────────────────┐ + │ │ batch │ + │ │ Group request into a Batch │ + │ └────────────────┬─────────────────┘ + │ │ BatchID + │ ▼ + │ ┌──────────────────────────────────┐ + ├─────────────────│ score │ + │ RequestLog×N │ Score the batch, persist score │ + │ └────────────────┬─────────────────┘ + │ │ BatchID + │ ▼ + │ ┌──────────────────────────────────┐ + │ ┌───►│ speculate (stub) │◄────┐ + │ │ │ Decide CI verify vs. land │ │ + │ │ └──────┬─────────────────┬─────────┘ │ + │ │ BatchID │ │ BatchID │ + │ │ ▼ ▼ │ + │ │ ┌──────────────────┐ ┌──────────────────┐ │ + │ │ │ build │ │ merge │ │ + │ │ │ Trigger CI build │ │ Merge + advance │─┤ + │ │ └────────┬─────────┘ └────────┬─────────┘ │ + │ │ Build │ │ BatchID │ + │ │ ▼ │ │ + │ │ ┌──────────────────┐ │ │ + │ └──│ buildsignal │ │ │ + │ BatchID │ Feed CI result │ │ │ + │ │ back to spec. │ │ │ + │ └──────────────────┘ │ │ + │ ▲ │ BatchID │ + │ │ Build (ext. CI) ▼ │ + │ │ ┌──────────────────┐ │ + │ │ │ conclude │ │ + │ │ │ Map batch state │ │ + │ │ │ → request state │ │ + │ │ └──────────────────┘ │ + │ │ │ + └─── any controller via core/request.PublishLog ──────┘ +``` + +## Per-controller summary + +| Controller | In | Out | One-line role | +|---|---|---|---| +| **gateway/Land** | RPC | start | Accept request, mint ID, log Accepted, hand off async | +| **start** | LandRequest | validate, log | Persist Request and emit Started log | +| **validate** | RequestID | batch | Check mergeability and fetch change metadata | +| **batch** | RequestID | score | Group request into a Batch with dependencies | +| **score** | BatchID | speculate, log | Score the batch (∏ per-request scores), persist score | +| **speculate** | BatchID | build, merge | (stub) Decide whether to verify via CI or land | +| **build** | BatchID | buildsignal | Trigger CI build for the batch | +| **buildsignal** | Build | speculate | Feed CI result back into speculation | +| **merge** | BatchID | conclude, speculate | Merge the batch and advance the queue | +| **conclude** | BatchID | — | Map terminal batch state to request state | +| **log** | RequestLog | — | Append-only sink for request log events |