Fleet Deployment Colony

Fleet Deployment — Colony + Ground Station

How a Theia cluster is deployed and operated in the field: colony provisions and orchestrates boards over SSH from an S3 runtime plane, and the Ground Station (GS) drives it from a web UI. This is the production analogue of the local theia manifest → dist → orchestrate loop — see Deployment-Manifest-Generation-and-Serialization for that.

The Tutorial walks the whole flow hands-on in Provision + Deploy; this page is the reference model behind it.

The two planes

Theia splits every deploy into two independent planes:

Plane	Ships	Delivered by	Verb
Base / runtime	supervisor + platform services (.deb)	colony (SSH/Ansible)	provision + orchestrate
App / SWP	the user's Functional Clusters, as an overlay	Mender (OTA)	theia release-swp → GS deploy

The runtime is FIXED per install; the app SWP is exchanged freely on top of it (runtime-dependency gated). That split is the whole point: you re-provision to change the base, but swap the app with a one-click Mender deployment.

Role-keyed cluster

The framework services manifest declares exactly two roles whose name is the role:

• master — the coordinator: etcd + every platform singleton (com, per, sm, phm, vucm, …) + the Mender gateway.
• zonal — a zone-of-responsibility worker: the minimal per-board slice (ucm + shwa). It reaches the master's singletons over TIPC and the shared etcd.

One master, N zonal. The framework names no boards — colony fans the one zonal slice onto every worker board, and each board's runtime identity is master / zonal-1 / zonal-2 / … (the instance disambiguates the shared role name). A day-2 user SWP can rename the deployed workers as it wishes.

Colony — S3-exclusive, two-step

colony is a registry-free deploy service (the colony-api container behind the GS). Every deploy fact comes from the request; the runtime .debs and the per-role manifest come from the S3 runtime plane (s3://theia-runtime/<version>/), never a local tree.

1. provision — one-time board setup: base dirs, OS packages, etcd, the TIPC bearer, the Mender client, and the Theia OTA update modules (theia-swp / theia-app / theia-release) + state-scripts.
2. orchestrate — per-deploy: pull the runtime + services .debs from S3, dpkg -i, lay the release-dir (current → releases/<ver>), push config, reload the supervisor.
3. cleanup — the inverse (stop supervisor, remove /opt/theia, dpkg -r).

Always pass role so colony pulls the right manifest/<role>/ slice.

The Ground Station flow

The GS is the operator's web UI. It reads the fleet from Mender + the S3 planes and fans a Distribution (per-role {abi, runtime_build, swp_build}) out to the matching boards: base → colony, app → Mender.

1 — Deployment Management

Pick a Distribution and the target boards; the GS resolves each board's ABI to the matching build and dispatches.

2 — Releases + Distributions (the S3 planes)

theia release publishes the runtime plane; theia release-swp the app plane. The GS catalogs both.

3 — Fleet view

Every enrolled rig, its base runtime, app SWP, and health.

4 — Enrol a device

A new board SSH-probes into the fleet and dual-registers (colony + Mender).

5 — Deploy + watch the Action History

The GS fans the Distribution out; the Action History shows each per-board Orchestrate / Provision / SWP action with a cancel control and a live status.

The HTTP API

colony-api listens on :8081 inside the GS docker network. The GS (and the theia MCP colony_deploy tool) drive it:

POST /deployments               {rig, kind, host, extra} → {id, status, …}
GET  /deployments/{id}          status + statistics
GET  /deployments/{id}/log      the Ansible output
POST /deployments/{id}/abort    abort a pending/scheduled run

kind ∈ {provision, orchestrate, cleanup}; host = "ip[:port]"; role, runtime_version, machine_instance, etcd_external ride in extra. Poll the id until status is finished; success = statistics.status.success > 0 && failure == 0.

// provision central as the master role
{"rig":"central","kind":"provision","host":"10.0.0.23:2201",
 "extra":{"role":"master","machine_instance":"0",
          "runtime_version":"0.3.0-amd64","etcd_external":"true"}}

Footguns

• ansible_connection inheritance. colony resolves the target in a hosts: localhost play then add_hosts the board. The add_host reads a DEDICATED target_connection var (default ssh) — NOT ansible_connection, which is already local in the localhost play and would make every board task run ON THE CONTROLLER. The docker-test-rig path overrides with -e target_connection=community.docker.docker.
• Machine instance is a DEPLOY fact. The N zonal workers share the name zonal; their distinct TIPC instances (1, 2, …) come from the machine_instance colony passes, persisted on-device so an OTA relaunch recovers the same instance instead of colliding at 0.
• Safe-base HW gating. The serialized manifest carries deploy/config/master/executor.json marking hardware-dependent FCs (fw/tsync/rds) run_on_start: false, so the runtime deploys cleanly on ANY rig regardless of HW/CAPA. An operator whose target has the subsystem re-enables it with a per-target override.

Local docker fleet (the dev mirror)

deploy/docker-compose.yml runs theia-{central,compute,frontal} + a theia-etcd sidecar (theia rig up) as network_mode: host SSH rigs (:2201 / :2202 / :2203), which colony provisions over SSH — the local mirror of a real fleet. Drive the whole flow against it, exactly as the Tutorial does.