Tutorial 9 Advanced

← 8. Observe in the supervisor GUI · Tutorial index

Chapter 9 — Advanced deployment: arity, versioning, and migrations

The earlier chapters deployed a single app onto a small rig. This chapter is the operator/release-engineer reference for running Theia at scale over time:

9.1–9.4 Higher-arity deployment — one master + N workers, the machine-vs-role identity split, and how a single manifest fans out to a heterogeneous fleet.
9.5–9.8 Semantic versioning — what PATCH / MINOR / MAJOR mean for an app SWP, and the one-command way to cut each.
9.9–9.11 Config migration — the rules for changing an app's .art interface or its config without bricking a running board.
9.14 Packages — the reusable-unit layer: write a node + protocol + impl once, publish it as its own repo, and import/link it into many workspaces.

These three are one story: as arity grows and versions churn, the framework needs a stable identity per board and a disciplined way to evolve the interface. Read it once end-to-end; come back to a section when you cut a real release.

9.1 Arity: one master, N workers

A deployment's arity is how many machines it spans. Chapter 4 showed arity-1 (everything on one board) and a 2-machine split. The rule generalizes:

One master, N zonal workers. The master is the coordinator — it hosts the single per-cluster etcd, the deployment-wide singletons (per, sm, phm, com, vucm), and the mender gateway. Every other board is a zonal worker: the minimal per-board FC set (ucm + shwa) plus a mender agent that reaches the server through the master's proxy. One master + as many workers as your rig has.

The framework runtime ships this shape as a role-based rig you consume (you don't author it): manifest/services/rig.py exports RIG (master only) and MULTI (master + workers). A Distribution's roles pick which materialize onto which boards.

# arity-1: the single-master runtime (the common case)
theia manifest services

# arity-N: master + named workers
theia manifest services --attr MULTI

9.2 The identity split: role vs machine name

This is the single most important idea in a multi-board deploy. There are three identities, and conflating them is the classic mistake:

Identity	What it is	Unique?	Used for
role	`master` / `zonal` — the deployment identity	No (N workers share `zonal`)	which manifest slice colony provisions; whether the board runs etcd / a mender gateway vs agent
machine name	`master` / `compute` / `frontal` — the runtime identity	Yes	how `com`, the GUI, and `rtdb`/`tdb` address one specific board
hostname	the OS hostname	not guaranteed	never used as an identity — informational only

The rule: role is the deployment identity; the machine NAME is the runtime identity. Two workers can share the role zonal, but they must have distinct names (compute, frontal). Never use a role as a machine name — a rig with two machines both named zonal is rejected at serialize time (§9.4).

Why it matters: com on the master aggregates every board's supervisor tree into one view. To keep compute and frontal as distinct subtrees — and to let an operator type rtdb ps compute — each needs a unique name. The role (zonal) can't disambiguate them; the name can.

9.3 How a machine name resolves — `machine_index`

Each machine gets a stable cluster index at serialize time: the master is 0, workers are 1, 2, … in canonical order. The supervisor's own TIPC control node binds at type=0x80020001, instance=<machine_index> — so com discovers a supervisor at instance N and maps N → machine_index → the unique name.

The name→index map is written into machines.json (the one manifest com references):

// dist/manifest/machines.json  (arity-3 MULTI)
{
  "machines": ["master", "compute", "frontal"],
  "machine_index": { "master": 0, "compute": 1, "frontal": 2 },
  "role_map":      { "master": "master", "compute": "zonal", "frontal": "zonal" }
}

machine_index is authoritative — it wins over whatever a board's supervisor reports about itself, so a machine name is correct even mid-upgrade. On a deployed board, machines.json is staged into /opt/theia/config/, and the launcher points com at it (THEIA_MACHINE_MANIFEST=/opt/theia/config). Provisioning does both for you; you never wire it by hand.

Verify the identity end-to-end — from the master, list the cluster:

rtdb machines
# INST  MACHINE   PRESENT  HOST
# 0     master    yes      board-a
# 1     compute   yes      board-b
# 2     frontal   yes      board-c

rtdb ps compute     # scope to ONE board by its unique name
rtdb ps 2           # …or by its instance number

9.4 The identity invariants (enforced at serialize time)

theia manifest runs the TIPC address-collision gate, then three machine-identity invariants. A violation fails the serialize with a clear message — so a broken manifest never reaches a board:

Machine names are unique. (The name is the runtime identity.)
machine_index is sequential from 0 — 0, 1, 2, …, no gaps.
Index 0 is the master (the etcd coordinator).

theia manifest services --attr MULTI
# serialize-manifest: machine NAMES must be unique — … Duplicated: ['zonal'].
#   Give each board a distinct name (role master/zonal is the deployment
#   identity, NOT the name).

If you author your own multi-machine rig, give each worker a distinct name= and its role=Explicit("zonal"); the index + validation are automatic.

9.5 Semantic versioning of an app SWP

The runtime/base is installed once and held fixed; your app SWP is exchanged freely on top of it. Semver tells you how freely:

Bump	Example	Meaning	A free swap?
PATCH	`1.0.0 → 1.0.1`	bug fix, no interface change	Yes — deploy at will
MINOR	`1.0.0 → 1.1.0`	compatible, additive change	Yes
MAJOR	`1.0.0 → 2.0.0`	`.art` interface change (a proto / port / message contract)	No — needs a migration + a runtime pin

The dividing line is the .art interface. If the change is invisible at the .art level (you fixed a handler bug, tuned a constant), it's a PATCH/MINOR — a free overlay swap on the same runtime. If you changed a message, a port, or a node address in .art, it's a MAJOR: the wire contract moved, so it needs an explicit migration and it pins the runtime it was built against.

9.6 Cutting a PATCH — the free swap

theia release-swp <app> --patch mints the next patch of an app: it reads the app's latest published SWP, increments the PATCH, and builds + publishes the new artifact to the package plane. That new artifact is exactly what a Ground Station Rollout advances a device group onto.

# 1.0.0 already published → this cuts 1.0.1 and pushes it, abi-keyed
theia release-swp counter --patch --s3 http://10.0.0.99:9000
#   theia release-swp: counter 1.0.0 → v1.0.1 (patch bump)
#   published → s3://theia-swp/user-software/theia-rig/counter/1.0.1-amd64/

--from V — bump from an explicit base instead of "latest on S3".
--to V — set an explicit target version (skips the auto-bump).
The abi (amd64, bookworm-arm64, …) is baked into the artifact name, so a per-role Distribution resolves the right binary for each board — "partial per machine".

A PATCH ships no migration. On-device it's a clean overlay: the app's FC binaries are swapped, the supervisor re-reads its tree, done.

9.7 Cutting a MAJOR — `--migrate`

A MAJOR crosses the .art interface, so it is refused unless you acknowledge it with --migrate:

theia release-swp counter --to 2.0.0 --s3 http://10.0.0.99:9000
#   ✗ 2.0.0 crosses a MAJOR boundary … Re-run with --migrate to confirm.

--migrate makes two things mandatory, because a major is not a free swap:

A runtime pin — --requires-runtime <V>. A major app depends on exactly one runtime (its ABI/proto are pinned at build time). The Ground Station deploy gate refuses to install a pinned app onto a board whose base_version differs ("update the base first"). Refused if empty.
A migration file — --migration <path>, else the conventional apps/<app>/migrations/v<from>-to-v<to>.py. If none exists an empty no-op stub is auto-created for you to edit — a migration always ships, even if it's a no-op, so the upgrade is explicit and reviewable.

theia release-swp counter --to 2.0.0 --migrate \
  --requires-runtime 0.2.2-amd64 --s3 http://10.0.0.99:9000
#   theia release-swp --migrate: created empty migration stub
#     apps/counter/migrations/v1-to-v2.py — EDIT it, then re-run.

Edit the stub, re-run, and the migration is packed into the SWP (and synced to s3://theia-swp/.../migration/) as a first-class part of the artifact.

9.8 One artifact, one gate

The three release facts to keep straight:

PATCH / MINOR  →  free overlay swap, same runtime, no migration
MAJOR (--migrate)  →  runtime pin (requires_runtime) + a shippable migration/

The runtime pin is enforced twice — at a direct app publish and at a Rollout — so a major SWP can never land on the wrong runtime. That's what makes "exchange the app freely on a fixed runtime" safe: the framework knows which swaps are free and which aren't.

9.9 Config migration: the on-device rule

A MAJOR often needs to move on-device config or state across the interface break — rename a key, split a message, seed a new default. That is what the migration file is for. It runs on the board during install, before the new binaries take over:

ArtifactInstall on the board:
  1. unpack the new SWP
  2. run  migration/v1-to-v2.py <THEIA_ROOT>   ← against the STILL-OLD release
  3. overlay the new FC binaries
  4. merge the executor subtree + reload the supervisor

Running the migration first, against the old release, is deliberate: it can read the pre-upgrade config and rewrite it for the new interface. If the migration exits non-zero the install aborts and Mender rolls back — a broken migration never leaves a half-migrated board.

9.10 Writing a migration

The stub is a plain Python script — migrate(theia_root) invoked with the install root:

"""Migration counter v1 → v2 (MAJOR / .art interface change)."""
import json, sys, pathlib


def migrate(theia_root: str) -> None:
    cfg = pathlib.Path(theia_root) / "config" / "counter.json"
    if not cfg.is_file():
        return                                  # nothing installed yet — no-op
    data = json.loads(cfg.read_text())
    # v2 renamed `limit` → `max_count`; carry the operator's value across.
    if "limit" in data:
        data["max_count"] = data.pop("limit")
        cfg.write_text(json.dumps(data, indent=2))


if __name__ == "__main__":
    migrate(sys.argv[1] if len(sys.argv) > 1 else "/opt/theia")

Rules for a good migration:

Idempotent. It may run more than once (a retried install); running it twice must be a no-op the second time — check before you transform.
Tolerant of "not there yet." A fresh board may not have the old config — return cleanly, don't raise.
Fail loud on a real problem. A genuinely un-migratable state should exit non-zero — that aborts + rolls back, which is safer than a corrupt config.
Config, not code. It moves data; it never patches binaries (the SWP overlay does that).

9.11 What NOT to migrate — the plane boundary

The migration is app-plane only. It never touches the runtime/base — that plane is re-provisioned, never rolled, and carries no migration. Two consequences:

If your change needs a new runtime (a platform ABI bump), that's a base re-provision through colony — do it first, then deploy the app major that pins it (the runtime pin from §9.7 is what sequences this correctly).
A PATCH/MINOR has no migration by construction. If you find yourself wanting to migrate config for a "minor" change, it wasn't minor — the interface moved, so cut it as a MAJOR.

9.12 Node multiplicity — N clones of one node

Arity (§9.1) was about machines. Multiplicity is the same idea one level down: N instances of the same node type, in one process. Same code, same TIPC type, distinct instance — a worker pool.

You express it in .art with prototype (attribute-replacement — each clone is the base node, re-addressed to its own instance):

node atomic CounterNode { tipc type=0xd0010001 instance=0  config CounterConfig … }

// nine more clones at the same type, instances 1..9
node atomic Counter1 prototype CounterNode { tipc type=0xd0010001 instance=1 }
node atomic Counter2 prototype CounterNode { tipc type=0xd0010001 instance=2 }
// …

composition CounterProc {
    prototype CounterNode counter0   on process P1   // instance 0 — the base
    prototype Counter1    counter1   on process P1
    // …
}

Two things multiplicity gives you, and they use the instance differently:

1. A node knows its own instance. main resolves each clone's instance and hands it to the node (GenServerBase::tipc_instance()), so handler code can act on which clone it is:

void CounterNode::init(CounterNodeState&) {
    const uint32_t inst = this->tipc_instance();      // 0..9 — this clone
    // key THIS clone's config as "counter/<instance>"
}

2. Per-instance config. Each clone owns its config in per under <component>/<instance> (counter/0, counter/1, …). A change to counter/3 notifies only clone 3 — per reads the instance straight from the key and casts ConfigUpdated to that exact instance. No fan-out, no clone reading another's config. (Omit the instance and it's a single-instance node, the common case.)

Addressing a clone. TIPC addresses a node by (type, instance). Send to a specific instance to hit one clone; several ports co-bound at the same (type, instance) are round-robined by the kernel — that's how you build a load-balanced worker pool (bind the pool at one address) vs an addressable set (bind each at its own instance). Multiplicity gives you both shapes.

Poking a node — `theia cast` / `theia call`

For test/demo, drive a running node straight from the shell — pack a JSON payload into the node's proto message and send it over TIPC:

theia cast CounterNode Inc  --data '{"n": 5}'                 # send to the base
theia cast CounterNode Inc  --data '{"n": 100}' --instance 3  # ONE clone (inst 3)
theia call CounterNode Get  --instance 5                      # call → prints reply

--instance N targets clone N; --machine M shifts the instance by a board's machine index (the per-board clone on a /N deploy).
cast is fire-and-forget; call blocks for the reply and prints it as JSON.
No --instance sends to the node's base address (round-robined if the pool is co-bound).

It resolves <node> + <msg>/<op> from your workspace's .art, so any node you model is reachable — a one-liner instead of a probe script or an rf test.

The in-repo tutorial_ws carries the 10-CounterNode composition above as the worked example: build it, then theia cast CounterNode Inc --instance 3 and watch [counter3] Inc (inst 3) → count=… in its log.

9.13 Running the ARA services locally — `theia init --with-services`

Every earlier chapter used a bare workspace (theia init): the supervisor + your own apps, no ARA platform services. That is the normal app-developer setup — your app reaches the platform (config via per, state via sm, …) over TIPC at runtime, and those services run where the platform runs (a provisioned board, or a local docker-compose rig), NOT inside your app workspace.

theia init --with-services is the advanced, single-slice case: it symlinks the framework's service FCs into your workspace so theia start brings up the whole platform (com/log/per/sm/ucm/shwa + your apps) as one supervised tree on one TIPC namespace. Use it only when you deliberately want the app and the platform co-located on a single TIPC slice — e.g. an all-in-one dev box or a CI smoke test — rather than the normal split where the platform is a separate deploy.

theia init --with-services          # symlinks system/services → $THEIA_ROOT
theia manifest bootstrap
theia install bootstrap             # stages supervisor + all service FCs
theia start                         # the full platform + your apps, one tree

Two things to know:

One TIPC namespace = one platform. host-network docker containers share a single TIPC nametable, so a --with-services workspace and a docker-compose platform rig on the same host collide at the service addresses. Run one or the other on a given slice — or isolate them with a per-rig TIPC cluster id (Machine.tipc_cluster_id). The common pattern is: the compose rig owns the platform slice, and your app workspace does a bare theia init and joins that slice, rather than starting a second platform.
per needs etcd; nm needs CAP_NET_ADMIN. A local --with-services start without etcd (or netadmin) will see per retry / nm fail. Either point per at a running etcd, or keep those services defined but down so the rest of the tree still comes up clean. theia install deep-merges a per-machine override onto the staged supervisor tree, so drop a deploy/config/<machine>/executor.json in your workspace:
```
{"children":[{"name":"services_sup","children":[
   {"name":"per","run_on_start":false},
   {"name":"nm", "run_on_start":false}]}]}
```
Re-run theia install and those FCs stay defined but not booted (the supervisor logs run_on_start=false — defined, not started at boot), while com/sm/the rest start normally.

9.14 Packages — reusable units you import and link

Every earlier chapter put your nodes in one workspace's system/apps. A package is the missing layer between the runtime and a workspace. The ROS analogue is runtime / libraries / workspace; Theia's is exactly that: runtime → PACKAGES → workspace. A package is a self-contained, independently-repo'd unit — nodes + protocol + impl, built once as a linkable lib, with no executable of its own — that any number of workspaces import and link. Write a SLAM estimator or a radio transport once; ship it as a repo; compose it into many apps.

App vs package. An app is a composition you deploy (theia init, Chapters 2–3). A package is a library you publish and others consume (theia init --kind package). A package owns nodes + a lib; it does not own a composition/cluster — the consuming workspace does.

Scaffolding a package — `theia init --kind package`

mkdir v2v && cd v2v
theia init --kind package --name v2v      # a PACKAGE repo (not a workspace)

This scaffolds two .art packages in two dirs, modelling the import/link relationship in one repo exactly as an external consumer would:

system/v2v/package.art          package system.v2v          — the node(s) + protocol + algo
system/v2v_tester/component.art  package system.v2v_tester   — imports system.v2v, LINKS its lib
system/system.art                package system              — aggregate parse entry
src/{lib,impl}                   ← gen-app --kind package  (lib generated; impl WRITE-ONCE)
proto/system/v2v/                ← the package's own self-contained proto

The whole toolchain then runs unmodified, split across the package and its tester (the loop from Chapters 2–3):

# 1. the PACKAGE (node + lib) — system/v2v source → src/{lib,impl}, //src/lib:v2v_lib:
artheia gen-app --kind package system/v2v/package.art --out src --proto-out proto --ns ara::v2v
# 2. the TESTER app — imports system.v2v, links //src/lib:v2v_lib (apps/ is gitignored):
artheia gen-app --kind fc      system/v2v_tester/component.art --out apps --proto-out proto
artheia gen-manifest system/v2v_tester/component.art manifest/apps/manifest.py
theia manifest rig && theia install rig && theia start
robot test/v2v.robot                 # a probe drives the node's ctl over TIPC

The conventions each fix a concrete resolver/label bug — don't "simplify" them away:

Source (system/) vs codegen (src/) are separate. Hand-edited .art lives under system/; gen-app --kind package --out src writes src/lib/ (regenerated every run) and src/impl/ (write-once — your handler bodies + state, yours to own after first emit).
The package and its tester are separate dirs. The loader merges a package.art + component.art pair in one dir as ONE package, so system.v2v and system.v2v_tester cannot share a dir.
Namespace ara::<name> (--ns) — co-composed packages otherwise collide on Log.hh.
No framework symlinks. A package .art never imports system.supervisor / system.platform.
The in-repo tester IS the consuming pattern — it imports system.v2v and links the prebuilt //src/lib:v2v_lib, proving the cross-package link in-repo.

Consuming a package — a bazel-module submodule

A package is a separate bazel module — a consumer references it module-qualified (@v2v//src/lib:v2v_lib), exactly like the framework's own @pero_theia, so the package's internal //src/... labels resolve within its module. In your workspace:

# 1. add the package repo as a git submodule under packages/<name>
git submodule add https://github.com/perotheia-packages/v2v.git packages/v2v

# 2. wire it as a bazel module in MODULE.bazel:
#      bazel_dep(name = "v2v", version = "0.1.0")
#      local_path_override(module_name = "v2v", path = "packages/v2v")

# 3. map the FQN to the submodule so artheia resolves the import:
ln -s ../packages/v2v/system/v2v system/v2v            # → import system.v2v

# 4. import + prototype it in your app composition (system/apps/component.art):
#      import system.v2v.*
#      composition MyApp   { prototype OsiV2v v2v }
#      cluster Applications { composition MyApp app }

# 5. generate the package's own lib inside the submodule (gitignored codegen),
#    then build — your app LINKS the prebuilt lib, never regenerates the node:
( cd packages/v2v && artheia gen-app --kind package system/v2v/package.art \
    --out src --proto-out proto --ns ara::v2v )
artheia gen-app --kind fc system/apps/component.art --out apps --proto-out proto
bazel build //apps/MyApp/main:apps       # links @v2v//src/{lib,impl}

The generated app depends on @v2v//src/lib:v2v_lib + @v2v//src/impl:v2v_impl + @v2v//proto/system/v2v:v2v_proto (module-qualified, from the package's MODULE.bazel name); the imported node is linked, never regenerated.

Composing several packages — the integration workspace

When packages depend on each other (a mesh transport provides the beacon stream v2v requires), the clean pattern is a dedicated integration workspace repo — the ROS colcon analogue — that submodules all the packages under packages/, imports them, and wires their ports in one composition:

import system.v2v.*
import system.meshtastic.*
composition MeshV2vApp {
    prototype Meshtastic mesh
    prototype OsiV2v     v2v
    connect mesh.beacons_out to v2v.beacons_in    # transport → estimator
}

bazel build links @v2v// + @meshtastic// — two prebuilt packages assembled into one deployable app, real beacons flowing transport → estimator over the connected ports. This is where the packages are built and tested together (a natural CI target), while each package repo stays lean.

The published packages live in the separate perotheia-packages GitHub org — clone them as worked examples: v2v (relative-topology SLAM + cooperative-alert consensus) and meshtastic (a swappable mesh transport that feeds v2v), assembled in connectivity-ws — the connectivity integration workspace, which submodules the related connectivity packages and composes them into deployable, port-connected apps (v2v + meshtastic today; add more as the connectivity stack grows).

You now have: the higher-arity model (one master + N uniquely-named workers, the role-vs-name split, and the serialize-time invariants that keep identity clean); the semver rules for an app SWP (PATCH/MINOR are free swaps, MAJOR pins a runtime and ships a migration); the on-device config-migration contract (runs first, against the old release, idempotent, aborts on failure, app-plane only); node multiplicity (N clones of one node, each knowing its instance + owning its per-instance config, poked from the shell with theia cast/call); and packages — the runtime→packages→workspace layer: scaffold with theia init --kind package, consume as an @<name>// bazel-module submodule, and compose several into one app in an integration workspace.

Next: you have finished the tutorial. Revisit Chapter 5 to bind a specific runtime + SWP version into a Distribution, and Chapter 6 to deploy it across a multi-board fleet.

← 8. Observe in the supervisor GUI · Tutorial index

Theia Wiki

Home

Tutorial

Tutorial index
1. Install
2. Workspace
3. First app
4. Rig
5. Distribution
6. Provision + Deploy
7. RF test
8. Operator GUI
9. Advanced

Reference

Uh oh!

Tutorial 9 Advanced

Chapter 9 — Advanced deployment: arity, versioning, and migrations

9.1 Arity: one master, N workers

9.2 The identity split: role vs machine name

9.3 How a machine name resolves — machine_index

9.4 The identity invariants (enforced at serialize time)

9.5 Semantic versioning of an app SWP

9.6 Cutting a PATCH — the free swap

9.7 Cutting a MAJOR — --migrate

9.8 One artifact, one gate

9.9 Config migration: the on-device rule

9.10 Writing a migration

9.11 What NOT to migrate — the plane boundary

9.12 Node multiplicity — N clones of one node

Poking a node — theia cast / theia call

9.13 Running the ARA services locally — theia init --with-services

9.14 Packages — reusable units you import and link

Scaffolding a package — theia init --kind package

Consuming a package — a bazel-module submodule

Composing several packages — the integration workspace

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Theia Wiki

Related wikis

Clone this wiki locally

9.3 How a machine name resolves — `machine_index`

9.7 Cutting a MAJOR — `--migrate`

Poking a node — `theia cast` / `theia call`

9.13 Running the ARA services locally — `theia init --with-services`

Scaffolding a package — `theia init --kind package`