Probe and Isolation Testing

Probe & Isolation Testing

rf-theia can test a Functional Cluster in isolation — without the rest of the stack — by speaking to it directly over its TIPC wire, impersonating the peer that would normally talk to it. Two ideas make this work: the probe and statem trace observation.

The probe — a .art-driven test client

A probe is a Python test node built from a .art spec. From the parsed .art it binds a tester identity (a sender/client node) to a real TIPC address and speaks the gen_server wire (cast / call) directly to a live FC — no gRPC, no com, no supervisor in the path. It is the artheia.gen_server.probe package; rf-theia wraps it in rf_theia.runtime.ProbeAdapter.

Because the probe is a real TIPC peer bound from a .art tester node, the FC under test cannot tell it apart from the production peer. That is what makes it a mock driver: stop the real driver, let the probe send the same messages.

Transport is swappable. Probes target a node's TIPC address resolved from the .art — change the runtime + probe and everything above keeps working. Always drive a node from a probe built off its .art, never from raw TIPC.

Mocking by stopping an FC and letting the probe touch the supervisor

The pattern for isolating one FC:

1. Stop the real peer — terminate the FC (or app) that normally drives the node under test, via the supervisor (Crash Child / Terminate Child, i.e. the supervisor's TerminateChild control op). The node under test stays up; its driver is gone.
2. Probe impersonates the peer — the probe, bound from a tester .art node, casts/calls the same messages the real peer would. It can also drive the supervisor's control surface directly (start/terminate children, push trace/log config) to set up the scenario.
3. Observe the result — over traces (below) or the FC's reply.

This isolates one FC's behavior from the live cluster: you control exactly what it receives and observe exactly what it does.

Testing a statem FC in isolation

A gen_statem node takes no wire messages directly — by design. External events reach it through a gate: a plain receiver node co-located with the FSM that post_event()s each incoming message into the FSM in-process.

Example (the sm FC): SmDaemon is the GenStateM (TIPC 0x8001000D); SmGate is the receiver (TIPC 0x8001001D). Lifecycle messages hit the gate, which forwards them into the FSM. The statem never sees the wire.

So to drive an FSM in isolation you send statem events through the gate via a TIPC test node (the probe), and observe the resulting state on the trace stream:

test → probe (.art tester) → TIPC cast → SmGate → post_event() → SmDaemon FSM
                                                                     │
                                          STATEM trace record  ◄─────┘
                                          (from_state, to_state, data)

Observing state on traces — StatemObserver

A gen_statem FC emits a STATEM trace record on every committed transition, carrying from_state / to_state and the FSM data (the OTP {State, Data} Data term, decoded from the trace payload). rf_theia.runtime.StatemObserver subscribes to that trace firehose via artheia.observer.TraceObserver (TIPC-native — no gRPC/com dependency), filters records for one statem node, and republishes each as a statem_transition event:

bus.publish("statem_transition",
            node=..., from_state=..., to_state=..., data={...})

So a test can assert FSM state without polling — it reacts to the trace:

Wait For Fsm State    sm    RUNNING    within=5s
Assert Fsm Data       sm    boot_count=1

This is the full isolation loop: probe drives events in through the gate; the STATEM trace stream reports the FSM's state and data back out.

Worked example — the sm_gate self-test

The shipped sm_gate self-test (in the consuming repo's testing/scenarios/) exercises exactly this:

• T1 — the supervisor, after children are up, casts SystemBoot + StartupComplete to the gate; the gate post_events them; sm reaches RUNNING (observed on the STATEM trace).
• T2 — a probe re-sends StartupComplete to the gate, impersonating the supervisor. sm is already RUNNING, so the gate forwards it and the FSM receives-and-ignores it (no transition) — proving the gate path and the idempotent handling, all driven by the probe.

When to use this

Goal	Approach
Drive one FC without the rest of the stack	probe bound from a tester .art, peer FC stopped
Inject a precise event sequence into an FSM	probe casts to the gate over one ordered connection
Assert FSM state / data	StatemObserver + Wait For Fsm State / Assert Fsm Data
Set up cluster topology for a test	probe drives the supervisor's control surface

See also: Writing-Scenarios, MCP-Server.