Port expanders as serial bus peers

[falkoschindler]

Lizard has two ways to use a second microcontroller: the Expander with Proxy modules, and a SerialBus with peers. With telemetry frames (#340) the serial bus gets a fast, binary way to bring values to the coordinator. The question is whether an expander could become a serial bus peer with one partner, so Lizard only has one concept to maintain.

How the two differ today

	Expander	Serial bus peer
Configuration	in the core startup: led = expander.Output(15) creates the remote module and a proxy	the peer's own startup script, written with bus.send strings or configure.py --serial-bus
Values to the core	text broadcasts, parsed by Owl on every step	telemetry frames (#344), copied into declared properties (#345) without Owl
Commands to the remote	method calls on the proxy are forwarded	bus.send(id, "…")
Remote reboots	proxies stay without their remote modules (#301), the core has to restart	the peer reports Ready., layouts and values recover
Firmware	expander.flash() copies the core's own image via boot and enable pins	OTB update over the bus; the first flash needs another way
Wires	RX, TX, boot, enable	RX, TX

What would get better

• Text broadcasts through Owl are expensive: on one robot the expander's !! lines cost about 40 ms per step (Jens). Frames avoid the parser.
• Proxies, their module defaults, and the reboot problems around them could go away.
• One protocol, one set of tests, one place to optimize.

What would get worse or needs a solution

• Writing the configuration. Defining remote modules inline in the core startup is the expander's biggest strength. A peer needs its own startup script, and writing it through bus.send strings is unpleasant. Options: a host tool that writes one script per microcontroller (see A Python package for talking to Lizard #346), or a Lizard construct that sends a block of code to a peer.
• Initial flashing. The expander can flash an empty microcontroller from the core. A bus peer would need that too, so boot and enable pins stay useful.
• Latency. Not obviously worse. Both paths end in the core's main loop; a bus peer waits for its poll, an expander's line waits for the next step. The hybrid prototype measured 6 ms median from plexus to host on F30. This needs measuring for a single-peer bus.
• Heap. A SerialBus with UART, two queues and a task stack cost about 34 KB in Jens' breakdown, an expander about 13 KB. A one-peer bus might need smaller queues.
• Migration. Existing startup scripts use Expander and proxies. The expander would have to stay for a long time, maybe deprecated, maybe rebuilt on top of the bus.

Open questions

• Can the bus run point to point on the expander's two wires without changes?
• Which expander features does the bus lack besides inline configuration and flashing?
• Is "configure through a host library" good enough, or does Lizard itself need a nicer way to write a peer's startup?
• Would we keep Expander as a thin layer over a one-peer bus, so existing scripts keep working? ※

[JensOgorek]

P0 ran as a serial bus peer on F30: its share of the main loop dropped from 14–17 ms to under 0.1 ms, and the loop holds its 10 ms

On F30 we replaced the port expander P0 with a serial bus peer on the same wires (a prototype on top of #365), together with telemetry frames for the core line and the plexus. With RoSys running, the main loop averaged 10.0 ms over about 2.5 hours, with every cycle within 20 ms; in production, with all values as text, it averages 36 ms and no cycle stays within 20 ms. With frames everywhere except P0, which was still an expander, the loop stayed at 18.4 ms, so the expander was the last big part. Next step: a PR for the bus side (a bus on the peer's console UART and flashing a peer through boot and enable pins) once #365 is merged.

Loop times on F30

Loop period with RoSys running:

Setup	Mean	≤ 12 ms	≤ 20 ms
Production: core line, plexus and P0 as text	35.9 ms	0 %	0 %
Core line and plexus as frames, P0 still an expander	18.4 ms	0.9 %	72.4 %
Frames everywhere, P0 as bus peer (about 2.5 h in several runs)	10.0 ms	97 %	100 %

Work per loop by part, from a diagnostic build, RoSys stopped:

Part	Production, all text	Frames everywhere, P0 as bus peer
P0 (expander broadcasts, then bus)	15.7 ms	0.09 ms
Core line (text, then frame)	9.2 ms	0.17 + 1.34 ms
Plexus bus	3.8 ms	0.10 ms
CAN	3.2 ms	0.75 ms
Rules	1.6 ms	0.75 ms
Wheels	0.9 ms	0.44 ms
Total	35.9 ms	4.6 ms

Answers to the open questions from F30

• Point to point on the expander's two wires: works, but not without changes. P0's link is its console UART, so the peer's SerialBus takes over the console: Serial(3, 1, 115200, 0) is accepted only by a bus, which then relays the peer's output to the coordinator as bus[2]: …. The core keeps its Serial(26, 27, 115200, 1); the bus gets the boot and enable pins the expander had: SerialBus(serial, 1, 25, 14).
• Initial flashing: bus.flash_peer(2) runs the expander's replicator through boot and enable, then writes two startup lines to the blank peer (its serial and its bus), so it comes back as bus node 2. Flashing P0 this way from RoSys worked. As with expander.flash(), the core's main loop stands still for about 50 s, and the plexus bus drops its inbound messages meanwhile (952 of 984 telemetry frames), so the robot has to be at rest.
• Configuration: the host wrote P0's startup over the bus and declared P0's two inputs in a BusTelemetry on the core. The order matters: a core that already declares frames while a peer still pushes text into variables that no longer exist got a 116 ms loop, the failure described in Binary telemetry frames: roadmap #340. Peers have to be configured first.
• Heap: P0 as a second bus took the core's free heap from 46.8 to 26.8 KB. With 16 instead of 32 queue slots per bus it was 43.7 KB, without drops in 10 minutes. Three more simulated arms in BusTelemetry modules then took 16.5 KB more (27.2 KB left); with 32 slots only about 10 KB would have remained. So a bus with one or few peers should have smaller queues.
• Latency: not measured for the P0 link. A poll round on it takes about 5 ms at 115200.
• Other expander features: BMS answers of P0's unmuted serial arrive as relayed lines (up to 247 characters, a JBD answer has about 40); method calls become bus.send(2, "…"), version and pin queries likewise.

Found on the way and still open

• P0 saw a garbled poll every 5 to 10 minutes, once even a poll addressed to the plexus: two tasks shared static line buffers. Fixed in Give every serial bus and every checked line its own buffer #367.
• A reset of the core also resets P0, even with the bus driving the enable pin.
• P0's GPIO15 (battery relay) is a strapping pin with a pull-up while P0 resets or is flashed; to be checked with the circuit before flashing with a charger connected.

※