Add OdinLink (odl_tb5) TP transport: --transport odl

[davidcanar]

Adds --transport odl to the two-node tensor-parallel data plane: the OdinLink (odl_tb5) Thunderbolt RDMA userspace library, loaded with dlopen at runtime exactly like the verbs stack — no link-time dependency, and builds without OdinLink installed fall back to TCP at zero cost. auto now prefers odl over verbs over tcp.

• Protocol v11: the hello's pad word becomes odl_ok, negotiated like rdma_ok.
• Each rank receives on its own stream id; a zero-byte ODL_READY frame at slab attach is the rendezvous (a send to a stream the peer has not opened yet is dropped by its kernel demux).
• Gate exchanges: one stream_send per direction (the kernel fragments at 4024-byte frames internally); receive side runs stream_recv on an O_NONBLOCK fd with the same peer-death sampling and gate deadline as the verbs CQ poll loop.
• Batch/big gates ride a symmetric bulk swap with a 4 MiB send-ahead window (bounded by the kernel frame ring).
• Loss recovery (second commit, 51f2b49): every stream message carries a 16-byte header and receivers request the chunks the driver dropped; details in the section below.
• Batch and big gates skip the TCP header rendezvous (third commit, 15b6d6b): the reliable exchange numbers its rounds and validates chunk sizes, so the header round trip only added ~150 us per gate (harness batch gate 265 -> 104 us; about 6 ms of a GLM MTP verify cycle).
• No vendor header dependency: an 8-entry function table mirrors odl_tb5.h signatures through void pointers.
• tp_odl_test.c included: a model-free transport harness that drives the real ds4_tp.o on both ranks with a synthetic identity, verifies payload integrity, and prints latency stats.

Test evidence (per CONTRIBUTING.md)

Machine: 2x AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 128 GB), Fedora 44, Thunderbolt 5 direct link, OdinLink-Five driver odl_tb5, ROCm build (make strix-halo).

Transport-level A/B on both ranks (24 KB gate vectors, payload integrity verified):

RESULT transport=odl vec=24576B iters=3000 min=6.4 p50=30.1 p90=41.9 p99=52.4 max=675.8 mean=33.1 us
RESULT transport=tcp vec=24576B iters=3000 min=17.2 p50=31.0 p90=38.2 p99=60.5 max=119.4 mean=32.3 us
BATCH transport=odl rows=8 bytes=196608 rtt=290.7 us
BIG   transport=odl bytes=2097152 rtt=1843.6 us   (~1.1 GB/s per direction)

A 50,000-gate soak is clean (no desync, no corruption). Notably, TCP over thunderbolt_net shows load-dependent variance (p50 between 31 and 262 us across runs taken while a large background download ran) while odl stays at ~30 us throughout — kernel bypass removes the tail under system load.

End-to-end (GLM 5.3 Flash Q4_K across the same two boxes, short-prompt exact prefill + decode, ctx 8192): prefill 25.8 t/s on odl vs 18.4 t/s on tcp; generation 8.7 t/s on both. (Decode turned out to be memory-bandwidth-bound on the replicated BF16 KDA attention weights rather than compute-bound; the kernel-level breakdown is in #1024.)

Loss recovery (added after the initial submission, 51f2b49)

In production the first version died twice in one afternoon: once in an MTP verify block, once 4096 tokens into an 11k-token ds4-server prefill. Both times dmesg on the receiving rank shows odl_tb5: RX CRC error ... frame dropped followed by fragment gap ... dropping message: when one Thunderbolt frame of a message fails CRC, the driver drops the whole message and the sender is never told. The gate then parks both ranks on their spin kernels (100% GPU) until the timeout and the TP session is dead for good. On this pair the link negotiates 10 Gb/s per lane (2 lanes) and drops a frame every few GB, so any long prefill was doomed.

The fix keeps the stream API and adds recovery on top:

• Every message carries a 16-byte header: magic, kind (DATA/NACK), chunk index, exchange sequence (both ranks number exchanges identically).
• A receiver asks for the first missing chunk as soon as a later index reveals the gap, or after a retry interval without progress: 2 ms for gate vectors, 50 ms for bulk (lockstep skew is far below both).
• A sender keeps the framed copies of its current and previous exchange. Lockstep bounds the peer's lag to one exchange (it cannot finish N+1 without our N+1 data, which we only send once we finished N), so those two buffers always cover a request. Chunks of the next exchange that arrive while the current one still misses data are parked and replayed.
• Requests are served from the exchange loop and, for a rank that is idle or waiting on the control channel (worker between commands, leader waiting for an ack, either side in the big-gate header handshake), from a polling wait that now fronts those reads. Without this an idle rank would leave a peer stuck one exchange behind until its deadline.
• No extra round trip on the fast path: a decode gate costs one 16 KB frame copy per side. DS4_TP_ODL_DROP_TEST=N drops every Nth data send as a diagnostic for this path; off unless set.

Harness over the real link (16 KB gate vectors, 3000 iterations, payload verified on every receive):

no drops:                 RESULT p50=29.0 p90=32.3 p99=52.7 us   BIG 2 MiB rtt=2196.7 us (1843.6 before: the header/frame copies)
DROP_TEST=5, both ranks:  RESULT p50=31.0 p90=2065.3 us, 751 drops -> 753 retransmissions, all payloads verified
DROP_TEST=3, leader only: RESULT p50=62.7 p90=2077.8 us, 1501 drops recovered, all payloads verified
no handshake (15b6d6b):    BATCH rows=8 rtt=104.3 us (264.6 before), gate p50 29.2 us unchanged

The p90 under injected loss is the 2 ms detection interval; the bulk case with the last chunk dropped waits the 50 ms interval.

End-to-end with the GLM-5.3-Flash TP setup of #1024 (2x gfx1151, odl, Q8-attention GGUF): decode 15.35 t/s without and 15.23 t/s with DS4_TP_ODL_DROP_TEST=400 (12 dropped gate messages per rank recovered, greedy output byte-identical over 96 tokens); a 323-token prompt prefilled at 69.1 t/s without and 67.5 t/s with DROP_TEST=50 (75 drops per rank recovered, output identical). Without injected drops the ranks still request a retransmission about once per second: a rank that reaches a gate more than the 2 ms interval late is asked as well, which costs a duplicate 16 KB send and nothing else, so the log prints only the first few and then every 100th request. With the batch/big-gate handshake removed the GLM MTP cycle went from 10.4 to 11.8 t/s and the 323-token prefill from 69 to 78 t/s (measurements in #1024).

Known limitation: bulk throughput is bounded by the odl stream API's per-fragment kernel path (~1.1 GB/s per direction, and the observed link speed above); the double-buffer DMA API should raise it and is left as future work. The full make test suite was not run for this change (needs the CPU test binary on hand); transport behavior was validated with the included harness on both ranks plus the end-to-end runs above.

🤖 Generated with Claude Code