JACCL Recv/Send -12 on 4-node TB5 after a successful first generate

[guruswami-ai]

JACCL Recv/Send -12 on a 4-node TB5 mesh after a successful first generate

Four M3 Ultra 512 GB Studios, Thunderbolt 5, JACCL, mlx.launch, world=4, MLX_METAL_FAST_SYNCH=0. Hostfile names rdma_enN devices. After a full-fleet reboot we run mlx.distributed_config --auto-setup once and check 3 PORT_ACTIVE devices per node (6 edges).

We are serving DeepSeek-V4-Pro-0813 TP4 (MXFP4/MXFP8, DSpark via an omlx overlay). Load + first generate is usually fine (e.g. 128 tokens at 19 tok/s, all ranks agree). A later generate in the same process (second BatchGenerator, second HTTP /v1/chat/completions, or second DeepSeek Harness turn) often dies with:

libc++abi: terminating due to uncaught exception of type std::invalid_argument:
[jaccl] Recv failed with error code -12

or Send failed with error code -12. Rank 1 or 3 exits 255. mlx.launch then prints exit 0. That exit code is wrong and hid a failure from our wrappers.

What is not sufficient to explain every -12

• Rank-divergent uuid4() nonces: we removed those. Lengths are all-gathered.
• Missing shared mx.random.seed: without it, sampled decode parked at 49/48/45/47 tokens and died. With a shared seed, sampled works. The remaining -12 still happens on greedy second requests.
• Rank 0 entering generate before workers: we added an all-gather arm barrier (same idea as a control-socket "armed" handshake). First-request desync is rarer. Second-request death remains.

After a crash, leftover Python ranks must be killed. If a rank is stuck Us+ and ibv_devinfo hangs, only a full-fleet reboot plus a fresh --auto-setup recovers the mesh.

Why we are filing this here

The Python overlay can be wrong and we will keep hardening it. The combination of (1) death on a second collective sequence after a long load of all-gathers, (2) mlx.launch reporting 0, and (3) RDMA needing a fleet reboot, looks like a transport/lifecycle problem as much as an app bug.

A four-node TB5 JACCL soak that does: load a large sharded model, generate, idle a few seconds, generate again, repeat, would catch this. We can help reproduce.

The rest of the stack (Pro-0813 + DSpark + gather, 15–23 tok/s interactive, 32 k context) is written up on mlx-lm and omlx discussions in the same batch. The model is usable when the mesh stays up. The mesh is the part we cannot fix from Python alone.

[guruswami-ai]

Recipe and scripts (convert / shard_mtp / BatchGenerator smoke / OpenAI serve). No converted weights — rebuild from deepseek-ai/DeepSeek-V4-Pro-0813.

https://huggingface.co/guruswami-ai/deepseek-v4-pro-0813-mlx-tp4-recipe

[guruswami-ai]

Follow-up: undrained BatchGenerator.close()

Same four-node TB5 / JACCL / Pro-0813 TP4 DSpark path. One load, two greedy 32-token generates, mlx.launch still prints exit 0 on a rank abort.

cell	after generate 1	idle	generate 2
A	close(), no mx.synchronize()	0 s	C++ Recv failed with error code -12
D	leave generator open, new generator for turn 2	0 s	ok
F	mx.synchronize() then close()	0 s	ok
H	same as F	30 s quiet	ok
I	same as F	300 s quiet	ok

Generate 1 is always fine. The abort is an uncaught std::invalid_argument, not a Python exception. Shared seed and the arm barrier were already in place; they do not cover this class.

Serve now drains with mx.synchronize() before close(). That is a workaround, not a library fix. A third generate and a multi-hour soak are not certified.

Scripts: https://huggingface.co/guruswami-ai/deepseek-v4-pro-0813-mlx-tp4-recipe

[guruswami-ai]

Follow-up: HTTP serve after a long idle

The drain (mx.synchronize() before BatchGenerator.close()) is now in the singleton OpenAI server, not just the ladder.

One mlx.launch process sat idle for about 2.5 hours after load, then three sequential POST /v1/chat/completions (greedy, 32 new tokens, same short prompt):

turn	cache	TTFT	decode	DSpark accept	HTTP
1	miss	0.70 s	18.6 tok/s	87%	200
2	trim (kept the 9-token prompt)	1.23 s	15.1 tok/s	73%	200
3	trim	0.29 s	14.8 tok/s	77%	200

No Recv/Send -12. The process is still serving.

This is not a multi-hour generate soak. It is the idle-then-second class on the real HTTP path, including a 2.5 h gap that is longer than the 300 s ladder cell.

The C++ abort if you close() without draining is unchanged. mlx.launch still prints exit 0 when a rank dies.

[guruswami-ai]

Follow-up, 15 August 2026

Same four-node TB5 JACCL mesh. We split the original “second generate dies with -12” report. There are at least two classes.

1. Undrained BatchGenerator.close() (Python-visible -12)

One-load cell matrix, all ranks on the same path:

• Cell A: close() with no mx.synchronize(), immediate second generate → C++ Recv failed … -12.
• Cell F: mx.synchronize() then close() → second generate lives.
• Leave the generator open (D), or sync+close then idle 30 s / 300 s (H / I) → also live.

Serve now synchronises before every close(). This class is no longer what kills us overnight. mlx.launch still prints exit 0 when a rank exits 255. Wrappers must grep the fail line, not trust the status.

2. Thunderbolt RDMA itself (often not a Python -12)

After a full-fleet reboot, first or second mlx.launch has died in:

tbt_post_send / tbt_post_recv
jaccl::Connection::post_{send,recv}
jaccl::MeshImpl::all_gather
mlx::core::scheduler::StreamThread::thread_fn

EXC_BAD_ACCESS / SIGSEGV. macOS DiagnosticReports name libthunderboltrdma.dylib + libjaccl.dylib. The serve log may only show Node with rank N exited with code 255 and no [jaccl] Recv failed line. Treating every 255 as class 1 will send you down the wrong hole.

Separately, after hours of generate, physical ports drop. We had certified 6/6 edges (3 PORT_ACTIVE per node). Overnight two nodes went to PORT_ACTIVE=2. The supervisor refused to relaunch (correct). mlx.distributed_config on that same boot is not the fix: a failed or partial --auto-setup taints the receipt. Recovery is a full-fleet reboot, then one --auto-setup, then confirm 6 edges again.

A 10-hour HTTP soak after a 129 k prompt did 36/40 short greedy maths and 12/13 4 k prompts, then the fabric dropped. First generate after load is the easy case. Hours later is not certified.

Still true from the first post

• Shared mx.random.seed on every rank, or sampled TP4 desyncs and dies.
• Rank-divergent uuid4() nonces are enough to -12 on their own. Do not do that.
• Kill leftover ranks before the next launch.

Happy to share the cell matrix script and IPS excerpts (no host names).

[guruswami-ai]

Status, 17 August 2026

The 14–15 August numbers stand. DeepSeek-V4-Pro-0813 (MXFP4/MXFP8 + DSpark) loads on four M3 Ultra 512 GB nodes under mlx.launch TP4, and a fresh process decodes at about 12–15 tok/s out to a 129 k prompt. Recipe: https://huggingface.co/guruswami-ai/deepseek-v4-pro-0813-mlx-tp4-recipe

We are parking further TP4 work on this model.

A later isolation needed no model: unique 256 MiB uint8 all_sum every 60 s, seven cycles md5-clean, then SIGSEGV in tbt_post_recv / RingImpl::all_reduce<unsigned char> with all ports still PORT_ACTIVE. That is Apple libthunderboltrdma, not Pro-0813. Until that is fixed, a long-lived or unattended TP4 serve is not a useful product on this mesh. Short attended runs still work.

• Isolation write-up: [BUG] JACCL SIGSEGV in tbt_post_recv during unique all_sum (no unplug, no model, ports still up) #4319
• Apple Feedback Assistant: FB24371487

No patch attached.