TrainerRank CP4: deterministic NCCL all-to-all hang in the context-parallel group on two real Ellavox groups

[bradhilton]

Found by the cost-model calibration harness (#836) and reproduced on a second fresh 4x H200 cluster. Two of the 44 Ellavox groups hang at CP4 (TP1, Qwen3.5-4B full model, active LoRA slot) with an NCCL watchdog timeout on ALLTOALL_BASE in CONTEXT_PARALLEL_GROUP; the ranks stop at different collective sequence numbers, so the CP ranks disagree on their exchange plan. The same groups run correctly at CP1 and CP2 with every layout.

Repro (deterministic, both runs hung at the identical point):

ART_MEGATRON_TENSOR_MODEL_PARALLEL_SIZE=1 ART_MEGATRON_CONTEXT_PARALLEL_SIZE=4 \
  torchrun --standalone --nproc-per-node=4 dev/trainer_rank_landing_acceptance.py \
  --phase cost-calibrate --cell cal-ellavox --group 4 --model Qwen/Qwen3.5-4B --repeat 8 --evidence /tmp/e.jsonl

• Group 4 (3 histories of 7,441 / 11,949 / 12,075 tokens): no_sharing and depth_one execute fine (3 + 2 runs); the first execution of the uniform_depth_2 layout hangs, both runs at NCCL SeqNum 721. That layout is nested sharing: a 494-token root shared by all three, an 11,150-token segment shared by two, and 305 / 431-token tails. The version-1 production score selects exactly this layout for the group at CP4, so dp_rank_forward on this data at CP4 would hang today. (The fitted table in Fit the TrainerRank layout cost model (coefficient version 2) with a calibration campaign #836 selects the neighbouring minimum_effective_span_495 layout, which is incidental, not a fix.)
• Group 1 (2 histories of 12,296 / 12,349 tokens): no_sharing runs three times and depth_one twice without issue; the first no_sharing run after those (the measured round) hangs, in both runs. Switching back to an already-executed layout is the trigger here, which points at state carried between calls in the CP runtime (bundle / rank-plan cache) rather than at the layout itself.

Both logs show the hang immediately after "Loading compiled function from disk" lines for the flash-attention cute-DSL backward kernels; the timing-out all-to-alls have NumelOut of 11.9M–18.5M elements (CP attention KV exchange) and one rank sending with NumelOut=0.

Evidence: scratch/trainer_rank_cost_calibration/cost2-*/cal-ellavox-0-g{1,4}.log and cost3-*/... on the (torn-down) clusters; local copies of the logs and the JSONL rows are with the calibration campaign artifacts. The two cells are excluded explicitly from the calibration certificate in #836 pending this fix.

Not caused by #826/#831/#832/#836 code paths as far as I can tell: the same execution path is exercised by the forced-anchor harness and by production selection, and the other six Ellavox groups plus all synthetic GRPO/heterogeneous shapes run at CP4 across dozens of alternating layouts without issue.