Fix multi-GPU model load hang: probe NCCL before trusting it, prefaul… - #114
Conversation
…t GGUF reads A TP model load on ggml_cuda (--tp 2) appeared to hang for 10+ minutes on cloud multi-GPU hosts. The model load itself finished in seconds; the hang was the first fused-TP decode inside WarmUpKernels. On these hosts the driver advertises GPU P2P (nvidia-smi topo -p2p all OK, NCCL topology detection reports intraNodeP2pSupport=1) but peer traffic never actually arrives: ncclCommInitAll succeeds in 0.35s and the first ncclAllReduce then enqueues kernels that spin forever waiting for their peers - both GPUs pinned at "100% util" in spin kernels, host pinned in stream polling. Capability flags cannot be trusted, so the fix verifies behaviour: - New tp_probe_cuda_collective (ggml_ops_tp_probe.cu): before ggml's comm init, run one small AllReduce end to end through dlopen'd NCCL on throwaway communicators and private non-blocking streams, bounded by a deadline (TS_GGML_TP_AR_PROBE_MS, default 10s), and verify the sums. On a wedge the probe recovers with ncclCommAbort and tp_comm_ensure reroutes ggml's selection to the pinned-host-memory "internal" AllReduce pipeline (which needs no P2P) via GGML_CUDA_ALLREDUCE=internal. The verdict is cached per driver/NCCL/PCI-bus-id set so a broken host pays the timeout once, not on every load. Explicit GGML_CUDA_ALLREDUCE skips the probe; inconclusive results (setup/init errors) are not cached and do not reroute. Windows and NCCL-less builds are no-ops. - GgufFile.PrefaultFileCache (called at LoadWeights entry): warm the page cache with 16 parallel sequential read streams, each with its own file handle (FUSE filesystems keep per-open readahead state; sharing one handle serialized the streams). The load path otherwise reads the file at one-or-two-stream speed, which was the whole cold-load time on network-backed model storage. Skips files larger than half of available RAM; TS_GGUF_PREFAULT=0 disables. Measured on 2x A40 (RunPod, MooseFS model storage), gemma-4-E4B Q8_0 --tp 2: load+warmup+first answer went from stuck (>10 min, killed) to 15.4s total; decode 74-83 tok/s (single GPU: 62). Cold loads: gemma-26B 52s -> 12.2s, Qwen3.5-35B 110s -> 14.8s (prefault reads at 2.3-2.5 GiB/s vs ~0.3 GiB/s effective before). All models in the test fleet (gemma-4-E4B, gemma-4-26B MoE, Qwen3.5-9B, Qwen3.5-35B MoE, all --tp 2; DeepSeek-V4-Flash 4-GPU) pass short/long/multi-turn/image tests with correct outputs, exact needle recall, and unchanged inference speed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59bb057a7c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| std::string key = "driver=" + std::to_string(driver_version) | ||
| + " nccl=" + std::to_string(nccl_version) + " devs="; |
There was a problem hiding this comment.
When $XDG_CACHE_HOME or $HOME is shared across cluster nodes, the driver/NCCL versions and PCI bus addresses can be identical even though the physical or virtualized peer path differs. An ok verdict written by a healthy node then causes a broken node to skip the behavioral probe and restores the indefinite first-AllReduce hang this change is intended to prevent; include a machine identity and preferably GPU UUIDs in the key, or keep the cache in host-local storage.
Useful? React with 👍 / 👎.
| NcclApi api; | ||
| if (!api.load()) | ||
| return -1; // No NCCL in the process: the backend cannot pick it either. |
There was a problem hiding this comment.
When TENSORSHARP_GGML_NATIVE_ENABLE_NCCL=OFF is used on a host that still has libnccl.so, this runtime dlopen succeeds even though CMakeLists.txt compiled ggml with GGML_CUDA_NCCL=OFF. Such a build unnecessarily invokes the explicitly disabled NCCL path and, on the targeted broken hosts, can spend the full probe timeout launching and aborting wedged kernels even though ggml would have selected its fallback directly; gate this code using the build-time NCCL decision.
Useful? React with 👍 / 👎.
| dir += "/tensorsharp"; | ||
| ::mkdir(dir.c_str(), 0755); | ||
| return dir + "/tp-collective-probe"; |
There was a problem hiding this comment.
On a fresh account where $HOME/.cache does not exist, or when the configured $XDG_CACHE_HOME parent is absent, this single mkdir fails with ENOENT and the later cache write silently fails. Each new process consequently repeats the NCCL probe, making affected hosts pay the timeout on every invocation; create the parent hierarchy before returning the cache path.
Useful? React with 👍 / 👎.
Engine comparison — TensorSharp vs llama.cpp (PR smoke)No report artifact was produced — the benchmark failed before generating results (see the workflow logs). |
1 participant
…t GGUF reads
A TP model load on ggml_cuda (--tp 2) appeared to hang for 10+ minutes on cloud multi-GPU hosts. The model load itself finished in seconds; the hang was the first fused-TP decode inside WarmUpKernels. On these hosts the driver advertises GPU P2P (nvidia-smi topo -p2p all OK, NCCL topology detection reports intraNodeP2pSupport=1) but peer traffic never actually arrives: ncclCommInitAll succeeds in 0.35s and the first ncclAllReduce then enqueues kernels that spin forever waiting for their peers - both GPUs pinned at "100% util" in spin kernels, host pinned in stream polling. Capability flags cannot be trusted, so the fix verifies behaviour:
New tp_probe_cuda_collective (ggml_ops_tp_probe.cu): before ggml's comm init, run one small AllReduce end to end through dlopen'd NCCL on throwaway communicators and private non-blocking streams, bounded by a deadline (TS_GGML_TP_AR_PROBE_MS, default 10s), and verify the sums. On a wedge the probe recovers with ncclCommAbort and tp_comm_ensure reroutes ggml's selection to the pinned-host-memory "internal" AllReduce pipeline (which needs no P2P) via GGML_CUDA_ALLREDUCE=internal. The verdict is cached per driver/NCCL/PCI-bus-id set so a broken host pays the timeout once, not on every load. Explicit GGML_CUDA_ALLREDUCE skips the probe; inconclusive results (setup/init errors) are not cached and do not reroute. Windows and NCCL-less builds are no-ops.
GgufFile.PrefaultFileCache (called at LoadWeights entry): warm the page cache with 16 parallel sequential read streams, each with its own file handle (FUSE filesystems keep per-open readahead state; sharing one handle serialized the streams). The load path otherwise reads the file at one-or-two-stream speed, which was the whole cold-load time on network-backed model storage. Skips files larger than half of available RAM; TS_GGUF_PREFAULT=0 disables.
Measured on 2x A40 (RunPod, MooseFS model storage), gemma-4-E4B Q8_0 --tp 2: load+warmup+first answer went from stuck (>10 min, killed) to 15.4s total; decode 74-83 tok/s (single GPU: 62). Cold loads: gemma-26B 52s -> 12.2s, Qwen3.5-35B 110s -> 14.8s (prefault reads at 2.3-2.5 GiB/s vs ~0.3 GiB/s effective before). All models in the test fleet (gemma-4-E4B, gemma-4-26B MoE, Qwen3.5-9B, Qwen3.5-35B MoE, all --tp 2; DeepSeek-V4-Flash 4-GPU) pass short/long/multi-turn/image tests with correct outputs, exact needle recall, and unchanged inference speed.