docs: align README, docs, and website with GGML-backend tensor parallelism - #105
Conversation
…elism
Tensor parallelism now runs on the GGML CUDA / Vulkan backends in addition
to the direct CUDA backend, both TensorSharp.Cli and TensorSharp.Server
expose --tp / --tp-node-id / --tp-peers, and the CLI has a full --help
page. The docs still described TP as CUDA-only, the server as having no
--tp flag, and GGML TP as a capacity-only feature that loses to a single
GPU. Update every surface to match.
- README / README_zh-cn: TP highlight, quick-start (ggml_cuda example,
server flags, CLI/server --help), Current Status row.
- FEATURES / FEATURES_zh-cn: backend coverage, rank worker pool vs CUDA
streams, MoE sharding strategies split into expert slicing vs expert
parallelism vs Gemma 4's in-expert Megatron split, Stage 1c numbers.
- USAGE / USAGE_zh-cn: fix the contradictory "Qwen 3.5 is direct-CUDA
only" row (its GGML TP path shipped), replace the stale "a single GPU
is still faster" note with the measured 1-GPU vs --tp 2 table, document
TENSORSHARP_TP_DEVICES / TS_GEMMA4_TP_FUSED_MOE /
GGML_CUDA_AR_BF16_THRESHOLD, add the CLI --help section, and port the
whole backend-support section to Chinese.
- docs/env_var_feature_matrix(_zh-cn): TP applies to cuda, ggml_cuda and
ggml_vulkan; add the GGML TP knobs.
- docs/models/{gemma4,qwen35}(_zh-cn): new tensor-parallelism sections
covering the packed per-rank GDN kernel, expert parallelism, the fused
MoE trunk, and their measurements.
- website: rewrite distributed(_zh-cn) (GGML local TP, server-as-node-0,
per-architecture strategies, new measured-results section, GGML tuning
knobs, corrected troubleshooting), plus backends, cli, server,
api-reference, features, overview, index, and both search indexes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URJi1NhfCz3eKP9hMUCWmo
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70e2291803
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | `TENSORSHARP_TP_HOST_ALLREDUCE` | local TP, `cuda` backend | `1` runs the local AllReduce through host memory (device→host, sum, host→device) instead of the device-to-device path — diagnostic fallback | off (device-to-device) | not registered | no | | ||
| | `TS_GGML_TP_PARALLEL` | local TP, GGML backends | `0` drives the ranks sequentially instead of concurrently (diagnostic) | on (concurrent rank workers) | not registered | no | | ||
| | `TS_GGML_TP_FUSED_MATMUL` | local TP, GGML backends | `1` submits both ranks' linears from one thread; allocates a device buffer per rank per call and measured 2.3× slower on Qwen 3.5 35B | off (generic per-rank path) | not registered | no | | ||
| | `TS_GGML_TP_DEVICE_AR_THRESHOLD` | local TP, GGML backends | Element count above which AllReduce uses the device collective instead of the host reduction | `262144` | not registered | no | |
There was a problem hiding this comment.
For --backend ggml_vulkan, this row incorrectly promises that raising or lowering the threshold switches large reductions to a device collective. GgmlTensorParallelGroup.AllReduce only attempts that route when _context.HasDeviceAllReduce, while the native TP implementation explicitly uses host staging for Vulkan, so this variable is a no-op there. Scope the entry to ggml_cuda when a device collective is available, and document that Vulkan remains on the host reduction.
Useful? React with 👍 / 👎.
Engine comparison — TensorSharp vs llama.cpp (PR smoke)No report artifact was produced — the benchmark failed before generating results (see the workflow logs). |
2 participants
Tensor parallelism now runs on the GGML CUDA / Vulkan backends in addition
to the direct CUDA backend, both TensorSharp.Cli and TensorSharp.Server
expose --tp / --tp-node-id / --tp-peers, and the CLI has a full --help
page. The docs still described TP as CUDA-only, the server as having no
--tp flag, and GGML TP as a capacity-only feature that loses to a single
GPU. Update every surface to match.
server flags, CLI/server --help), Current Status row.
streams, MoE sharding strategies split into expert slicing vs expert
parallelism vs Gemma 4's in-expert Megatron split, Stage 1c numbers.
only" row (its GGML TP path shipped), replace the stale "a single GPU
is still faster" note with the measured 1-GPU vs --tp 2 table, document
TENSORSHARP_TP_DEVICES / TS_GEMMA4_TP_FUSED_MOE /
GGML_CUDA_AR_BF16_THRESHOLD, add the CLI --help section, and port the
whole backend-support section to Chinese.
ggml_vulkan; add the GGML TP knobs.
covering the packed per-rank GDN kernel, expert parallelism, the fused
MoE trunk, and their measurements.
per-architecture strategies, new measured-results section, GGML tuning
knobs, corrected troubleshooting), plus backends, cli, server,
api-reference, features, overview, index, and both search indexes.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01URJi1NhfCz3eKP9hMUCWmo