docs(examples): add a GLM-5.2 1P1D SGLang deployment example - #84
Merged
Conversation
dorado269
force-pushed
the
yihou.dev.glm52.example
branch
from
August 4, 2026 08:21
1413784 to
d7c8223
Compare
dorado269
force-pushed
the
yihou.dev.glm52.example
branch
from
August 4, 2026 08:27
d7c8223 to
ded7954
Compare
dorado269
marked this pull request as ready for review
August 4, 2026 08:27
dorado269
requested review from
JohnQinAMD,
jiejingzhangamd,
limou102 and
xiaobochen-amd
as code owners
August 4, 2026 08:27
dorado269
force-pushed
the
yihou.dev.glm52.example
branch
3 times, most recently
from
August 4, 2026 09:43
65653a5 to
eb7951d
Compare
dorado269
commented
Aug 4, 2026
| | preflight says | wrapper | why | | ||
| |---|---|---| | ||
| | `peermem: present`, **mode A viable** | [`cluster.peermem.sh`](cluster.peermem.sh) | bare `ibv_reg_mr` hands the NIC the GPU pages directly — nothing pinned, KV pool not duplicated, **every** rail can carry KV | | ||
| | `peermem: absent`, **mode B viable** (an ODP NIC exists) | [`cluster.dmabuf.sh`](cluster.dmabuf.sh) | dma-buf is the only GPUDirect path without peer-mem, and it is only safe on an ODP NIC; KV is locked to that one NIC | |
dorado269
force-pushed
the
yihou.dev.glm52.example
branch
2 times, most recently
from
August 4, 2026 09:54
114dc96 to
32f02e5
Compare
A runnable kit for GLM-5.2-MXFP4 on two MI355X nodes: one prefill leg and one
decode leg, KV moved between them over Mooncake RDMA, fronted by the infera
router, with DP-attention, MTP (EAGLE) and the kvd cache tiers on.
Everything site-specific lives in one of two files under cluster/ -- one per
RDMA registration mode, picked by the preflight probe rather than by
autodetection at launch. common.sh and engine/*.sh carry the tuned recipe and
contain no addresses, paths, NIC names or GID indices.
Three settings in engine/leg.sh are coupled in ways that produce a plausible
result rather than an error when got wrong, so each is commented in place and
written up in the README:
* --ep-size is emitted unconditionally, outside the DP-attention branch.
Expert and attention parallelism are different axes; gating both on one
condition collapses the MoE whenever DP-attention is off.
* --chunked-prefill-size is a global budget that SGLang divides by dp_size
only when DP-attention is on. One value serves both modes; a per-rank
number hardcoded in a DPA-off branch cuts the global budget 8x.
* Prefill activation OOM is fixed by LOWERING --mem-fraction-static, the
opposite of the decode-side retract fix. Diagnose by phase.
The host RDMA provider library, if the image injects one, needs HOST_RDMA_LIB,
HOST_RDMA_MOUNT and ENTRYPOINT_KEEP together: the mount path must be the one
that image's entrypoint reads. Set only some of them and the library lands
where nothing reads it, which is not an error -- the leg boots and serves,
the container sees ZERO RDMA devices, and mooncake moves KV 5-20x slower.
start_container now warns on a zero device count rather than printing it as a
number to skim past, and smoke.sh reports the count alongside its MC_FORCE_TCP
and GID-is-NULL counters, which are both 0 in that state too.
Three checks in smoke.sh were reading nothing and printing a pass:
* engine logs carry binary bytes, so a bare grep answers "binary file
matches" and every counter reads 0. Filter through strings.
* the Rust router prints `router_policy: "kv-aware"` inside a colourised
Config{...} dump, never `router-policy=<x>`; the old pattern printed empty
whatever the policy was. It now also reports whether the tokenizer loaded,
without which kv-aware silently degrades to load-only routing.
* MTP acceptance was read off `tail -5`, and a few percent of batches sit at
4.00 even on a healthy leg (measured 4.7% at median 2.88), so that tail
routinely reads as a repetition loop. Report the distribution and the
median.
results/ carries concurrency-8 numbers from two independent agentic benchmarks
on two clusters with different fabrics, recomputed from raw per-request records.
The cross-cluster gap is presented as context rather than as a measurement:
more than one variable moves between those clusters, four candidate causes are
listed, and none is confirmed. The customer's benchmark is referenced by URL and
not vendored; this kit ships no agentic bench client, only a service self-check
and a reference sweep using SGLang's own bench_serving.
The AgentX Case-A harness is closed-loop, not open-loop: --concurrency N runs
N workers and each waits for its response before issuing the next. In-flight
sat at mean 5.13 against a cap of 8, which pinned lanes could not produce. The
difference from the infera bench is the session population (fixed vs growing),
not the control loop. Its corpus also does not meet the workload spec's ~88-89%
prefix-hit target: met per turn (median 88.0%) but not overall (mean 66.0%,
token-weighted 59.7%), for three reasons stated with the generator's own
construction -- turn 0 is all-miss and inside the target, the draw is per-turn
and unweighted by tokens, and 51% of turns are shorter than their predecessor,
which permanently truncates the accumulated prefix. No claim is made about a
usable concurrency limit; the c8/c16 shape change is kept as an observation.
The engine image is stated as an infera-sglang build newer than 0.2.0, carried
as the literal placeholder <infera-sglang-image>. The README states plainly
which parts are validated end-to-end and which are not: the peer-mem wrapper
and round-robin routing were never exercised, as no such cluster was available.
Signed-off-by: yihou <yihou@amd.com>
dorado269
force-pushed
the
yihou.dev.glm52.example
branch
from
August 4, 2026 09:54
32f02e5 to
7609bb1
Compare
xiaobochen-amd
approved these changes
Aug 4, 2026
llying-001
added a commit
that referenced
this pull request
Aug 4, 2026
Not this PR's code. examples/sglang_1p1d_glm5.2/README.md arrived on main with 33 trailing spaces on one line of its topology diagram (#84, 7609bb1), and CI's lint job runs `pre-commit run --all-files`, so the trailing-whitespace hook fails for any branch rebased onto current main -- and `unit` needs `lint`, which blocks everything behind it. A pristine origin/main worktree reproduces it with none of this branch's changes present. Whitespace only: the rendered line is byte-identical, and those spaces padded nothing because the diagram has no right border. It is the only such line in the repo. Kept as its own commit so it can be dropped if the fix lands on main separately. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: liyingli <liyingli@amd.com>
jiejingzhangamd
added a commit
that referenced
this pull request
Aug 4, 2026
Not from this branch. examples/sglang_1p1d_glm5.2/README.md arrived on main in 7609bb1 (#84) with a trailing-whitespace line, which the pre-commit hook rejects -- so the lint job fails on the merge result for every open PR, including this one, while each branch passes on its own. Fixed here only because it blocks this PR's CI. One line, whitespace only, no content change. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a runnable deployment example for GLM-5.2-MXFP4 under
examples/sglang_1p1d_glm5.2/:one prefill node and one decode node, KV moved between them over Mooncake RDMA, fronted by the
infera router, with DP-attention, MTP (EAGLE speculative decoding) and the kvd cache tiers on.
Everything site-specific lives in one of two files under
cluster/— one per RDMA registrationmode, picked by the preflight probe rather than by autodetection at launch.
common.shandengine/*.shcarry the tuned recipe and contain no addresses, paths, NIC names or GID indices,so adapting to a cluster means editing a wrapper, never an engine script.
Directory shape mirrors
examples/deepseek_v4(common.sh+ anengine/subdirectory).Type of change
Changes
cluster/— two wrappers, the only files a user edits:cluster.peermem.sh(multi-railwith a peer-memory module) and
cluster.dmabuf.sh(single ODP NIC, no peer-mem). Noautodetection:
preflight_rdma.sh modetells you which one to use.engine/leg.sh— the launcher for one PD leg. Carries the tuned recipe and nothingsite-specific.
engine/up.sh/down.sh— bring up both nodes (containers → etcd + kvd → both legs →router) and tear down, waiting for VRAM to actually drain.
engine/smoke.sh— service check plus positive evidence for each of the five features.A green
/healthproves the process is alive, not that PD paired or that speculation isdoing anything; every check here goes red if its feature is silently absent.
engine/bench.sh— reference throughput sweep using SGLang's ownbench_serving.preflight_rdma.sh— registration-mode probe and cross-node fabric measurement.results/— concurrency-8 numbers from two independent agentic benchmarks on two clusters.Three couplings that fail quietly
Each of these produces a plausible result rather than an error when got wrong, so each is
commented at the point of use and written up in the README's "Notes & gotchas":
--ep-sizeis emitted unconditionally, outside the DP-attention branch. Expert andattention parallelism are different axes; gating both on one condition collapses the MoE
from ep8 to the TP default whenever DP-attention is off, so a run billed as "DPA off"
differs in the expert-dispatch collective too and no latency delta is attributable.
--chunked-prefill-sizeis a global budget that SGLang divides bydp_sizeonly whenDP-attention is on. One value serves both modes; hardcoding the per-rank number in a DPA-off
branch cuts the global budget 8×.
--mem-fraction-static— the opposite ofthe decode-side retract fix. Low token usage at the moment of the abort is the tell: the KV
pool is nearly empty, so it was never KV exhaustion.
On
results/Numbers are recomputed from raw per-request records, not copied from summary lines. The
cross-cluster comparison is presented as context, not a measurement: more than one variable
moves between those clusters, four candidate causes are listed with the evidence behind each,
and none is confirmed. Where a benchmark's own summary metric is misleading it is called out —
the customer harness's
Theoretical Prefix Cache Hitis computed from the trace file and neverasks the server, so it is invariant to the deployment under test.
The customer's benchmark is referenced by URL, not vendored; this kit ships no agentic bench
client, only the service self-check and the
bench_servingreference sweep.Checklist:
No tests. This is a documentation/example change: shell scripts that drive a two-node GPU
deployment, with no unit-testable surface and no CI cluster to run them against. What was done
instead is recorded below.
Validation status
Stated plainly rather than implied — the README carries the same table.
preflight mode→up→smoke→bench→downon a 2-node MI355X mode-B cluster, no edits outside the wrapper. Long context checked separately (needle, to 238K tokens) and under a real agentic workload at concurrency 8preflight_rdma.sh fabricmodewas exercisedcluster.peermem.sh,round-robinroutingkv-awaredefault is what ranStatic checks:
bash -non all 9 scripts; every infera flag grepped againstinfera/**/args.pyon
main; the deliverable grepped for leaked local information (host names, absolute paths,scheduler identifiers, credentials) before submission.
Known placeholder
The image tag
inferaimage/infera-sglang:0.2.0is a placeholder, flagged as such in theREADME. It needs replacing with the released infera-sglang tag before this is useful to a
reader.