Skip to content

docs(examples): add a GLM-5.2 1P1D SGLang deployment example - #84

Merged
xiaobochen-amd merged 1 commit into
mainfrom
yihou.dev.glm52.example
Aug 4, 2026
Merged

docs(examples): add a GLM-5.2 1P1D SGLang deployment example#84
xiaobochen-amd merged 1 commit into
mainfrom
yihou.dev.glm52.example

Conversation

@dorado269

@dorado269 dorado269 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds a runnable deployment example for GLM-5.2-MXFP4 under examples/sglang_1p1d_glm5.2/:
one prefill node and one decode node, KV moved between them over Mooncake RDMA, fronted by the
infera router, with DP-attention, MTP (EAGLE speculative decoding) and the kvd cache tiers on.

Everything site-specific lives in one of two files under cluster/ — one per RDMA registration
mode, picked by the preflight probe rather than by autodetection at launch. common.sh and
engine/*.sh carry the tuned recipe and contain no addresses, paths, NIC names or GID indices,
so adapting to a cluster means editing a wrapper, never an engine script.

Directory shape mirrors examples/deepseek_v4 (common.sh + an engine/ subdirectory).

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • cluster/ — two wrappers, the only files a user edits: cluster.peermem.sh (multi-rail
    with a peer-memory module) and cluster.dmabuf.sh (single ODP NIC, no peer-mem). No
    autodetection: preflight_rdma.sh mode tells you which one to use.
  • engine/leg.sh — the launcher for one PD leg. Carries the tuned recipe and nothing
    site-specific.
  • engine/up.sh / down.sh — bring up both nodes (containers → etcd + kvd → both legs →
    router) and tear down, waiting for VRAM to actually drain.
  • engine/smoke.sh — service check plus positive evidence for each of the five features.
    A green /health proves the process is alive, not that PD paired or that speculation is
    doing anything; every check here goes red if its feature is silently absent.
  • engine/bench.sh — reference throughput sweep using SGLang's own bench_serving.
  • preflight_rdma.sh — registration-mode probe and cross-node fabric measurement.
  • results/ — concurrency-8 numbers from two independent agentic benchmarks on two clusters.

Three couplings that fail quietly

Each of these produces a plausible result rather than an error when got wrong, so each is
commented at the point of use and written up in the README's "Notes & gotchas":

  • --ep-size is emitted unconditionally, outside the DP-attention branch. Expert and
    attention parallelism are different axes; gating both on one condition collapses the MoE
    from ep8 to the TP default whenever DP-attention is off, so a run billed as "DPA off"
    differs in the expert-dispatch collective too and no latency delta is attributable.
  • --chunked-prefill-size is a global budget that SGLang divides by dp_size only when
    DP-attention is on. One value serves both modes; hardcoding the per-rank number in a DPA-off
    branch cuts the global budget 8×.
  • Prefill activation OOM is fixed by lowering --mem-fraction-static — the opposite of
    the decode-side retract fix. Low token usage at the moment of the abort is the tell: the KV
    pool is nearly empty, so it was never KV exhaustion.

On results/

Numbers are recomputed from raw per-request records, not copied from summary lines. The
cross-cluster comparison is presented as context, not a measurement: more than one variable
moves between those clusters, four candidate causes are listed with the evidence behind each,
and none is confirmed. Where a benchmark's own summary metric is misleading it is called out —
the customer harness's Theoretical Prefix Cache Hit is computed from the trace file and never
asks the server, so it is invariant to the deployment under test.

The customer's benchmark is referenced by URL, not vendored; this kit ships no agentic bench
client, only the service self-check and the bench_serving reference sweep.

Checklist:

  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

No tests. This is a documentation/example change: shell scripts that drive a two-node GPU
deployment, with no unit-testable surface and no CI cluster to run them against. What was done
instead is recorded below.

Validation status

Stated plainly rather than implied — the README carries the same table.

what status
the deployment shape (1P1D + mooncake + DPA + MTP + kvd + kv-aware) validated end-to-end on two clusters, both fabric types
the tuned values (GMU, chunk, ctx, EAGLE settings, DSA env, router weights) validated — each carried over from a run that completed cleanly
these scripts as written validatedpreflight modeupsmokebenchdown on a 2-node MI355X mode-B cluster, no edits outside the wrapper. Long context checked separately (needle, to 238K tokens) and under a real agentic workload at concurrency 8
preflight_rdma.sh fabric not validated — only mode was exercised
cluster.peermem.sh, round-robin routing not validated — no peer-mem cluster was available; the shipped kv-aware default is what ran

Static checks: bash -n on all 9 scripts; every infera flag grepped against infera/**/args.py
on main; the deliverable grepped for leaked local information (host names, absolute paths,
scheduler identifiers, credentials) before submission.

Known placeholder

The image tag inferaimage/infera-sglang:0.2.0 is a placeholder, flagged as such in the
README. It needs replacing with the released infera-sglang tag before this is useful to a
reader.

@dorado269
dorado269 force-pushed the yihou.dev.glm52.example branch from 1413784 to d7c8223 Compare August 4, 2026 08:21
@dorado269 dorado269 changed the title init glm5.2 example docs(examples): add a GLM-5.2 1P1D SGLang deployment example Aug 4, 2026
@dorado269
dorado269 force-pushed the yihou.dev.glm52.example branch from d7c8223 to ded7954 Compare August 4, 2026 08:27
@dorado269
dorado269 marked this pull request as ready for review August 4, 2026 08:27
@dorado269
dorado269 force-pushed the yihou.dev.glm52.example branch 3 times, most recently from 65653a5 to eb7951d Compare August 4, 2026 09:43
| preflight says | wrapper | why |
|---|---|---|
| `peermem: present`, **mode A viable** | [`cluster.peermem.sh`](cluster.peermem.sh) | bare `ibv_reg_mr` hands the NIC the GPU pages directly — nothing pinned, KV pool not duplicated, **every** rail can carry KV |
| `peermem: absent`, **mode B viable** (an ODP NIC exists) | [`cluster.dmabuf.sh`](cluster.dmabuf.sh) | dma-buf is the only GPUDirect path without peer-mem, and it is only safe on an ODP NIC; KV is locked to that one NIC |

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verified

@dorado269
dorado269 force-pushed the yihou.dev.glm52.example branch 2 times, most recently from 114dc96 to 32f02e5 Compare August 4, 2026 09:54
A runnable kit for GLM-5.2-MXFP4 on two MI355X nodes: one prefill leg and one
decode leg, KV moved between them over Mooncake RDMA, fronted by the infera
router, with DP-attention, MTP (EAGLE) and the kvd cache tiers on.

Everything site-specific lives in one of two files under cluster/ -- one per
RDMA registration mode, picked by the preflight probe rather than by
autodetection at launch. common.sh and engine/*.sh carry the tuned recipe and
contain no addresses, paths, NIC names or GID indices.

Three settings in engine/leg.sh are coupled in ways that produce a plausible
result rather than an error when got wrong, so each is commented in place and
written up in the README:

  * --ep-size is emitted unconditionally, outside the DP-attention branch.
    Expert and attention parallelism are different axes; gating both on one
    condition collapses the MoE whenever DP-attention is off.
  * --chunked-prefill-size is a global budget that SGLang divides by dp_size
    only when DP-attention is on. One value serves both modes; a per-rank
    number hardcoded in a DPA-off branch cuts the global budget 8x.
  * Prefill activation OOM is fixed by LOWERING --mem-fraction-static, the
    opposite of the decode-side retract fix. Diagnose by phase.

The host RDMA provider library, if the image injects one, needs HOST_RDMA_LIB,
HOST_RDMA_MOUNT and ENTRYPOINT_KEEP together: the mount path must be the one
that image's entrypoint reads. Set only some of them and the library lands
where nothing reads it, which is not an error -- the leg boots and serves,
the container sees ZERO RDMA devices, and mooncake moves KV 5-20x slower.
start_container now warns on a zero device count rather than printing it as a
number to skim past, and smoke.sh reports the count alongside its MC_FORCE_TCP
and GID-is-NULL counters, which are both 0 in that state too.

Three checks in smoke.sh were reading nothing and printing a pass:
  * engine logs carry binary bytes, so a bare grep answers "binary file
    matches" and every counter reads 0. Filter through strings.
  * the Rust router prints `router_policy: "kv-aware"` inside a colourised
    Config{...} dump, never `router-policy=<x>`; the old pattern printed empty
    whatever the policy was. It now also reports whether the tokenizer loaded,
    without which kv-aware silently degrades to load-only routing.
  * MTP acceptance was read off `tail -5`, and a few percent of batches sit at
    4.00 even on a healthy leg (measured 4.7% at median 2.88), so that tail
    routinely reads as a repetition loop. Report the distribution and the
    median.

results/ carries concurrency-8 numbers from two independent agentic benchmarks
on two clusters with different fabrics, recomputed from raw per-request records.
The cross-cluster gap is presented as context rather than as a measurement:
more than one variable moves between those clusters, four candidate causes are
listed, and none is confirmed. The customer's benchmark is referenced by URL and
not vendored; this kit ships no agentic bench client, only a service self-check
and a reference sweep using SGLang's own bench_serving.

The AgentX Case-A harness is closed-loop, not open-loop: --concurrency N runs
N workers and each waits for its response before issuing the next. In-flight
sat at mean 5.13 against a cap of 8, which pinned lanes could not produce. The
difference from the infera bench is the session population (fixed vs growing),
not the control loop. Its corpus also does not meet the workload spec's ~88-89%
prefix-hit target: met per turn (median 88.0%) but not overall (mean 66.0%,
token-weighted 59.7%), for three reasons stated with the generator's own
construction -- turn 0 is all-miss and inside the target, the draw is per-turn
and unweighted by tokens, and 51% of turns are shorter than their predecessor,
which permanently truncates the accumulated prefix. No claim is made about a
usable concurrency limit; the c8/c16 shape change is kept as an observation.

The engine image is stated as an infera-sglang build newer than 0.2.0, carried
as the literal placeholder <infera-sglang-image>. The README states plainly
which parts are validated end-to-end and which are not: the peer-mem wrapper
and round-robin routing were never exercised, as no such cluster was available.

Signed-off-by: yihou <yihou@amd.com>
@dorado269
dorado269 force-pushed the yihou.dev.glm52.example branch from 32f02e5 to 7609bb1 Compare August 4, 2026 09:54
@xiaobochen-amd
xiaobochen-amd merged commit 3e443c7 into main Aug 4, 2026
24 checks passed
llying-001 added a commit that referenced this pull request Aug 4, 2026
Not this PR's code. examples/sglang_1p1d_glm5.2/README.md arrived on main with
33 trailing spaces on one line of its topology diagram (#84, 7609bb1), and CI's
lint job runs `pre-commit run --all-files`, so the trailing-whitespace hook
fails for any branch rebased onto current main -- and `unit` needs `lint`, which
blocks everything behind it. A pristine origin/main worktree reproduces it with
none of this branch's changes present.

Whitespace only: the rendered line is byte-identical, and those spaces padded
nothing because the diagram has no right border. It is the only such line in
the repo. Kept as its own commit so it can be dropped if the fix lands on main
separately.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: liyingli <liyingli@amd.com>
jiejingzhangamd added a commit that referenced this pull request Aug 4, 2026
Not from this branch. examples/sglang_1p1d_glm5.2/README.md arrived on
main in 7609bb1 (#84) with a trailing-whitespace line, which the
pre-commit hook rejects -- so the lint job fails on the merge result for
every open PR, including this one, while each branch passes on its own.

Fixed here only because it blocks this PR's CI. One line, whitespace
only, no content change.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants