feat(recipes): per-model Kubernetes recipes for GLM-5.2 and Kimi-K3, on stock vendor images by jiejingzhangamd · Pull Request #64 · AMD-AGI/Infera

jiejingzhangamd · 2026-07-31T22:02:46Z

Adds examples/recipes/ — a recipe page per model in the shape people actually consume one: pick a combo, check prerequisites, kubectl apply, smoke test, link to source.

examples/recipes/<model>/<combo>/deploy.yaml
examples/recipes/<model>/README.md

Four combos per model — mixed, mixed+kvd, pd, pd+kvd — so the two independent choices (how requests split across GPUs; whether KV survives past the GPU) are selected rather than hand-assembled from a wiki page.

Every manifest is stock base + overlay + sidecar

The vendor image is never forked. An initContainer drops the overlay payload into an emptyDir, and infera-exec runs infera out of it:

initContainer  infera-overlay   busybox carrying /payload  ──cp──▶ emptyDir
container      main             STOCK vllm/sglang image, runs /overlay/bin/infera-exec
container      kvd              same vendor image, also runs from /overlay

Following an upstream release becomes an image-tag edit. That is the point — forking the vLLM base for Kimi-K3 is what broke serving for every other model in CI (#55).

Validated end-to-end, and that is where most of this PR came from

Both mixed recipes were run as written, with no infera-built engine image anywhere in the Pod:

	Kimi-K3 (vLLM)	GLM-5.2 (SGLang)
base	stock `vllm/vllm-openai-rocm:kimi-k3`	stock `lmsysorg/sglang`
weight load	502 s, 1453 GiB, local NVMe	408 GiB over NFS
worker Ready	~11 min	~13 min
output	`The capital of France is Paris.`	`The capital of France is Paris.`
extra	multimodal identified the test image	reasoning split into `reasoning_content`

Running them is what surfaced the four fixes below. Each page carries a Validation status table separating what has run on hardware from what has only been composed; pd/pd-kvd are structural and need two nodes on a routable RoCE fabric.

Fixes this turned up

The overlay was shadowing the vendor's Python stack. The engine logged ImportError: Numba needs NumPy 2.4 or less. Got NumPy 2.5 on repeat. pip install --target ignores the environment and installs the full closure, and a --target tree lands first on PYTHONPATH. Of 52 distributions, two were new; eleven shadowed the base at a different version (numpy 2.5.1 over 2.3.5, cryptography 50.0.0 over 3.4.8). prune_base_dists.py now builds each tree inside the base it will be laid onto and drops what the base already has — 325 MB → 5.8 MB.

The overlay shipped no native tree for SGLang. It harvested only from a vLLM image, so native/rocm7-py312 existed and nothing else; PD on SGLang could not work at all. Now harvested once per ABI family.

The native guard asked the wrong question. INFERA_REQUIRE_NATIVE=1 meant "the tree must exist", but the families deliberately differ — hipFile is vLLM-only, because GPU-direct L3 is a vLLM-only path. As written it would have rejected two working configurations: SGLang PD, whose tree correctly has no hipFile, and SGLang mixed-kvd, which needs no native code at all. A guard that blocks working deployments is worse than the silent downgrade it was added to prevent. Each tree now records a CAPABILITIES file and manifests name what they need.

GLM-5.2 had no reasoning parser. The first run answered correctly but inlined the whole chain-of-thought and a raw </think> into content. Added --reasoning-parser glm45 and re-ran rather than shipping the flag untested.

Release CI

deploy/overlay/Dockerfile.payload was only ever built by hand, so the overlay these recipes depend on was in no release. It now builds as inferaimage/infera:overlay-<id>, in its own job after the engines — it is the one image here that is not self-contained, since it builds its Python trees inside the stock bases and harvests native code from the engine images. Both classes of ref are derived, not restated, because either drifting fails silently.

Verification

All eight manifests pass kubectl apply --dry-run=server against the live CRD
The capability guard tested in both directions on real images: sglang starts with no requirement and with mooncake, fails on hipfile saying it is vLLM-only; vllm starts with mooncake,hipfile. That last case failed the first time — CAPABILITIES was written one word per line and the match needs space delimiters, so a complete tree reported itself incomplete
Harvest output confirms the split: rocm7-py312 [mooncake hipfile], rocm7-py310 [mooncake]
Not verified: the release job has not been dispatched, so the SLURM path is unexercised

Also

.claude/CLAUDE.md records the DCO requirement and is now tracked (.claude/* plus a negation, since git will not descend into an excluded directory). The sign-off example uses a placeholder — main carries five distinct signer identities, and copying someone else's line inverts what the DCO asserts.

Depends on

feat(k8s): Kubernetes recipes — single-node (mixed) + PD 1P1D (Mooncake) #55 — model-cache/, manual/examples/ pages, examples/kimi_k3/mm_test.py
feat(deploy): overlay payload — decouple everything we ship from the vendor base #61 — deploy/overlay/

🤖 Generated with Claude Code

https://claude.ai/code/session_01Pw7JSvdQb796xh5pEcctLz

vllm_serve.sh: serve moonshotai/Kimi-K3 on the vllm/vllm-openai-rocm:kimi-k3 image at TP8 per the AMD Kimi-K3 guide (low-bit MXFP4/A8W4 done at runtime by aiter via AITER_SITUV2_A8W4=1 — native checkpoint, no separate quantized download). Image ENTRYPOINT is /bin/bash, so it serves via `vllm serve <model> <flags>`; points at the cached checkpoint on the shared mount with HF_HUB_OFFLINE=1. README notes the vLLM + SGLang base images. SGLang launcher TBD. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Only /mnt/vast was bind-mounted, so a MODEL under /mnt/k3local (local NVMe) was invisible inside the container and HF treated the path as a repo id (HFValidationError). Derive the top-level mount from MODEL and add it. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Concrete, validated InferaDeployment recipes for the operator, the model-cache (PVC + download Job, the Dynamo model-cache analog), and the libionic fix the SGLang engine images needed for RDMA. - examples/k8s-deployments/: single-node-qwen-{sglang,vllm}.yaml, model-cache/, kimi-k3-vllm.yaml, pd-1p1d-mooncake.yaml, RECIPE-single-node.md - manual/examples/: k8s_kimi_k3.md, k8s_pd_1p1d_mooncake.md (+ _toc.yml.in) - Dockerfile.sglang{,.gfx942}: bake ABI-4 libionic (INSTALL_LIBIONIC=1) so ibv_get_device_list sees the ionic HCAs and cross-node PD KV transfer over Mooncake does not silently degrade to TCP. Validated on single/two-node k3s (MI355X): operator reconcile, sglang+vllm mixed serving to real tokens, Kimi-K3 TP8 multimodal end to end, model-cache Job -> PVC -> serve, and PD dual-dispatch up to the cross-node RoCE-fabric boundary (KV transfer needs the two nodes on a routable RoCE fabric). Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

… fixes - model-cache: remove Dynamo/vendor mentions; default PVC 20Gi -> 2000Gi so it's safe for Kimi-K3 (~1.5 TB), noted small models need only ~20Gi. - pd-1p1d-mooncake: --mem-fraction-static 0.6 -> 0.8 (each worker owns a full GPU); DROP --disaggregation-ib-device so Mooncake auto-discovers every ionic HCA (pinning ionic_0 limited KV transfer to one rail); drop the now-unneeded NCCL_IB_HCA. Doc: Mooncake auto-discovers vs mori lists all NICs; node-token path follows the server --data-dir. - vLLM image -> rocm/infera:vllm-v0.1.1 (and rocm/infera:vllm-kimi-k3) across the recipes. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…ors N/A Replace skipReadinessProbe with a generous startupProbe (/health, failureThreshold 120 x 10s) so the pod surfaces a real Ready signal after the ~9 min weight load + graph capture (operator readiness gated by it). Validated on k3s (MI355X): worker Ready at 556s, multimodal image request via the router returns the correct colour. Also document why two upstream/Dynamo tunings do NOT apply to Kimi-K3 on AMD: --enable-prefix-caching forces the experimental Mamba "align" cache (hybrid model) and fails engine init; --load-format fastsafetensors needs GPU Direct Storage (cufile) and otherwise stalls with ~30s per-batch queue waits. Neither is in the upstream vLLM ROCm command. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

- image: engine image -> python:3.11-slim (engine-agnostic, tiny vs 78 GB) - securityContext: allowPrivilegeEscalation=false, drop ALL caps, RuntimeDefault seccomp (the download needs no privileges) - HF_XET_HIGH_PERFORMANCE=1 + huggingface_hub[hf_xet] to accelerate large Xet-backed repos (Kimi-K3); harmless for plain LFS repos Keeps snapshot_download(local_dir=...) for a clean /models/<name> mount path. Validated on k3s: Job Complete in 13s, Qwen3-0.6B populated into the PVC. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

… reality Kimi-K3 is plain LFS (0 Xet blobs), so HF_XET_HIGH_PERFORMANCE is a no-op for it; the LFS accelerator is hf_transfer. Add HF_HUB_ENABLE_HF_TRANSFER=1 + huggingface_hub[hf_transfer]. Measured on this cluster both accelerators are no-ops — the download is already network-bound (~600 MB/s; hf_transfer 545 vs standard 611 MB/s on a 4 GB LFS shard). They help only on a bandwidth/latency- constrained link where a single stream underutilizes the pipe. Comment says so. Validated: Job Complete in 16s with both extras installed and both envs set. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Measured no benefit on this cluster (download is network-bound ~600 MB/s; HF_XET is a no-op for LFS Kimi-K3, hf_transfer 545 vs standard 611 MB/s). Keep the worthwhile changes — python:3.11-slim (light, engine-agnostic) + hardened securityContext — and plain huggingface_hub, no accelerator envs/extras. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

GLM-5.2-MXFP4 (GlmMoeDsaForCausalLM — MLA + DeepSeek Sparse Attention) on one 8-GPU node via the operator + router + a TP8 infera.engine.sglang worker. SGLang is the correct/fastest GLM-5.2 engine on ROCm; vLLM loads and serves but its MLA/DSA decode is numerically buggy (garbage "!!!!" output) — documented. Bakes the REQUIRED ROCm DSA env (SGLANG_OPT_USE_TILELANG_INDEXER=1 / USE_TOPK_V2=0 / USE_JIT_NORM=0 + aiter) so the CUDA-only DSA top-k JIT kernel does not crash init on gfx950, plus --nsa-{prefill,decode}-backend tilelang and fp8 KV. startupProbe for the ~408 GiB load. Validated on k3s (MI355X): worker Ready ~6 min, coherent /v1/chat/completions via the router (correct "capital of France is Paris"); the vLLM image on the same request returns garbage. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…-v0.1.2 Move Dockerfile.vllm's base from v0.25.1 to the vLLM kimi-k3 build (digest-pinned) so the canonical infera vLLM image carries kimi_k3 model support (KimiK3ForConditionalGeneration) out of the box, with broader model coverage. Validated the FULL build on this base on MI355X: aiter + vLLM/DSv4 patches + Mooncake + hipFile + libionic + infera + rust router all build; the image has infera.server/infera.engine.vllm, KimiK3ForConditionalGeneration, ABI-4 libionic, the router binary and ais-check. Tagged rocm/infera:vllm-v0.1.2. Point the vLLM recipes at it: single-node-qwen-vllm.yaml and kimi-k3-vllm.yaml now use rocm/infera:vllm-v0.1.2 (kimi-k3-vllm no longer needs the special --build-arg base override — the standard build produces a kimi_k3-capable image). Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

The kimi-k3 vLLM build does not run the other models: e2e-mixed (vllm) failed on this PR with GLM-5.1-FP8-tp4 and test_mixed_kvd, while the Qwen case passed. Every other e2e tier was green, so the base swap is the cause. Restores Dockerfile.vllm's base to the v0.25.1 digest (byte-identical to main) and points single-node-qwen-vllm.yaml back at rocm/infera:vllm-v0.1.1. Kimi-K3 keeps its recipe, but through a self-contained image built with a --build-arg base override rather than by changing the canonical default, so it cannot affect any other model. Deeper Kimi-K3 enablement gets its own branch once this merges. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…k base Today every engine image bakes infera onto a digest-pinned vendor base, so following an upstream bump means rebuilding everything, and forking the base for one model (Kimi-K3) broke the others. Most of what we add is not actually tied to the base, so it does not have to be baked in at all. Adds a single `infera-overlay` image carrying the base-agnostic half — infera and kvd (pure Python), the Rust router (links only libc/libstdc++), and an `infera-exec` launcher — mounted over an UNMODIFIED vendor image via a volume (an initContainer + emptyDir in Kubernetes). Four dependencies are C extensions (msgspec, xxhash, pyzmq, msgpack) so the Python tree is per-CPython-minor, and the vendor bases have split: vLLM ships 3.12, SGLang 3.10. Rather than publishing two payloads, the image carries both trees and infera-exec selects by the container's Python. Mooncake and hipFile stay in the engine image: they bind the CPython minor AND the ROCm major, so they genuinely have to be compiled against their base. Verified on MI355X against unmodified vendor images: - vllm/vllm-openai-rocm:kimi-k3 (py312): engine ready in 70s, serving /v1/chat/completions; infera resolves to /payload/py312/infera - rocm/infera:sglang-v0.1.1 (py310): infera resolves to /payload/py310/infera, C-extension deps import, engine.sglang and kvd both present - infera-router runs from the same payload on both Handles one non-obvious trap: Python puts the working directory at sys.path[0], ahead of PYTHONPATH, so on a base that already ships infera under its WORKDIR the baked-in copy silently shadows the payload. infera-exec cd's to / first. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…ndor base Extends the overlay to the compiled artifacts, which turns out to be possible: Mooncake and hipFile do not bind the specific vendor image, only the CPython minor and the ROCm major. Ship one native tree per (rocm, python) pair with the extension's ~40 transitive system libraries bundled alongside — the same thing auditwheel does for a manylinux wheel — and they load on a stock base. /payload/py310, py312 pure-Python trees (per CPython minor) /payload/native/rocm7-py312/ mooncake + hipFile + their libs /payload/bin/ infera-router, infera-exec infera-exec now keys the native tree off the container's own libamdhip64 soname, so a vendor ROCm bump selects the right tree instead of loading a mismatched one. A missing native tree warns rather than fails, since aggregated serving needs none of it; INFERA_REQUIRE_NATIVE=1 makes it an error for PD deployments. The payload is now rebuilt only when the ROCm major or the CPython minor moves — not when the vendor image does. Verified on MI355X against an unmodified vllm/vllm-openai-rocm:kimi-k3: infera -> /payload/py312/infera (kvd + engine.vllm present) router -> runs mooncake -> TransferEngine OK (PD available) hipFile -> libhipfile.so loaded; ais-check reports HIP runtime + amdgpu True (kvd GPU-direct L3 available) Native artifacts are harvested from an already-built engine image rather than recompiled, so the payload build stays fast and reuses validated binaries. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Two gaps found when auditing the recipes against what was really run. GLM-5.2 had no 'what is validated' block, unlike the other three recipes, so the evidence for it lived only in the PR discussion: TP8 on single-node k3s (MI355X), worker Ready in ~6 min, a correct and coherent completion through the router — and the same request on the vLLM image returning '1!!!!!...', which is the whole reason that recipe is SGLang-only. The Kimi-K3 manifest header still said the structure was validated but told the reader not to expect it to serve. It does serve: TP8 Ready in ~9 min and a multimodal image request through the router returns a correct colour-grounded answer. Replaced the stale caveat with what was measured. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

… way Adds the second of the four recipe shapes (mixed, mixed+kvd, pd, pd+kvd): the aggregated topology with the kvd tiered KV cache, using cluster resources rather than host paths. * kvd runs as a SIDECAR in the worker Pod. The operator only owns the container named `main` (mainContainerNames in builders.go), so any extra container passes through verbatim — no CRD change needed. * the engine <-> kvd Unix socket is an `emptyDir` shared by the two containers: no hostPath, nothing pinned to a node. * kvd's L3 is a **PVC**, so capacity is requested from the cluster and the scheduler places the Pod where it can be satisfied. * the L2 arena is pinned host RAM, charged to the sidecar's memory limit, and needs IPC_LOCK to mlock. Verified on single-node k3s (MI355X): worker reaches 2/2 Ready in ~200 s, and a 3650-token request through the router lands in kvd — entries=3674 sets_total=3674 host_bytes=421363712 long_bytes=421363712 so both the RAM tier and the PVC-backed L3 receive it. kvd's own self-check on the PVC reported WRITE 3.72 GB/s / READ 51.27 GB/s. Two things this recipe has to get right that bare-metal runs never surface: - `--infera-kvd-socket` enables SGLang's hierarchical cache, whose host-RAM tier is sized from the GPU KV pool. Uncapped it asked for 416 GB and the container was OOMKilled; `--hicache-size` must fit the engine container's memory limit. Documented in the manifest. - kvd takes `--long-path`/`--long-bytes` for the file tier; there is no `--spillover-path`. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Fourth of mixed / mixed+kvd / pd / pd+kvd. PD-disaggregated 1P1D over Mooncake with the kvd tiered cache layered on BOTH roles: prefill and decode each get their own kvd sidecar, their own emptyDir socket, and their own L3 PVC. Per-role rather than one shared kvd, deliberately: the roles hold different KV (prefill produces reusable prefix KV, decode holds the transferred working set), they sit on different nodes, and a ReadWriteOnce claim shared between them would pin both to one node and undo the disaggregation. Verified on the two-node k3s (MI355X): prefill 2/2 Running on chi2800, decode 2/2 Running on chi2866, both kvd sidecars listening on their emptyDir socket with PVC-backed L3 (self-check WRITE 2.2-2.8 GB/s, READ ~50 GB/s), and prefill registering through the k8s Pod annotation. The Mooncake KV transfer itself still needs the two nodes on a mutually routable RoCE fabric, as documented in pd-1p1d-mooncake.yaml — that limit is unchanged here. Documents a trap this shape is uniquely exposed to: with a node-local StorageClass each PV is bound to the node that first consumed it, and that binding outlives the Pod. Moving a role to another node afterwards leaves it Pending forever with 0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) didn't match Pod's node affinity which is exactly what happened here when decode moved from chi2800 to chi2866. Recreate the PVC when relocating a role, or use a non-node-local StorageClass. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

kvd reaches the two engines by different routes and they are not equivalent: SGLang --infera-kvd-socket -> HiCacheStorage backend: L2 host-RAM arena and the L3 file tier vLLM InferaKvdConnector -> the same tiers PLUS hipFile GPU-direct L3, which loads L3 straight into VRAM instead of bouncing through host memory Both work — mixed-kvd.yaml (SGLang) was measured at 3674 entries / 421 MB — but only the vLLM path gets GPU-direct L3, and the SGLang manifest now says so rather than leaving readers to assume the two are interchangeable. Adds mixed-kvd-vllm.yaml: same sidecar + emptyDir socket + PVC-backed L3 layout, wired through --kv-transfer-config instead. It also drops the --hicache-size cap, which is an SGLang-side knob with no vLLM equivalent. Verified on single-node k3s (MI355X): worker 2/2 Ready, a 3510-token request lands in kvd — entries=2 sets_total=2 host_bytes=271589376 long_bytes=271589376 on both the RAM tier and the PVC. The run also confirmed the manifest's own KV-dtype note is accurate: on an engine image predating #63 the default fp8 KV is refused by the offload guard (two "NOT a plain MLA fp8" rejections, zero writes), and adding `--kv-cache-dtype auto` as the note prescribes makes it offload. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Adds examples/recipes/, a recipe page per model in the shape people actually consume one: pick a combo, check prerequisites, kubectl apply, smoke test. examples/recipes/<model>/<combo>/deploy.yaml examples/recipes/<model>/README.md Four combos per model — mixed, mixed+kvd, pd, pd+kvd — so the two independent choices (how requests split across GPUs; whether KV survives past the GPU) are selected rather than hand-assembled. Every manifest is stock vendor image + overlay + sidecar. The vendor image is never forked: an initContainer drops the overlay payload into an emptyDir and `infera-exec` runs infera out of it. Following an upstream vLLM/SGLang release becomes an image-tag edit, which is the point — forking the vLLM base for Kimi-K3 is what broke serving for every other model in CI. Details worth calling out, each of which was a real failure first: * kvd runs from the SAME vendor image as the engine, not from the overlay image. The overlay is a busybox that only carries the payload; it has no interpreter to run kvd with. * resource names are glm52-*, not glm5.2-*. A Service name is a DNS-1123 label, which does not admit dots, so `glm5.2-mixed-server` is unschedulable. * SGLang gets --served-model-name too, or the model id in the smoke test is the model path and the request 404s. * vLLM pd+kvd wraps MooncakeConnector and InferaKvdConnector in a MultiConnector — vLLM takes one --kv-transfer-config, and PD plus kvd both want it. Verified the JSON round-trips into the shape infera/engine/vllm/ args.py expects. Each page carries a Validation status table separating what has been run on hardware from what has only been composed, per model and per combo. All eight manifests pass `kubectl apply --dry-run=server` against the live CRD. Depends on #55 (model-cache, manual/examples pages) and #61 (deploy/overlay). Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

The overlay's native/ tree carries Mooncake (PD KV transport) and hipFile (kvd GPU-direct L3). infera-exec treats it as optional — correctly, since aggregated serving needs none of it — and only warns when it is absent. That warning is easy to miss in engine startup logs, and the failure mode is the bad kind: the deploy is green, the model serves, and the tier you built the manifest for is quietly not there. Hit exactly this while validating: the locally built infera-overlay:latest predated the native harvest, so it had no native/ tree at all. So the kvd and pd manifests now set INFERA_REQUIRE_NATIVE=1 on both the engine and the kvd sidecar, turning a silent downgrade into a startup failure. The mixed manifests deliberately do not — they genuinely do not need it. Documents the build arg that produces a native-carrying payload. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Found while running the Kimi-K3 recipe on a stock vLLM base: the engine logged `ImportError: Numba needs NumPy 2.4 or less. Got NumPy 2.5` on repeat. The base ships numpy 2.3.5; the overlay was shipping 2.5.1 and winning. The cause is that `pip install --target` ignores the environment. It installs the full transitive closure regardless of what the base already has, and a --target tree lands FIRST on PYTHONPATH — so every duplicate shadows the vendor's copy at whatever version PyPI happened to resolve. The baked engine images never had this problem: a plain `pip install` onto the base leaves satisfied requirements alone and pulls only what is missing. Moving to --target silently dropped that. Measured against vllm/vllm-openai-rocm: of 52 distributions in the tree, exactly TWO were new (nats-py, grpcio-health-checking). Eleven shadowed the base at a different version: numpy 2.5.1 over 2.3.5 <- the numba breakage cryptography 50.0.0 over 3.4.8 <- 47 major versions protobuf 7.35.1 over 6.33.6 grpcio 1.83.0 over 1.78.0 fastapi, fsspec, huggingface_hub, filelock, tqdm, uvicorn, websockets Two fixes that did NOT work, for the record. Relying on pip to skip installed packages: --target ignores them, verified by --dry-run --report. Constraining to the base's `pip freeze`: ResolutionImpossible. What works is to build each tree inside the vendor base it will be overlaid onto, then delete whatever the base already provides. prune_base_dists.py walks each dist-info's RECORD and removes base-provided distributions, refusing to exit 0 if infera itself went missing. Verified in vllm/vllm-openai-rocm:kimi-k3 — tree 325 MB -> 5.8 MB (49 pruned, 3 kept), infera resolves from the payload, numpy resolves to the base's 2.3.5, and numba imports cleanly. Also makes the payload honest about what it is: an overlay that adds infera, not a second opinion on the vendor's dependency pins. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…rlay Ran examples/recipes/kimi-k3/mixed/deploy.yaml as written on k3s (MI355X, 8 GPUs). No infera-built engine image in the Pod at all — stock vllm/vllm-openai-rocm:kimi-k3 with the overlay mounted in, which is the whole claim the recipe makes. router Ready ~20 s (infera.server running from /overlay) weight load 502 s 96 shards, 1453 GiB from local XFS NVMe worker Ready ~11 min including aiter JIT + CUDA graph capture text "The capital of France is Paris." (73 tokens, stop) multimodal correctly identified the generated crimson image Two things the run corrected in this page. The smoke test asked for max_tokens 60. Kimi-K3 spends 73 completion tokens on that one sentence, so the response truncated to the single word "The" and the recipe read as broken when it was working perfectly. Raised to 200 and the expected output is now stated, so the next person can tell success from truncation without reading the token count. And the engine logged `ImportError: Numba needs NumPy 2.4 or less. Got NumPy 2.5` on repeat, which led to the payload-shadowing fix now in deploy/overlay/prune_base_dists.py. The Validation status table now separates the combo that has actually run from the ones that have only been composed, rather than hedging the whole page. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Running examples/recipes/glm5.2/mixed/deploy.yaml on the stock SGLang base, the engine's first line was: infera-exec: no native payload for rocm7-py310; PD (Mooncake) and kvd GPU-direct L3 (hipFile) will be unavailable. The overlay harvests its native tree from NATIVE_IMAGE, which defaults to a vLLM image — CPython 3.12. SGLang bases are 3.10, so a default build has no native tree for them at all, and GLM-5.2's mixed-kvd / pd / pd-kvd combos cannot work until the overlay is built with an SGLang-based NATIVE_IMAGE. Worth noting that this is the guard added two commits ago earning its keep: with INFERA_REQUIRE_NATIVE=1 those combos fail at startup instead of coming up green and serving with no kvd L3 and no KV transport. mixed does not set it, does not need the native tree, and ran fine. Documents the constraint on the GLM-5.2 page rather than leaving it as a warning line in the engine log. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…ning parser Ran examples/recipes/glm5.2/mixed/deploy.yaml as written on k3s (MI355X, 8 GPUs) against the stock lmsysorg/sglang image with the overlay mounted in. Output was correct and coherent, so the ROCm DSA env wiring survives the move off the baked image. router Ready ~20 s (infera.server from the overlay's py310 tree) worker Ready ~13 min (408 GiB over NFS + DSA graph capture) output "The capital of France is Paris." That first run also showed the answer arriving with the full chain-of-thought and a raw </think> inlined in `content`: GLM-5.2 is a thinking model and the manifest had no reasoning parser. Right output, unusable shape. Added --reasoning-parser glm45 and re-ran rather than shipping the flag on the strength of it being the obviously correct one — putting an unverified flag in a recipe is the specific failure these pages exist to prevent. Confirmed on the second run: content is exactly "The capital of France is Paris.", the trace is in reasoning_content, and no </think> leaks. With this, the pruned overlay is validated on both ABI families it ships — py312 via Kimi-K3/vLLM and py310 via GLM-5.2/SGLang. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…ilities Two defects, one of them mine from two commits ago. 1. The build harvested native code (Mooncake, hipFile) only from NATIVE_IMAGE, a vLLM image, so the payload shipped native/rocm7-py312 and nothing else. SGLang bases are CPython 3.10, and Mooncake binds the CPython minor, so PD on SGLang could not work no matter how the manifest was written. Observed running the GLM-5.2 recipe: infera-exec: no native payload for rocm7-py310 Fixed by harvesting once per ABI family: SGLANG_NATIVE_IMAGE supplies the 3.10 tree. The harvest is now a script both stages share rather than a duplicated RUN block. 2. INFERA_REQUIRE_NATIVE was a boolean meaning "the tree must exist", which is the wrong question, because the families deliberately differ: hipFile is vLLM-only, since kvd's GPU-direct L3 is a vLLM-only path. As written the guard would have rejected two configurations that work — SGLang PD, whose tree correctly has no hipFile, and SGLang mixed-kvd, which reaches its L2 and file L3 through pure-Python HiCacheStorage and needs no native code at all. A guard that blocks working deployments is worse than the silent downgrade it was added to prevent. Now each harvest records a CAPABILITIES file, INFERA_REQUIRE_NATIVE names the capabilities a deployment actually needs ("mooncake", "hipfile", or both), and infera-exec checks those. 1/true/all still mean "everything this family has". Manifests updated to declare only what they need: SGLang mixed-kvd nothing, SGLang pd mooncake, vLLM mixed-kvd hipfile, vLLM pd-kvd both. Verified against real images, both directions: sglang + no requirement starts <- the case the old guard broke sglang + mooncake starts sglang + hipfile fails, and says hipfile is vLLM-only vllm + mooncake,hipfile starts vllm + "1" starts That fourth case failed the first time: CAPABILITIES was written one word per line and the word-match needs space delimiters, so a complete tree reported itself as incomplete. Normalised on both sides. Harvest output confirms the split — rocm7-py312 [mooncake hipfile], rocm7-py310 [mooncake]. All eight manifests still pass --dry-run=server. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

…r-checkout .claude/ was ignored wholesale, so the DCO rules written there bound only this checkout — the next person's agent session, and every reviewer, saw nothing. That is backwards for a file whose entire purpose is stating repository conventions. Un-ignored by file rather than by removing the directory entry. .claude/ also holds settings.local.json, which is per-developer, and worktrees/, which is 5.2 MB of whole nested git checkouts; neither belongs in the repo. The exclusion is now `.claude/*` rather than `.claude/`, because git does not descend into an excluded directory — with the directory form, `!.claude/CLAUDE.md` would never be reached and the negation would silently do nothing. Verified: CLAUDE.md is tracked, settings.local.json and worktrees/ are still ignored. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

The sign-off example named a specific person and address. Anyone following it literally would sign off as someone else, which inverts what the DCO says: the trailer is an assertion that the *signer* has the right to submit the code, so it has to match the commit's actual author. The last 400 commits on main carry five distinct sign-off identities, so this is a real hazard here, not a hypothetical. Uses a placeholder instead, and adds the check worth running before the first commit in a fresh clone or container, where git may have inherited a default user.email from the environment and `-s` will quietly stamp the wrong person. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

deploy/overlay/Dockerfile.payload was only ever built by hand, so the overlay that recipes depend on was not part of any release. Adds it to the release workflow as inferaimage/infera:overlay-<id>. It cannot simply become a sixth entry in the engine matrix. The overlay is the one image here that is not self-contained: * its Python trees are built inside the STOCK vendor bases so they can be pruned to what those bases lack, and * its native trees are harvested out of the vllm and sglang images. So it needs four image refs and it must run after those two engines. It gets its own job, gated on `build`. Both classes of ref are derived rather than restated, because either one drifting fails silently rather than loudly: * NATIVE_IMAGE / SGLANG_NATIVE_IMAGE are computed with the same id logic as `build`, so a release harvests the engines it just pushed instead of whatever is latest — otherwise a release can ship an overlay carrying the previous release's Mooncake. * VLLM_BASE_IMAGE / SGLANG_BASE_IMAGE are read out of the engine Dockerfiles. The prune keeps exactly what the base lacks, so a base ref that drifts from the one the engine image was built on yields a payload wrong in both directions: missing packages the real base does not have, and shadowing ones it does. A partial dispatch (say `engines: sglang`) skips the overlay rather than building one from stale native code; `prepare` decides this and reports why. The Release gate now covers the overlay too, but distinguishes skipped from failed: a skipped overlay must not block publishing, which a plain `needs:` would do, so `manual` checks job results explicitly. Empty-array expansion uses ${a[@]+"${a[@]}"} — the plain form errors under `set -u` on bash <= 4.3, and this script cannot assume the compute nodes' bash. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

… when asked for the impossible The overlay job added in 11939be is correct for a tag push and for the default dispatch, but the `engines` dispatch input was still a free-text box that accepts "overlay". Typing it routed the overlay into the engine matrix, where it would build in parallel with — or entirely without — the images it harvests from, so it would pick up whatever the registry held from a previous run. Meanwhile the gated overlay job saw no vllm in the list and skipped. Exactly the failure the separate job exists to prevent, reachable from the UI. `overlay` is now stripped from the matrix unconditionally: it is not an engine. An explicit request that cannot be satisfied — "overlay" without both engines — now fails the prepare step instead of silently skipping. A skip on an explicit request reads as "the overlay was published" when nothing was; a skip on a partial dispatch that never mentioned it is just correct, and stays quiet. Exercised every input shape by extracting the step and running it: "" engines=[5 engines] overlay=true "sglang vllm atom kvd server" same overlay=true "sglang" engines=["sglang"] overlay=false, quiet "sglang vllm" engines=[2] overlay=true "sglang vllm overlay" engines=[2] (stripped) overlay=true "overlay" - error, exit 1 "overlay sglang" - error, exit 1 Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

An atom e2e job failed on a Spur node with: mktemp: failed to create directory via template '/tmp/infera-test.XXXXXX': Permission denied mkdir: cannot create directory '/hf': Permission denied tests/run_tests.sh: line 49: /failures.txt: Permission denied and then reported: (a tier failed but no per-test detail was captured — likely an image build error or a native crash before pytest ran; scan above.) The diagnosis was wrong in a way that costs real time. This script has no `set -e`, so the unchecked mktemp left $SCRATCH empty; every path built from it then retargeted the filesystem root ("$SCRATCH/hf" -> /hf), those failed too, and the run limped on to blame the image for what was actually an unwritable /tmp on one node. Check the result and stop, naming the directory and pointing at the node, so the next person reads one line instead of chasing a phantom build error. Verified by forcing the failure (TMPDIR at a nonexistent path, which fails for root too — an unwritable-mode directory does not reproduce it under root): before mktemp error, then CONTINUED with SCRATCH=[] after FATAL naming the directory, exit 1 after with a writable TMPDIR, unchanged: SCRATCH set, $SCRATCH/hf created Does not address why that node's /tmp was unwritable — that is an infra question — only that the run says so. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

… name examples/recipes/ referenced `rocm/infera-overlay:latest`, which is the name I built under locally. It does not exist on Docker Hub — 404 — so every recipe told the reader to deploy an image they cannot pull, and `imagePullPolicy: IfNotPresent` turns that into `manifest unknown` rather than anything that points at the cause. This is precisely the defect this branch already fixes for `rocm/infera:vllm-kimi-k3` in the Kimi-K3 page. I introduced the same one in #64 while writing the recipes. Now `inferaimage/infera-overlay:v0.2.1`, verified pullable with no credentials. The header comments carried the wrong name too and are updated with it — a stale image name in a comment misleads exactly as well as one in a field. Audited every image referenced by a manifest under examples/ and deploy/ against Docker Hub while here: inferaimage/infera-overlay:v0.2.1 ok lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x ok vllm/vllm-openai-rocm:kimi-k3 ok vllm/vllm-openai-rocm:v0.25.1 ok python:3.11-slim ok infera/engine-sglang-mi300x:dev 404 The last one is in the operator's CRD samples, where a `:dev` tag reads as "your image" rather than something to pull. Left alone deliberately, noted here so it is a decision rather than an oversight. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Two problems, one fix. The recipes still carried 20 references to rocm/infera-overlay:latest — the name I built under locally while writing #64, which 404s on Docker Hub. #67 fixed those on a branch cut from main; this branch predates that merge and had them back. A name that only exists on one laptop keeps coming back because a tag is a promise nobody checks. And the tags that do resolve are mutable. vllm/vllm-openai-rocm:kimi-k3 was rebuilt on 2026-07-27 during this very session, which is what invalidated the "prefix caching fails engine init" note: the tag was the same and the image was not. A recipe that says "this was validated" has to name the thing that was validated. So every image is now pinned by digest: inferaimage/infera-overlay @6918eff3 (v0.2.2, this session's fixes) vllm/vllm-openai-rocm:kimi-k3 @5aa7e626 lmsysorg/sglang-rocm:...k3-20260727 @3c01f73f lmsysorg/sglang:v0.5.15...mi35x @40e940a0 Zero mutable tags remain under examples/recipes/. Each digest is one I pulled and ran, not one read off a page. All nine manifests still pass `kubectl apply --dry-run=server` against the live CRD. The readable tag is kept alongside the digest so the ref still says what it is. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

jiejingzhangamd added 20 commits July 30, 2026 21:07

feat(examples): Kimi-K3 multimodal smoke test client (image + question)

d2337e2

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

style(examples): ruff-format mm_test.py (line wrapping)

c6684a1

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>

Merge branch 'feat/overlay-payload' into feat/recipes

18b3410

jiejingzhangamd requested review from JohnQinAMD, limou102 and xiaobochen-amd as code owners July 31, 2026 22:02

jiejingzhangamd added 4 commits July 31, 2026 22:16

jiejingzhangamd force-pushed the feat/recipes branch from 0601282 to c5debb2 Compare July 31, 2026 22:16

jiejingzhangamd added 2 commits July 31, 2026 22:24

jiejingzhangamd added 4 commits July 31, 2026 22:43

jiejingzhangamd changed the title ~~docs(recipes): per-model Kubernetes recipes for GLM-5.2 and Kimi-K3~~ feat(recipes): per-model Kubernetes recipes for GLM-5.2 and Kimi-K3, on stock vendor images Jul 31, 2026

jiejingzhangamd mentioned this pull request Jul 31, 2026

feat(deploy): overlay payload — decouple everything we ship from the vendor base #61

Merged

jiejingzhangamd added 2 commits August 1, 2026 00:39

jiejingzhangamd merged commit 59d540a into main Aug 1, 2026
28 checks passed

jiejingzhangamd mentioned this pull request Aug 1, 2026

fix(recipes): point at the published overlay image, not a local build name #67

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(recipes): per-model Kubernetes recipes for GLM-5.2 and Kimi-K3, on stock vendor images - #64

feat(recipes): per-model Kubernetes recipes for GLM-5.2 and Kimi-K3, on stock vendor images#64
jiejingzhangamd merged 32 commits into
mainfrom
feat/recipes

jiejingzhangamd commented Jul 31, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jiejingzhangamd commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Every manifest is stock base + overlay + sidecar

Validated end-to-end, and that is where most of this PR came from

Fixes this turned up

Release CI

Verification

Also

Depends on

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jiejingzhangamd commented Jul 31, 2026 •

edited

Loading