kimik3-fp4-b300-vllm-agentic-dspark: add LMCache DRAM KV-offload arm - #2593
kimik3-fp4-b300-vllm-agentic-dspark: add LMCache DRAM KV-offload arm#2593sammshen wants to merge 5 commits into
Conversation
Add a dedicated config key kimik3-fp4-b300-vllm-agentic-dspark-lmcache with an LMCache 0.5.4rc2 DRAM KV-offload arm at TP8 conc 4/8/16, on top of the unchanged DSpark MTP serving stack of kimik3-fp4-b300-vllm-agentic-dspark (same image, script, and topology). A separate key means the changelog selects only the LMCache points; the resident and vllm-simple arms of the base key are not re-run. B300 sister of the MI355X arm in SemiAnalysisAI#2583.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
sammshen
added
agentx
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
/stage-results |
|
Superseded by #2597, which carries the same commits on a branch in this repo rather than a fork. Same-repo PRs get secrets, so |
1 participant
What
Adds a dedicated config key
kimik3-fp4-b300-vllm-agentic-dspark-lmcachewith an LMCache DRAM KV-offload arm at TP8 conc 4 / 8 / 10 / 16, on top of the unchanged DSpark MTP serving stack ofkimik3-fp4-b300-vllm-agentic-dspark(same image, script, and topology). A separate key means the changelog selects only the LMCache points — the resident and vllm-simple arms of the base key are not re-run.B300 sister of the MI355X arm in #2583; same LMCache version (0.5.4rc2), same MP-server topology, CUDA install path and NVIDIA-side chunk size.
Changes
kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.4rc2" }, conc-list: [4, 8, 10, 16], spec-decoding: mtp, at the base key'sdram-utilization: 0.63(1,889 GB aggregate at TP8). 4/8/16 land on the base key's vllm-simple ladder at its top three offload points, so LMCache is directly comparable there; 10 fills the 8->16 gap.lmcache)case arm:v0.5.4rc2-cu129release assets — the CUDA counterpart of the-rocmassets the MI355X sister arm installs), not the generic PyPI wheel, via the upstream install line with--extra-index-url https://download.pytorch.org/whl/cu129.--index-strategy unsafe-best-matchhas no pip counterpart because it needs none: pip already resolves best-match across every configured index, which is exactly what that flag restores in uv.torchis an unpinned LMCache requirement the image already satisfies, so its tested build is left alone. A fail-fast import check follows.--chunk-size 768(that recipe's CUDA-path value),--separate-object-groups(one object group per sliding-window size for the hybrid KDA/MLA layout, which has more than one KV-cache group under MTP),--enable-extra-logging,--max-cpu-workers 8 --max-gpu-workers 1,--l1-size-gb $TOTAL_CPU_DRAM_GB, LRU eviction,--shm-name ""so L1 lives in ordinary process memory rather than being capped by /dev/shm.LMCacheMPConnector(lmcache.mp.port), keeping the DSpark--speculative-configuntouched.lmcachetoo.TOTAL_CPU_DRAM_GBverbatim per the agentic README.Note on
--chunk-size: the connector requires the chunk to be a multiple of every engine KV group'stokens_per_block. 768 is the published CUDA-path value; the ROCm sister arm needs 3072 because its hybrid layout registers 1536-token attention groups and a 3072-token KDA state group. If the B300 server log reports a mismatch, this is the one value to raise — the script comment says so and names the log lines to read.Validation
process_changelog.pyrun exactly as CI does (base = main) emits a 4-row matrix, nothing else:generate_sweep_configs.py test-configpasses for both the new key and the (unchanged) base key;bash -npasses on the modified script;utils/changelog_gate_tests/test_validate_perf_changelog.pyandutils/test_process_changelog.pypass (35 tests).