Skip to content

v0.6.0: bug hunting on vLLM offloading and LMCache

Choose a tag to compare

@sameer-sde sameer-sde released this 17 Nov 14:15
· 10 commits to main since this release
ac44665

Three targets, each run with its own code unmodified. vLLM's two CPU
offloading connectors: clean, planted faults caught, storage failures
recovered. LMCache: two real, open bugs found independently and shrunk to
minimal reproductions (both already reported upstream, so no duplicates
filed); Phantom's A/B tests show both open fixes work. SGLang, llm-d and
the NIXL fault classes were not covered (they need GPUs).

  • vLLM's SimpleCPUOffloadConnector on the CPU backend: its manager runs
    unmodified, only the CUDA copy engine is replaced. Six vllm-fuzz
    profiles (cpu-offload*); 0 violations so far, 3,753 blocks stored and
    317 loaded back and verified.
  • Offload copy engine with separate load/store queues (like vLLM's two CUDA
    streams); planted store-reported-early caught; offloading in CI.
  • phantomkv vllm-diff: run a seed twice, show the first step where the
    runs differ. Found that the harness stopped stepping while vLLM still had
    pending connector work; fixed.
  • vLLM's OffloadingConnector (target 2) through its pluggable
    OffloadingSpec, no vLLM code patched: offload2* profiles, planted
    store-reported-early caught, and storage failures (load-fails,
    store-fails) recovered correctly by vLLM; in CI.
  • Partial-chunk evidence, revisit-heavy chunk profiles and
    scripts/offload_campaign.sh: 50 seeds per run, 8 profiles clean, planted
    fault caught 24/50, storage failures recovered. README Phase 5 section.
  • LMCache (target 3): scripts/lmcache_smoke.py, vLLM + LMCache's
    LMCacheMPConnector + a CPU-only LMCache server; 928 tokens stored and
    loaded back through LMCache, verified by Phantom.
  • LMCache under vllm-fuzz (lmcache* profiles, one server per campaign,
    restart fault, planted bug in LMCache's scatter path caught 10/10).
    Lead: after a fast server restart vLLM never re-registers with LMCache.
  • lmcache-restart-settle profile. The restart lead is a known LMCache bug
    (issue #4642), found independently; an A/B test of the open fix
    (PR #6106) shows caching recovers after every restart.
  • LMCache lock and object counts after every run (from /status),
    lmcache-only-noabort, and three scripts: a one-abort probe, a lock-leak
    shrinker (ddmin with a custom failure test) and a replayer for the saved
    minimal cases. The Day 22 replay differences are a read-lock leak on
    aborted waiting requests (known upstream, issue #5656); A/B of the open
    fix (PR #6540): 0 leaked locks, all seeds identical on replay.