Skip to content

fix(vllm): preserve cache keys across restarts - #278

Merged
ketor merged 2 commits into
dingodb:mainfrom
ketor:fix-rdma-sg-lane-rail-affinity
Aug 9, 2026
Merged

fix(vllm): preserve cache keys across restarts#278
ketor merged 2 commits into
dingodb:mainfrom
ketor:fix-rdma-sg-lane-rail-affinity

Conversation

@ketor

@ketor ketor commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • require a fixed PYTHONHASHSEED together with SHA-256 block hashing because current vLLM seeds the root block hash from os.urandom() when unset
  • invalidate process-shared absent rendezvous entries after successful scalar, batch, and SG PUTs
  • document the cross-process identity requirement and cover both guards with regression tests

0064 evidence

  • before: identical 94k-token prompt after engine restart probed 5,748 candidates with 0 present; first block key changed across restarts
  • after PYTHONHASHSEED=0: first block key remained identical and 5,872/5,872 objects were found after restart
  • cold SSD retrieval: 35.85 GB useful TP payload in 1.64-1.75 s; nvme0n1 2.11 GB/s + nvme2n1 2.18 GB/s; request 2.82 s; zero RDMA completion errors
  • warm retrieval: 35.85 GB useful TP payload in 1.11-1.18 s; request 2.72 s; zero RDMA completion errors

Checks

  • python3 -m unittest test_determinism_guard: 6 passed
  • client_metrics_test filters for PUT invalidation and REMOVE invalidation: 2 passed
  • clean end-to-end restart smoke: 94k-token cached prompt completed in 2.16 s

@ketor
ketor added this pull request to the merge queue Aug 9, 2026
Merged via the queue into dingodb:main with commit b1e7335 Aug 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant