Skip to content

Design the minimal vLLM integration boundary #7

Description

@DaBestCode

Define how RandKV should integrate with vLLM before coupling the policy to a fast-moving internal API. The output is a reviewed design plus the smallest testable prototype.

Acceptance criteria:

  • Pin and document the target vLLM version/commit.
  • Map prompt protection, recent-buffer protection, deterministic request identity, eviction cadence, and observability onto vLLM concepts.
  • Explain whether selection can remain independent per KV head when vLLM allocates paged blocks; call out any semantic compromise explicitly.
  • Identify the narrow public or internal hook and its upgrade risk.
  • Include call-site-first configuration requiring no more than one RandKV option.
  • Add CPU-testable policy/adapter boundary tests before requiring CUDA.
  • Document cancellation, prefix caching, chunked prefill, and multi-request behavior.

Start with a design document or draft PR; do not optimize kernels in this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: vllmvLLM integration workhelp wantedExtra attention is neededlaunch blockerRequired before performance-focused official launch

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions