feature: Support for vLLM V1 Sleep & Wake_up Mode

[dumb0002]

Describe the feature

vLLM V1 provides a new feature that allows to put an engine into a sleep mode, as well as waking up an engine. A sleeping engine does not process any requests and free up resources (e.g, GPU memory). For example: Level 1 sleep mode will offload the model weights into CPU for backed up and discard the kv cache.

Hence, the Production Stack router needs to be extended to support serving request for vLLM engines with the flag --enable-sleep-mode set. The router should not forward requests to a sleeping engine.

More details:

• vLLM sleep function: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py#L1239
• vLLM wake_up function: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py#L1261

Why do you need this feature?

• Provides support for vLLM V1 sleep and wake_up mode feature that allows for better resource management for a fleet of running vLLM engines.
• Allows high level applications that uses this feature to be built on top of the Production Stack

Additional context

No response

[dumb0002]

I'm willing to propose a solution and take a stab at implementing this feature.

[dumb0002]

Here is a high level description of the requirements for the solution that I have in mind (cc: @YuhanLiu11 ):

• Extend the router to support sending a request to a specific vLLM engine, for example: sleep OR wake_up request for a specific vLLM engine;

• When handling sleep or wake_up requests, the router will save the state for an engine (if it was put to sleep or not) - the router already keeps track of the url, model_name and added_timestamp for each discovered engine. Then, a new state could be added is_sleeping= true or false. Furthermore, the router can also make call to is_sleeping to double-check if necessary;

• When the router confirms that an engine is sleeping it can add a label to the pod so that if there are other routers, they can pick up the change and update their local cache (e.g, set is_sleeping=true) - the router is already doing a k8s watch to discovery new engines and updates. Hence, all the routers will become aware that a specific vLLM instance is sleeping and not ready to handle requests;

• Similarly, when a vLLM engine is wake up (same process as described above);

• The router would then check this new saved state is_sleeping when selecting an engine instance to handle a request;

[YuhanLiu11]

@dumb0002 This sounds like a good design! Looking forward to your implementation! Thanks 😄

[dumb0002]

@YuhanLiu11, this PR adds the first feature to enable support of the v1 sleep and wake_up mode: #438 (the first bullet point outlined above). The remaining missing features will be added in following PRs.

[dumb0002]

This PR enables the Enable first level of support for vLLM V1 Sleep & Wake_up Mode: #441

[dumb0002]

This PR completes adding the support for the vLLM v1 Sleep & Wake_up Mode: #498