[RFC]: Create VllmState to save immutable args in VllmConfig #10666
Description
Motivation.
The VllmConfig becomes huge as more and more features are added in vLLM. But some of that I think is immutable to users, thus not suitable to be as a configure, e.g. worker_cls, and other private args.
Inspired by this comment from Nick, I think creating a dataclass VllmState could help making VllmConfig lighter and safer for users to specify.
Proposed Change.
I think VllmState is similar to VllmConfig, composed by similar member vars as VllmConfig.
@dataclass
class VllmState:
model_state: ModelState
cache_state: CacheState
parallel_state: ParallelState
scheduler_state: SchedulerState
device_state: DeviceState
load_state: LoadState
lora_state: Optional[LoRAState]
speculative_state: Optional[SpeculativeState]
decoding_state: Optional[DecodingState]
observability_state: Optional[ObservabilityState]
prompt_adapter_state: Optional[PromptAdapterState]
quant_state: Optional[QuantizationState]
compilation_state: CompilationStateAnd the initial member of VllmState will absorb the immutable args of the corresponding members in VllmConfig as its member variables. And in the meantime, xxxConfig will also be a member of xxxState.
VllmState will be constructed at the begining of the initialize of LLMEngine, and we could just use vllm_state as argument for executor/worker/model_runner/model.
Maybe other modules are missing, plz help me add if you know it
I'm sorting out the immutable vars of all the xxxConfig, I'll update this info in a table later. And there is a tiny example for ParallelState:
@dataclass
class ParallelState:
parallel_config: ParallelConfig
placement_group: Optional["PlacementGroup"]
worker_cls: str
world_size: int
rank: intFeedback Period.
No response
CC List.
cc @youkaichao @njhill @WoosukKwon @DarkLight1337
Any Other Things.
[Help wanted]
There are quite a lot args in VllmConfig, and only a few are marked as private. Is there any other way for me to find out the immutable member vars for users in VllmConfig? Or you can just post the vars you think should put into VllmState here. Thanks!
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.