[RFC]: Async scheduler and Multi-step in v1

[chengda-wu]

🚀 The feature, motivation and pitch

We propose reducing the framework’s overhead by introducing two key improvements—an asynchronous scheduler and a multi-step approach—all with minimal code modifications.

1. Asynchronous Scheduler

For the async scheduler, we suggest the following design:

This solution requires changes only in EngineCore without modifying other modules. By incorporating an update_schedule module, the framework can also seamlessly support speculative decoding.

2. Multi-Step Approach

Although v1’s preprocessing and postprocessing are lighter compared to v0, we still observe notable inefficiencies on some platforms (e.g., ARM + XPU). In particular, there exists a significant gap between input preparation and launching the forward model, and the device-to-host (D2H) communication for each model output further increases overall latency.

To address these issues, we introduce a multi-step strategy that differs from v0 in several key ways:
• We propose a simple_prepare_input function to reduce unnecessary CPU operations.
• We defer the D2H communication to avoid excessive stream synchronizations.
• We integrate the multi-step process with the asynchronous scheduler, thereby alleviating the scheduler’s load when handling multiple outputs.

─────────────────────────────

This refined solution minimizes code modifications while significantly improving performance and reducing overheads.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[mgoin]

There is work in progress specifically for the async scheduler #19970

[chengda-wu]

There is work in progress specifically for the async scheduler #19970

Based on our idea, we only need to modify engine/core.py by adding a new step function, while in #19970 , several files are modified.

[Ling-CF]

Has the code been submitted, or when will it be made public?

[chengda-wu]

Has the code been submitted, or when will it be made public?

Currently, we work on omniinfer, see async scheduler, multi step and model runner. All these codes will be contributed to vllm.

[noooop]

Based on my own experience implementing asynchronous scheduling, this bubble is almost negligible(14ms vs 0.3ms), meaning that asynchronous scheduling is almost useless for decoding models.

[plops655]

It would allow for better scheduling algorithms which take longer latency. The longer latency would be masked away.