[Issue]Recompute mode affects the latency of the request

[wjj19950828]

Issue

In high concurrency situations, the latency of some requests is too long.

describe

Hello, thank you vllm for your contribution in large model deployment

Currently I have encountered a problem: when the bs/concurrency is high, the recompute mode will be triggered. This situation is not friendly to the latency of some requests because it needs to be recalculated.

Is there any good solution to this situation? Thanks ~

[wjj19950828]

@viktor-ferenczi @WoosukKwon
Do you have any suggestions over there?

Thanks~

[hmellor]

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.