How to enable vllm

[lucasjinreal]

Feature request

How to enable vllm

Motivation

How to enable vllm

Your contribution

How to enable vllm

[OlivierDehaene]

Use 0.9 and a supported model

[lucasjinreal]

@OlivierDehaene Hi, don' know if you are stuff in huggingface or not,
but PLEASE indicates more details not just type use 0.9

1. Does it enabled by default?
2. if not, how to enable?
3. any documentations indicates how to use?
   I never know a huggingface stuff can be so impatient to github community member.

[Narsil]

can be so impatient to github community member.

Please read your initial "Feature request" and tell me you did the effort to actually express intelligibly a feature request ?
Is this a bug ?

There are many ways you could have phrased that, starting with a request to improve the docs because you couldn't find what you want. What is the actual question you had, where did you look for it, and what did you find instead.

Also filling in properly the template instead of repeating the same thing over.

Our effort in replying scale with your effort.

1. Yes
2. It's always on ,there's no opt-out, it's just better overall that what was there previously.

There's no doc for it because well, it's not necessary, if you use flash models, you just get it.

It's also NOT vllm. It's a custom variant of PagedAttention, which is what makes vllm faster. We do reuse a slightly modified version of their low level kernel.

[lucasjinreal]

@Narsil Hi, where does PagedAttention custom kernel included in?