[Bug]: Phi-3-Small model reporting AttributeError: 'NoneType' object has no attribute 'prefill_metadata'

[dnaihao]

Your current environment

I am using VLLM 0.9.1, and the flash_attention installed by
MAX_JOBS=4 pip install flash-attn --no-build-isolation

🐛 Describe the bug

When I try to run Phi-3 Small models (both 8k and 128k), it would throw an error of

AttributeError: 'NoneType' object has no attribute 'prefill_metadata'

specifically in vllm/attention/backends/blocksparse_attn.py line 416, where the code tries to access prefill_metadata by calling if prefill_meta := attn_metadata.prefill_metadata:.

I tried to trace the error myself, it went all the way in the phi3_small.py where the model calls the self.attn in the pass, but I cannot find where the model fills the attn_metadata, I saw from other posts that the model is using the context manager to handle this, which I am not quite familiar with.

Also, other models such as Llama would not have such an issue, I suspect this to be related to the attention type, as for phi 3 small, it would run the torch.ops.vllm.unified_attention in vllm/attention/layer.py, but Llama model would not touch such a line.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[lengrongfu]

Can you provide run vllm command? @dnaihao

[dnaihao]

@lengrongfu
Sure! Here is the minimal vllm command to reproduce the issue:

import transformers
from vllm import LLM, SamplingParams

model = "microsoft/Phi-3-small-128k-instruct"

prompts = "<|endoftext|><|user|>\n hello world.<|end|>\n<|assistant|>"

llm = LLM(
      model=model,
      tensor_parallel_size=1,
      gpu_memory_utilization=0.85,
      trust_remote_code=True,
  )

tokenizer = transformers.AutoTokenizer.from_pretrained(model)

sampling_params = SamplingParams(
            max_tokens=512,
            temperature=0.0,
            top_p=1.0,
            top_k=-1,
        )

print(llm.generate(prompts, sampling_params))

[lengrongfu]

like this issue #16493

[dnaihao]

@lengrongfu Thanks for the pointer! It seems that in #16493 there is still no solution yet for V1 to support flash attention. Shall I revert to V0 if I am to run models such as Phi 3 small?

[dnaihao]

My current way of running it through is to use the engine v0, so basically by setting

export VLLM_USE_V1=0

In addition, when running the Phi model, you will encounter issue of #7787, therefore we need to set enable_chunked_prefill=False if you are using LLM imported from vllm.

This makes me run Phi-3-Small-8k model successfully.

---

Notes:

It is confusing as it seems #7787 suggests there is a fix for this issue already, but I have checked v0.7.2, v0.6.5, both versions still have the issue described in #7787, while v0.6.4 gives another issue mentioned in #12124.

In addition, it seems that as suggested in #7787, the solution of using enable_chunked_prefill=False would make the program slower, haven't run my whole program through and do not have an idea of how slow it would be.

Furthermore, I believe setting enable_chunked_prefill=False would make it hard to fit the 128K version of the Phi-3 Small model (on a single GPU Mem of 46068MiB), it would throw something like the max_seq_len greater than the token length in KV stuff. It would not solve the issue even if we increase the gpu_memory_utilization (max_seq_len in this case is too large). And it seems that it would be tricky to set max_seq_len in VLLM as mentioned in #7195.

Considering all of these, does the VLLM team consider somehow better supporting the Phi-related models in the future version? I have also noticed many open/closed issues related to Phi-3, Phi-4 models.