RuntimeError with CUDA 12.2 on Windows using vLLM and Llava: No Kernel Image Available

[OualidBougzime]

I installed vLLM using the following command:
!pip install vllm==0.4.0 kaleido python-multipart torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2

Below is the code I am using with Llava:

import torch

from vllm import LLM
from vllm.sequence import MultiModalData

def run_llava_pixel_values():
    llm = LLM(
        model="llava-hf/llava-1.5-7b-hf",
        enforce_eager=True,
        tensor_parallel_size=1,
        image_input_type="pixel_values",
        image_token_id=32000,
        image_input_shape="1,3,336,336",
        image_feature_size=576,
    )

    prompt = "<image>" * 576 + (
        "\nUSER: What is the content of this image?\nASSISTANT:")

    # This should be provided by another online or offline component.
    import base64

    with open("2014 - Computational and Experimental Fluid Mechanics wit_image_1.jpg", "rb") as f:
        image_file = f.read()
        encoded = base64.b64encode(image_file).decode("utf-8")

    outputs = llm.generate(prompt,
                           multi_modal_data=MultiModalData(
                               type=MultiModalData.Type.IMAGE, data=encoded))
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

run_llava_pixel_values()

However, I am encountering the following error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Versions:
CUDA: 12.2
OS: Windows
Python: 3.12.2

Any insights or suggestions to resolve this error would be greatly appreciated.

[msk-nightly]

I'm facing a similar issue for a different project and found this to be insightful.