Invalid bitcast %222 = bitcast <8 x i32> %221 to <8 x half>

[mgagvani]

Output from running mlc_chat_cli:

WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Use lib /cluster/2024mgagvani/dist/lib/vicuna-v1-7b_vulkan_float16.so
Initializing the chat module...
Finish loading
You can use the following special commands:
  /help    print the special commands
  /exit    quit the cli
  /stats   print out the latest stats (token/sec)
  /reset   restart a fresh chat

USER: what is 1 + 1
ASSISTANT: Invalid bitcast
  %222 = bitcast <8 x i32> %221 to <8 x half>
LLVM ERROR: Broken function
Aborted (core dumped)

Debug info:

(mlc-chat) 2024mgagvani@snowy:~$ uname -r
5.15.0-67-generic

System Specs: AMD Threadripper 1950X CPU and Nvidia GeForce 2080 GPU.

[junrushao]

WARNING: lavapipe is not a conformant vulkan implementation, testing use only.

Seems that the Vulkan driver is not properly installed, because lavapipe is a CPU simulation of Vulkan

[zwm88]

I have the same issue.
My driver is :nvidia-driver-530 (530.41.03-0ubuntu0.20.04.2).
5.15.0-71-generic

The demo video also shows "WARNING: lavapipe is not a conformant vulkan implementation, testing use only.", so it might not be the issue.

[jlmouton]

If you already have your NVIDIA driver installed, make sure your system is picking it up by setting the Vulkan ICD environment variable:
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

If that's not solving your issue (it did for me), you probably already know that on some Linux distribution (Debian for instance), there is no support for NVIDIA GPUs, so you have to go through the setup. See here: https://phoenixnap.com/kb/nvidia-drivers-debian

I hope this helps. Once the Vulkan driver started to use my NVIDIA Tesla T4 on my VM, all went running fine.

[junrushao]

The demo video also shows "WARNING: lavapipe is not a conformant vulkan implementation, testing use only.", so it might not be the issue.

That's correct. lavapipe is a CPU-simulated Vulkan implementation. If this message pops out, it probably means you might want to install a proper version of Vulkan