Get error when compiling.

[Cortega13]

Hello! I am trying to run exllama on wizard-vicuna-13b-uncensored-gptq, and when i try to run any of the commands I get the following error. I am running it using the nvidia pytorch image nvcr.io/nvidia/pytorch:23.05-py3. I am using the newest version of cuda 12.1.1 and running it on a google vm with an L4 on ubuntu 18.04 LTS. I know the documentation says its not compatible with all gpu's, is it compatible with the L4? Any help would be very much appreciated. Thank you!!

error.txt

[turboderp]

I don't know anything about the L4, sadly. But the error you're getting seems to be because for some reason it's targeting some very old compute versions. Perhaps the PyTorch image is just set up like that. But you could try this before running it:

export TORCH_CUDA_ARCH_LIST="sm_89"

The L4 should support compute 8.9, I think. Then at least you shouldn't get those errors. But whether it will actually work and what the performance will be, no idea. It's a low-bandwidth, "energy efficient" GPU by the looks of it.

[Cortega13]

Thank you!!! The command was export TORCH_CUDA_ARCH_LIST="8.9"