Please describe the feature you want
On my "old" computer (having core i7 + RTX 3070 FE + 32Go RAM) I cannot launch any of the docker container or the binary.
My CPU doesn't support AVX2, it only accepts AVX.
I think it's the problem as I already had this error (invlaid instruction) when I tried LM Studio.
Using llama.cpp with clblast for some other tests (outside of TabbyML) worked. For example, llama.cpp server (python binding) is OK with Mistral and Llama2 models.
On my other computer, with the same Fedora version, it works (laptop with RTX 3060). This other computer has AVX2 support.
Additional context
I tried to compile myself the Docker image, the problem remains. I'm not sure if AVX is correctly set.
EDIT:
Let me rephrase.
LM Studio doesn't offer a binary that supports AVX, only AVX2. On this machine, I have exactly the same problem with TabblyML.
When I compiled llama.cpp (the python binding), AVX instructions are supported. This means I can run llama.cpp server and use GUFF models.
So I'm sure llama.cpp will work, provided the compilation options aren't restricted to AVX2.
However, with TabblyML, I get this error with binaries, the official Docker image, and the image I build myself. So I “think” that TabbyML disables the AVX option in favor of AVX2.
What I'm asking is if there's an option somewhere for me to force the use of AVX.
Please reply with a 👍 if you want this feature.
Please describe the feature you want
On my "old" computer (having core i7 + RTX 3070 FE + 32Go RAM) I cannot launch any of the docker container or the binary.
My CPU doesn't support AVX2, it only accepts AVX.
I think it's the problem as I already had this error (
invlaid instruction) when I tried LM Studio.Using llama.cpp with clblast for some other tests (outside of TabbyML) worked. For example, llama.cpp server (python binding) is OK with Mistral and Llama2 models.
On my other computer, with the same Fedora version, it works (laptop with RTX 3060). This other computer has AVX2 support.
Additional context
I tried to compile myself the Docker image, the problem remains. I'm not sure if AVX is correctly set.
EDIT:
Let me rephrase.
LM Studio doesn't offer a binary that supports AVX, only AVX2. On this machine, I have exactly the same problem with TabblyML.
When I compiled llama.cpp (the python binding), AVX instructions are supported. This means I can run llama.cpp server and use GUFF models.
So I'm sure llama.cpp will work, provided the compilation options aren't restricted to AVX2.
However, with TabblyML, I get this error with binaries, the official Docker image, and the image I build myself. So I “think” that TabbyML disables the AVX option in favor of AVX2.
What I'm asking is if there's an option somewhere for me to force the use of AVX.
Please reply with a 👍 if you want this feature.