Run on CPU without AVX2

[ZanMax]

Hello,
I have a server with Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz and 5x WX9100 and want to run Mistral 7b on each GPU.
But I received an error: "Illegal instruction (core dumped)" when I tried to do it.
Is it possible to run exllama on the CPU without AVX2?

[turboderp]

Are you on the latest version?

[ZanMax]

steps:

git clone https://github.com/turboderp/exllama
cd exllama
pip install -r requirements.txt
python test_benchmark_inference.py -d <path_to_model_files> -p -ppl

result

python test_benchmark_inference.py -d /home/dev/models/Mistral-7B-Instruct-v0.2-GPTQ/ -p -ppl
Successfully preprocessed all matching files.
-- Perplexity:
-- - Dataset: datasets/wikitext2_val_sample.jsonl
-- - Chunks: 100
-- - Chunk size: 2048 -> 2048
-- - Chunk overlap: 0
-- - Min. chunk size: 50
-- - Key: text
-- Tokenizer: /home/dev/models/Mistral-7B-Instruct-v0.2-GPTQ/tokenizer.model
-- Model config: /home/dev/models/Mistral-7B-Instruct-v0.2-GPTQ/config.json
-- Model: /home/dev/models/Mistral-7B-Instruct-v0.2-GPTQ/model.safetensors
-- Sequence length: 2048
-- Tuning:
-- --sdp_thd: 8
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --rmsnorm_no_half2
-- --rope_no_half2
-- --matmul_no_half2
-- --silu_no_half2
-- Options: ['perf', 'perplexity']
** Time, Load model: 21.56 seconds
** Time, Load tokenizer: 0.02 seconds
-- Groupsize (inferred): 128
-- Act-order (inferred): yes
** VRAM, Model: [cuda:0] 3,877.87 MB - [cuda:1] 0.00 MB - [cuda:2] 0.00 MB - [cuda:3] 0.00 MB - [cuda:4] 0.00 MB
** VRAM, Cache: [cuda:0] 256.00 MB - [cuda:1] 0.00 MB - [cuda:2] 0.00 MB - [cuda:3] 0.00 MB - [cuda:4] 0.00 MB
-- Warmup pass 1...
Illegal instruction (core dumped)

As I know Illegal instruction (core dumped) means that problem with AVX2 instruction. When I tried the GGUF format with llama.cpp I received the same Illegal instruction (core dumped).

[ZanMax]

Maybe this gives more information about an error:

gdb --args python3 test_benchmark_inference.py -d /home/dev/models/Mistral-7B-Instruct-v0.2-GPTQ/ -p -ppl

#0 0x00007fff4e89540e in rocblas_hgemm () from /home/dev/workspace/numpy_no_avx2/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
#1 0x00007fff86e491dd in hipblasHgemm () from /home/dev/workspace/numpy_no_avx2/venv/lib/python3.10/site-packages/torch/lib/libhipblas.so
#2 0x00007ffe8ba50855 in q4_matmul_recons_cuda(ExLlamaTuning*, __half const*, int, Q4Matrix*, __half*, void*, bool) () from /home/dev/.cache/torch_extensions/py310_cpu/exllama_ext/exllama_ext.so
#3 0x00007ffe8ba364e8 in q4_matmul(at::Tensor, unsigned long, at::Tensor) () from /home/dev/.cache/torch_extensions/py310_cpu/exllama_ext/exllama_ext.so
#4 0x00007ffe8ba4e423 in pybind11::cpp_function::initialize<void (&)(at::Tensor, unsigned long, at::Tensor), void, at::Tensor, unsigned long, at::Tensor, pybind11::name, pybind11::scope, pybind11::sibling, char [10]>(void (&)(at::Tensor, unsigned long, at::Tensor), void ()(at::Tensor, unsigned long, at::Tensor), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&, char const (&) [10])::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) () from /home/dev/.cache/torch_extensions/py310_cpu/exllama_ext/exllama_ext.so
#5 0x00007ffe8ba4aa4d in pybind11::cpp_function::dispatcher(_object, _object*, _object*) () from /home/dev/.cache/torch_extensions/py310_cpu/exllama_ext/exllama_ext.so
#6 0x00005555556ae10e in ?? ()
#7 0x00005555556a4a7b in _PyObject_MakeTpCall ()
#8 0x000055555569d096 in _PyEval_EvalFrameDefault ()
#9 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#10 0x000055555569ccfa in _PyEval_EvalFrameDefault ()
#11 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#12 0x000055555569745c in _PyEval_EvalFrameDefault ()
#13 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#14 0x000055555569745c in _PyEval_EvalFrameDefault ()
#15 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#16 0x000055555569745c in _PyEval_EvalFrameDefault ()
#17 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#18 0x000055555569745c in _PyEval_EvalFrameDefault ()
#19 0x00005555556bc7f1 in ?? ()
#20 0x000055555569853c in _PyEval_EvalFrameDefault ()
#21 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#22 0x000055555569726d in _PyEval_EvalFrameDefault ()
#23 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#24 0x000055555569726d in _PyEval_EvalFrameDefault ()
#25 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#26 0x000055555569726d in _PyEval_EvalFrameDefault ()
#27 0x00005555556939c6 in ?? ()
#28 0x0000555555789256 in PyEval_EvalCode ()
#29 0x00005555557b4108 in ?? ()
#30 0x00005555557ad9cb in ?? ()
#31 0x00005555557b3e55 in ?? ()
#32 0x00005555557b3338 in _PyRun_SimpleFileObject ()
#33 0x00005555557b2f83 in _PyRun_AnyFileObject ()
#34 0x00005555557a5a5e in Py_RunMain ()
#35 0x000055555577c02d in Py_BytesMain ()
#36 0x00007ffff7c7ed90 in __libc_start_call_main (main=main@entry=0x55555577bff0, argc=argc@entry=6, argv=argv@entry=0x7fffffffe328) at ../sysdeps/nptl/libc_start_call_main.h:58
#37 0x00007ffff7c7ee40 in __libc_start_main_impl (main=0x55555577bff0, argc=6, argv=0x7fffffffe328, init=, fini=, rtld_fini=, stack_end=0x7fffffffe318)
at ../csu/libc-start.c:392
#38 0x000055555577bf25 in _start ()