While executing the following code:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_id = "EleutherAI/gpt-neox-20b"
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)
Encountered the following Issue:
AttributeError:
/envs/myenv/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_fp16_fp4
I was trying Lightweight Inference with Large Language Models Using QLoRa, as mentioned in the blog:
https://medium.com/@bnjmn_marie/lightweight-inference-with-large-language-models-using-qlora-335a3f029229
but got this error. I tried to reinstall the latest version of bitsandbytes but it doesn't solve the problem.
While executing the following code:
Encountered the following Issue:
I was trying Lightweight Inference with Large Language Models Using QLoRa, as mentioned in the blog:
https://medium.com/@bnjmn_marie/lightweight-inference-with-large-language-models-using-qlora-335a3f029229
but got this error. I tried to reinstall the latest version of bitsandbytes but it doesn't solve the problem.