ValueError: The current device_map had weights offloaded to the disk.

[beratcmn]

Here is the complete error message:

ValueError: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for 
them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in 
this format.

Error in the second line

from nomic.gpt4all.gpt4all import GPT4AllGPU
m = GPT4AllGPU("decapoda-research/llama-7b-hf")

[dxenos]

I have the same problem when I use the command line:
python generate.py --config configs/generate/generate.yaml --prompt "Write a script to reverse a string in Python"

here is my generate.yaml file:

# model/tokenizer
model_name: decapoda-research/llama-7b-hf # REPLACE HERE with the base llama model
tokenizer_name: "gpt2"  # REPLACE HERE with the llama tokenizer
lora: true
lora_path: "nomic-ai/gpt4all-lora"

max_new_tokens: 512
temperature: 0
prompt: null

---

I use a macbook pro Ventura 13.3

[clxyder]

This article has a pretty good explanation on how to fix that issue.

https://huggingface.co/blog/accelerate-large-models

[NewtonJr4108]

This article has a pretty good explanation on how to fix that issue.

https://huggingface.co/blog/accelerate-large-models

How would you apply the fix to running gpt4all command?

[clxyder]

This article has a pretty good explanation on how to fix that issue.

https://huggingface.co/blog/accelerate-large-models

How would you apply the fix to running gpt4all command?

I think you just add offload="offload" to the from_pretrained() call

[NewtonJr4108]

Thanks!

[chaitanya-uc]

@NewtonJr4108 did you fix this? If yes, how? Because we won't be explicitly calling the from_pretrained() method.
@clxyder

[Celppu]

I have same problem. How do i provide offload_folder ?

[sadidul012]

Here is a example of offload_folder, found here - https://huggingface.co/blog/accelerate-large-models

import torch
from transformers import AutoModelForCausalLM

# Will go out of RAM on Colab
checkpoint = "facebook/opt-13b"
model = AutoModelForCausalLM.from_pretrained(
    checkpoint, device_map="auto", offload_folder="offload", torch_dtype=torch.float16
)

[helperton]

mkdir offload

Then inside of your python code,

model = AutoModelForCausalLM.from_pretrained(
...
...
offload_folder="/path/to/your/offload"
)

[niansa]

Stale? Please open a new issue if this is still relevant.

[djdylex]

I supplied a folder, double checked the path and everything - still same issue. Not sure what's going on. I'm using transform.from_pretrainer().

[cebtenzzre]

GPT4AllGPU is no longer provided by this repo, and is not supported by Nomic anymore.