Help with spacy-llm Llama2 config

[fireblonde]

Hi,

I am following the tutorial for Llama2. My config is as follows:

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 15

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["INSULT", "THREAT"]

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-13b-hf"

and I am getting this error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 23.67 GiB total capacity; 22.21 GiB already allocated; 157.81 MiB free; 22.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I tried using all of the available llama2 models. If I use the smallest model, then I get this error (similar to another discussion):

ValueError: The following `model_kwargs` are not used by the model: ['use_auth_token'] (note: typos in the generate arguments will also show up in this list)

Can you please help me with this? Is there a possibility to use accelerate and if yes, can you help me with a more detailed instructions?

Thank you so much in advance! Looking forward to your answers! :)

[rmitsch]

Hi @fireblonde! Let's start with the second error message. Have you

• requested and been granted permission to use the Llama 2 model and
• are logged in to your Hugging Face account on the machine you're running this from?

See the "Gated models on Hugging Face" section here.

[rmitsch]

Alright. Have you tried passing the device_map setting to spacy_llm? I. e.

...
config_init = {"device_map": "auto"}
...

Passing torch.dtype isn't possible yet, but on our roadmap.

[fireblonde]

If I have both the device to be -1 and device_map to be auto I get this when running 13b:

ValueError: The model has been loaded with `accelerate` and therefore cannot be moved to a specific device. Please discard the `device` argument when creating your pipeline object.

If I only run it with device_map to be auto then it outputs the same:

ValueError: The model has been loaded with `accelerate` and therefore cannot be moved to a specific device. Please discard the `device` argument when creating your pipeline object.

[fireblonde]

I was also reading on HF website about the models' GPU specs:

They mention:

    For 7B models, we advise you to select "GPU [medium] - 1x Nvidia A10G".
    For 13B models, we advise you to select "GPU [xlarge] - 1x Nvidia A100".
    For 70B models, we advise you to select "GPU [2xlarge] - 2x Nvidia A100" with bitsandbytes quantization enabled or "GPU [4xlarge] - 4x Nvidia A100"

I work on NVIDIA RTX A5000 :(

[rmitsch]

If I only run it with device_map to be auto then it outputs the same:

That's because we move this model to cuda:0 by default...I guess that's not a good idea after all since device_map="auto" is an option. We'll change that in the next patch version.

[rmitsch]

spacy-llm v0.6.3 is out and should fix these device binding issue. I recommend updating the library and giving it another try.