Which HF LLMs in spacy-llm are available for textcat?

[anammari]

I'm trying to use HuggingFace LLMs for text classification. I've tried the OpenAI code example under usage_examples/textcat_openai with OpenAI models such as spacy.GPT-3-5.v1 and spacy.Text-Davinci.v1 etc and they all seem to work. I am now trying to apply the same use case using HF LLMs (e.g. Dolly) but I can't get these to work.

For example, I've looked at what registered LLMs are available in spacy_llm/models/hf/dolly.py and tried to change the config file fewshot.cfg accordingly:

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-3b"

This however has returned error:

ValueError: Install CUDA to load and run the LLM on the GPU, or install 'accelerate' to dynamically distribute the LLM on the CPU or even the hard disk. The latter may be slow.

Are these HF LLMs supported for textcat use cases?
If so, am I using them in the wrong way?
Some guidance on this will be much appreciated.

[rmitsch]

Hi @anammari, all models are supported for all tasks, including textcat. The error message indicates that you don't have the necessary libraries installed to run this model (as not all users will want to run local/HF models, we don't require the installation of these libraries by default).

So if you have a GPU on your system: make sure CUDA is properly installed (see here for Windows and here for Linux). If you don't have a GPU: run pip install accelerate.

If this doesn't work for you, let me know.

[cobaltautomationdev]

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

This code return [] based on below config . is there anything wrong?
https://spacy.io/usage/large-language-models

config.cfg
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["PERSON", "ORGANISATION", "LOCATION"]

[components.llm.model]
@llm_models = "spacy.Dolly.v1"

For better performance, use dolly-v2-12b instead

name = "dolly-v2-3b"

[rmitsch]

dolly-v2-3b is a rather small and (relatively, in this field) old model. I recommend using a newer/larger model. In my experience, OS models so far are not as finely tuned w.r.t. returning responses in the specified fashion as commercially available ones. So you might have to experiment with the prompt a bit as well to get decent results.

Also, inspect the raw prompts and responses to debug the problem:

[components.llm]
factory = "llm"
save_io = True

...
# Raw prompt:
print(doc.user_data["llm_io"]["llm"]["response"])
# Raw response:
print(doc.user_data["llm_io"]["llm"]["response"])