NeMo-Guardrails does not work with many other LLM providers #50
Comments
|
Hi @serhatgktp! Yes, at the end of May we made it technically possible to connect multiple types of LLMs. But, in order to get them to work properly the prompts would also need to be tweaked. We've been working on a set of mechanisms for that. They will be pushed to the repo next week and released to PyPI at the end of next week as 0.3.0. We did manage to successfully use |
|
@serhatgktp: you can check out this example: https://github.com/NVIDIA/NeMo-Guardrails/tree/main/examples/llm/hf_pipeline_dolly for how to use HuggingFacePipeline to run a local model. I did not get a chance to look into your specific configuration just yet. Let me know if this helps. |
|
Thanks @drazvan, this example seems to work! However, I had to modify the configuration as I was getting the following error with regards to the I modified @lru_cache
def get_dolly_v2_3b_llm():
repo_id = "databricks/dolly-v2-3b"
params = {"temperature": 0, "max_length": 1024}
llm = HuggingFacePipeline.from_model_id(
model_id=repo_id, task="text-generation", model_kwargs=params
)
return llmIt seems that my computer does not have any CUDA-enabled GPUs, which causes an issue when we try to select the "first" CUDA device. Thanks again! |
|
@serhatgktp I was wondering if you got this warning when you successfully ran the example. |
@QUANGLEA Yes, I'm getting that warning as well. However, it seems that using I haven't had much time to look into it so I've chosen to ignore it for now. |
|
@serhatgktp I'm getting the same response when using |
serhatgktp
changed the title
@QUANGLEA I've tried several other models, mainly the promising models from Hugging Face Hub such as Falcon. Unfortunately, I haven't been able to get any of them to work yet. |
|
@serhatgktp What LLMs are you interested in? Were there any specific errors? I'm asking because we will be testing a few more LLM providers on our end in the next couple of weeks. So, maybe we can align our efforts. Thanks! |
|
Thank you for the follow-up @drazvan. I've been trying to choose LLMs that are seemingly powerful, accurate, and not too large. The following two are the ones I'm most interested in at the moment:
1) Issue with Hugging Face Hub Modelsa) Using the Built-in Feature for Hugging Face HubThe built-in support for Hugging Face Hub currently has bugs as it is giving me the following error: Configuration: models:
- type: main
engine: huggingface_hub
model: tiiuae/falcon-7b-instruct Error: it seems that the code below is failing: elif hasattr(self.llm, "model_kwargs") and param in getattr(
self.llm, "model_kwargs", {}
):
self.original_params[param] = self.llm.model_kwargs[param]
self.llm.model_kwargs[param] = valuebecause the following line is returning None: getattr(self.llm, "model_kwargs", {})b) Importing Hugging Face Hub Models ExternallyWe can also use custom wrappers to fetch models externally, similar to how it's done here. However, when we do so with Hugging Face Hub models, we get the following error: I believe the code attempts to use the LLM with 2) GPT4Alla) Built-inIt looks like the issue here is related to how the model path is being passed to langchain. It looks like the Configuration: models:
- type: main
engine: gpt4all
model: gpt4all-j-v1.3-groovy
# (I also tried by using the path to the model)
# model: ./models/ggml-gpt4all-l13b-snoozy.binError: b) ExternalThe error is below. Not too sure what this one is about. Error: To conclude, I think all problems mentioned here are worth of having their own issue but that might not be necessary if you are already working on it and it's close to completion. Thanks! |
|
Any progress? |
I also need the same...running Guardrail using local LLM |
Same here... Any progress? |
A same kind of issue is been solved over here /issues/155 |
This issue somewhat overlaps with #27. However, I've chosen to create this issue because it was mentioned in the thread of #27 that support for other LLM models would be added by the end of May. Support was seemingly added with commit e849ee9, but I am unable to use several of the engines due to various bugs.
NeMo-Guardrails doesn't seem to work with the following engines:
huggingface_pipelinehuggingface_textgen_inferencegpt4allI am not confirming whether it works properly with any other engines. I have only tested with these three engines and failed to interact with the model for each engine.
I have included my configuration and output for
gpt4allonly. Attempting to use the other two engines above also causes similar issues. If you are able to construct a chatbot with guardrails using any of these engines, please let me know and I will re-evaluate.Here are my configurations and code:
./config/colang_config.co:./config/yaml_config.yaml./demo_guardrails.pyAfter running
demo_guardrails.pywith the above configurations, I receive the following output:It seems that the code fails when initializing the GPT4All model. Please let me know if I've missed anything. Thanks!
The text was updated successfully, but these errors were encountered: