Self hosted LLMs Support #263

gsaivinay · 2023-05-09T10:08:23Z

Hello,

Thanks for this awesome work.

Is there any support for custom self hosted LLMs? Like I host multiple models in AWS EC2 instances using https://github.com/huggingface/text-generation-inference. If so, could you please point me to example.

Happy to help or contribute regarding this if not already exists.

TMRolle · 2023-05-12T17:09:14Z

Seconded, even just support for HuggingFacePipeline LLM's would be really useful

ogabrielluiz · 2023-05-12T19:08:47Z

Hey all!
I completely agree. I had a hard time testing them and thought the problem was due to streaming which is supported now so maybe all give it another go.

Feel free to try it out too. We might add the pipeline to dev just to set it up for dev testing.

ogabrielluiz · 2023-05-13T11:20:29Z

I'll open an issue where we'll track the missing modules for each type starting with LLMs.

pounde · 2023-05-16T12:03:40Z

Thanks for the consideration. Big +1 here. Thanks!

ogabrielluiz · 2023-05-16T12:23:47Z

Should it be SelfHostedHuggingFaceLLM or HuggingFaceTextGenInference?

…FaceLLM to llms list in config.yaml fix(langflow): fix import of SUFFIX_WITH_DF in custom.py refactor(langflow): refactor LLMCreator to import llms and chat_models modules and create type_to_loader_dict from them feat(langflow): add inference_server_url, max_new_tokens, top_k, top_p, typical_p, temperature, and repetition_penalty fields to LLMFrontendNode and show them in the advanced section Issue #263

ogabrielluiz · 2023-05-16T12:47:22Z

I've added both of them in this branch but I don't think implementing SelfHostedHuggingFaceLLM will be trivial as it seems it needs some Runhouse objects that are not inside LangChain.

What do y'all think?

Could you take it for a spin to see what breaks?

pounde · 2023-05-16T12:50:45Z

You'll have to forgive my ignorance but I'm just getting up to speed on hosting. I created a simple API to implement Dolly using the HuggingFacePipeline in langchain. In the one minute look, I think that's more akin to SelfHostedHuggingFaceLLM as it's hosted entirely on our own system. Perhaps someone with more experience in this domain has some light to shed on that. I'll try to carve some time to try that branch.

ogabrielluiz · 2023-05-16T13:21:52Z

SelfHostedHuggingFaceLLM seems to require Runhouse to be set up and to pass a hardware object of some kind.

We could (and probably should) implement that but then we'd have to define a maintainable way of doing so.

gsaivinay · 2023-05-16T13:31:37Z

I've actually contributed to HuggingFaceTextGenInference, usually we use this server on local machines or Cloud service like AWS EC2, to which we can connect via the API.

If the local machine is able to run a model with SelfHostedHuggingFaceLLM then mostly it can also run the same model withHuggingFaceTextGenInference. Since the latter gives an API to interact with, it'll be easy to use in multiple applications.

feat(langflow): add support for HuggingFacePipeline in loading.py feat(langflow): add model_id field to LLMFrontendNode's SHOW_FIELDS list Issue #263

pounde · 2023-05-16T14:50:54Z

@ogabrielluiz -- I built the dockerfile in the branch and I'm not seeing the self-hosted models listed. Has that not caught up to the backend perhaps?

ogabrielluiz · 2023-05-16T15:06:11Z

@pounde in the config.yml there's this section:

llms:
  - OpenAI
  # - AzureOpenAI
  - ChatOpenAI
  - HuggingFaceHub
  - LlamaCpp
  - HuggingFaceTextGenInference
  - SelfHostedHuggingFaceLLM
  - HuggingFacePipeline

Theoretically all of these should show up but there could be a bug preventing one of them of showing in the frontend.
Since SelfHostedHuggingFaceLLM requires runhouse, maybe we should focus on @gsaivinay HuggingFaceTextGenInference and HuggingFacePipeline.

pounde · 2023-05-16T16:37:41Z

@ogabrielluiz -- sure enough. It's in my config.yaml. So luck on the frontend though. I have:

OpenAI
ChatOpenAI
LlamaCpp
HuggingFaceHub
No luck on the others.

gustavoschaedler · 2023-06-05T16:38:22Z

I've added the LLM HuggingFaceTextGenInference, locally the behavior was as expected, could you check if it's okay on your side?

dongreenberg · 2023-06-07T09:22:24Z

Hey folks, just stumbled upon this. I work on Runhouse - you're correct that the HFTextGen LLM can offer the same functionality for an optimized set of models and a relatively simple setup for access to the server, whereas the SelfHosted models via Runhouse can support any model and a more flexible set of compute (e.g. launching automatically on any cloud), but without automatically handling distribution and model-specific optimizations. The increased flexibility is particularly important in enterprise, but I'm not sure if that's your target userset? I'm happy to help if you're interested in supporting that use case. If you're mainly focused on local compute with a specific set of models HFTextGen should be totally fine.

ogabrielluiz · 2023-06-08T03:30:37Z

Hey, @dongreenberg.
Thanks for reaching out. Runhouse's solution fits very well into our plans.
We'd have to build a new way of setting up models to work with Runhouse and help is definitely appreciated.

Please let me know if I can assist you with anything.

toby-lm · 2023-06-30T07:06:15Z

I've added the LLM HuggingFaceTextGenInference, locally the behavior was as expected, could you check if it's okay on your side?

I've tried building from the 263-self-hosted-llms-support branch and I can't seem to get any response in the chat window that pops up. I can connect it to my local text-generation-inference API, and there are responses in the browser developer console but no text appears as a reply. Can you show how this should work please?

EDIT: I get a response correctly if it's just the LLM node. Once I connect it to a ConversationChain it didn't display the LLM reply (but still received it).

2good4hisowngood · 2023-07-08T00:07:11Z

Just to throw another option out there, LangChain supports Ooobabooga's TextGen Web API but it's not in LangFlow yet. In my experience testing different tools, it's one of the most consistently functional and improving locally hosted options for running models and using Nvidia GPUs. Many tools default to the CPU and require advanced setup efforts. TextGen has a one click installer that helps configure it for your system. They also quickly adopt new features like ExLlama to increase token rates. It has many advanced options configurable through the launch flags, or through the ui, allowing users to modify the configurations, and retest to validate if their changes improve performance on their specific machine, rather than trying to shoehorn a non-complete feature list into langchain arguments.

This example goes over how to use LangChain to interact with LLM models via the text-generation-webui API integration. Please ensure that you have text-generation-webui configured and an LLM installed. Recommended installation via the one-click installer appropriate for your OS. Once text-generation-webui is installed and confirmed working via the web interface, please enable the api option either through the web model configuration tab, or by adding the run-time arg --api to your start command.

LangChain Page for TextGen: https://python.langchain.com/docs/modules/model_io/models/llms/integrations/textgen
GitHub Page: https://github.com/oobabooga/text-generation-webui/tree/main

RandomInternetPreson · 2023-07-09T05:50:25Z

I've added the LLM HuggingFaceTextGenInference, locally the behavior was as expected, could you check if it's okay on your side?

I don't know if you are still working on this but I would really like to try it out! Being able to use langflow with oobabooga would be amazing!!

I found the repo you made here: https://github.com/logspace-ai/langflow/tree/263-self-hosted-llms-support

But I don't know how to install it. I Can install the current version of langflow with pip install langflow, but I'm not sure how to install your version.

I would give you feedback on the branch if you could tell me how to install it. Seriously langflow with oobabooga would be amazing!!!

pounde · 2023-07-09T07:58:11Z

Still no luck on my end. I do have additional options now but no TextGenInterface.

thomclae33 · 2023-08-02T19:09:09Z

Any news on this? Would love to know how to use the Textgen API with Langflow.

Jirito0 · 2023-08-27T01:56:28Z

Also looking for an update on this please! Been scouring the entire internet for a solution but cant find anything

vvlEURO · 2023-09-02T16:35:05Z

Its work in 0.5.0a0 version

tonypius · 2023-09-11T14:36:49Z

Hey, would really like an update on this feature for LLM HuggingFaceTextGenInference

Also, is the custom component a good workaround ?

ogabrielluiz · 2023-09-11T17:14:55Z

hey!
The CustomComponent is a good workaround and we've added the Hugging Face Inference API component last week.

Can it be used in place of the TextGen?

tonypius · 2023-09-11T18:15:00Z

HuggingFaceTextGenInference is different from Hugging Face Inference API right ?

m1ll10n · 2023-09-14T09:56:25Z

Its work in 0.5.0a0 version

I wonder if the whole reason this thing work is due to different langchain versions as Im facing a "streaming option currently unsupported" issue in both 0.4.17 and 0.4.18

vvlEURO · 2023-10-06T16:29:35Z

Its work in 0.5.0a0 version

I wonder if the whole reason this thing work is due to different langchain versions as Im facing a "streaming option currently unsupported" issue in both 0.4.17 and 0.4.18

Yes, related. Support for langchain where streaming is added starts with langflow 0.5.0a0.

stale · 2023-11-20T16:44:05Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ogabrielluiz mentioned this issue May 13, 2023

LangChain LLMs support #290

Closed

8 tasks

ogabrielluiz self-assigned this May 16, 2023

ogabrielluiz added the feature request label May 16, 2023

ogabrielluiz added a commit that referenced this issue May 16, 2023

feat(langflow): add HuggingFacePipeline to list of supported LLMS

c6bd44f

feat(langflow): add support for HuggingFacePipeline in loading.py feat(langflow): add model_id field to LLMFrontendNode's SHOW_FIELDS list Issue #263

gustavoschaedler added a commit that referenced this issue Jun 5, 2023

Add LLM HuggingFaceTextGenInference (#263)

7952c14

gustavoschaedler added a commit that referenced this issue Jun 5, 2023

Add LLM HuggingFaceTextGenInference (#263)

ecd4dc0

dewankpant mentioned this issue Aug 12, 2023

[Snyk] Upgrade tailwind-merge from 1.13.0 to 1.14.0 dewankpant/langflow#3

Open

stale bot added the stale label Nov 20, 2023

stale bot closed this as completed Nov 27, 2023

dosubot bot mentioned this issue Dec 28, 2023

trouble calling hosted model in langflow #1258

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Self hosted LLMs Support #263

Self hosted LLMs Support #263

gsaivinay commented May 9, 2023

TMRolle commented May 12, 2023

ogabrielluiz commented May 12, 2023

ogabrielluiz commented May 13, 2023

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023

ogabrielluiz commented May 16, 2023

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023

gsaivinay commented May 16, 2023 •

edited

Loading

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023 •

edited

Loading

pounde commented May 16, 2023

gustavoschaedler commented Jun 5, 2023

dongreenberg commented Jun 7, 2023 •

edited

Loading

ogabrielluiz commented Jun 8, 2023

toby-lm commented Jun 30, 2023 •

edited

Loading

2good4hisowngood commented Jul 8, 2023 •

edited

Loading

RandomInternetPreson commented Jul 9, 2023

pounde commented Jul 9, 2023

thomclae33 commented Aug 2, 2023

Jirito0 commented Aug 27, 2023

vvlEURO commented Sep 2, 2023 •

edited

Loading

tonypius commented Sep 11, 2023

ogabrielluiz commented Sep 11, 2023

tonypius commented Sep 11, 2023

m1ll10n commented Sep 14, 2023

vvlEURO commented Oct 6, 2023 •

edited

Loading

stale bot commented Nov 20, 2023

Self hosted LLMs Support #263

Self hosted LLMs Support #263

Comments

gsaivinay commented May 9, 2023

TMRolle commented May 12, 2023

ogabrielluiz commented May 12, 2023

ogabrielluiz commented May 13, 2023

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023

ogabrielluiz commented May 16, 2023

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023

gsaivinay commented May 16, 2023 • edited Loading

pounde commented May 16, 2023

ogabrielluiz commented May 16, 2023 • edited Loading

pounde commented May 16, 2023

gustavoschaedler commented Jun 5, 2023

dongreenberg commented Jun 7, 2023 • edited Loading

ogabrielluiz commented Jun 8, 2023

toby-lm commented Jun 30, 2023 • edited Loading

2good4hisowngood commented Jul 8, 2023 • edited Loading

RandomInternetPreson commented Jul 9, 2023

pounde commented Jul 9, 2023

thomclae33 commented Aug 2, 2023

Jirito0 commented Aug 27, 2023

vvlEURO commented Sep 2, 2023 • edited Loading

tonypius commented Sep 11, 2023

ogabrielluiz commented Sep 11, 2023

tonypius commented Sep 11, 2023

m1ll10n commented Sep 14, 2023

vvlEURO commented Oct 6, 2023 • edited Loading

stale bot commented Nov 20, 2023

gsaivinay commented May 16, 2023 •

edited

Loading

ogabrielluiz commented May 16, 2023 •

edited

Loading

dongreenberg commented Jun 7, 2023 •

edited

Loading

toby-lm commented Jun 30, 2023 •

edited

Loading

2good4hisowngood commented Jul 8, 2023 •

edited

Loading

vvlEURO commented Sep 2, 2023 •

edited

Loading

vvlEURO commented Oct 6, 2023 •

edited

Loading