huggingface text-generation-inference support?

[sioan]

We're standing up a local hf text gen service ( it's pretty good )
https://github.com/huggingface/text-generation-inference

Griptape already has HuggingFaceHubPromptDriver, HuggingFacePipelinePromptDriver, could you add one that can interface with locally hosted huggingface text-generation-inference?

Here's the alternative
https://python.langchain.com/docs/modules/model_io/models/llms/integrations/huggingface_textgen_inference

Thank you for developing griptape! It looks like it's gonna be awesome!

[maziyarpanahi]

@collindutter this would be very useful as most productions have their own TGI/vLLM deployments and I guess they are pretty similar in principal (same as OpenAI and other endpoints).

[collindutter]

@maziyarpanahi roger that! I will take a look at implementing this today!

[collindutter]

@maziyarpanahi it looks like you can use TGI through the Inference Client which is already a part of HuggingFaceHubPromptDriver. Do you have an existing TGI endpoint you could help us test with?

[maziyarpanahi]

HuggingFaceHubPromptDriver

This is great! I have TGI endpoints up and running. I can test any example you provide and share the results.
Thanks again.

[collindutter]

Fantastic, thank you for reporting back! I will add a note to the the docs indicating that HuggingFaceHubPromptDriver can be used with TGI!