Support for custom assistant endpoints (not Azure / OpenAI)

[bendavis78]

It's great that we're able to add custom endpoints for completion. This allows us to build our own AI chat tools w/o having to worry about the frontend because LibreChat can just hook up to your custom endpoint. Can we add support for custom-built endpoint that exposes a compatible assistants API? It seems right now we can only add OpenAI and Azure.

[danny-avila]

Already implemented: https://www.librechat.ai/docs/features/agents

You can create agents with Anthropic, AWS Bedrock, Azure, OpenAI, and custom endpoints.

Tools will only work with compatible provider/models.

Google will be added to that list soon.

[hibukipanim]

With the new Agents feature the "orchestration" and creation of the agent happens inside LibreChat.
I think that what @bendavis78 suggests (and what we'd also like to do) is to build a custom external service which behaves as an assistant (i.e. it has conversational API and can internally execute complex flows) so that LibreChat can be used as the chat interface with it, with the ability to stream the progress of the assistant run (but without the need to select tools, system-message, integrations etc. for the Agent as it happens internally in the service).

[danny-avila]

compatible assistants API

This is the main point of complication that makes the request not very feasible. The external service would either comply with completions API anyway, and thus no change is needed, or some new standard would have to be adopted. The latter is impractical, especially since I've developed my own framework now that works quite well.

I'm aware of the needs for tasks or workflows, and that's something coming down the pipeline.

[hibukipanim]

Thanks for the quick reply.
for a good experience of interacting with an AI service/agent the chat-completions API is lacking IMO:

• no way to stream non-message-delta events (like server-side tool calls, similar to run.step events in Assitant API)
• not suitable for long waits / polling / resume-streaming for quite common cases when the AI service might take few minutes to start streaming the response after running some llm-chain internally
• stateless - although it's convenient in simples cases, in more complex AI-services there is some accumulating state on the server and the client should be aware of it and send the context/thread/conversation id (which can already be done with simple extension of the chat-completions API I guess) and maybe avoid resending history every time.

maybe indeed some new standard is missing today: something between chat-completions's simplicity and assistants-api, as maybe assistants-API is too stateful/complex which is not always needed 🤔. Interestingly LangChain tried to define a framework-agnostic "agent protocol API" here: https://github.com/langchain-ai/agent-protocol and it quite resembles OpenAI's Assistant API. So implementing a compatible Assistants API is probably a good idea currently.