Anthropic Claude Support

[corysabol]

Hello, I see from some of the examples that Kalosm supports cloud hosted model inference. The example shows the use of OpenAI GPT4.
This is great, but in my application I also need to support the use of Anthropic Claude.

My question is whether or not this is coming, already supported and I missed something, or if I can be pointed to the right place to contribute support for this myself, which I am happy to do.

For a little more context my application supports local and cloud inference (chatgpt and claude). I was initially building my own inference frame work for this based on candle, but then I found Kalosm and figured it better not to re-invent the wheel :)

[ealmloff]

#328 adds support for the Anthropic API. Anthropic doesn't support constrained generation (outside of tools), so the chat functionality is limited to just streaming text output

[corysabol]

Ah, okay. Thanks for pointing me in the right direction.
I may focus on local and openai support first then. I was only aiming for Claude as it is what I currently have an API account for.

I the lack of constrained generation due to a limitation with the Anthropic API itself, or is it just that support it has not yet been built into Kalosm?

[ealmloff]

It is a limitation of their API. It seems like they support tools within the response with constrained generation, but they don't support controlling the whole response format