[FEAT]: Support Ktransformers as LLM

[WaterYue]

What would you like to see?

When will AnythingLLmm support ktransformers, the ktransformers can let the AI use less hardware to complete high quantity answers!!!

Here is the link:
https://github.com/kvcache-ai/ktransformers

[timothycarambat]

If this is just another inference service that is OpenAI Compatible, you can just use the base URL and connection information in the Generic OpenAI Connector for the LLM and it will work the same.

[jiazhen-code]

I am currently trying to run KTransformer (Deepseek-R1) + Generic OpenAI Connector with AnythingLLM on an Apple M1 computer. While the server shows normal responses, the client keeps getting stuck at the last returned character until I manually click the pause button, after which the final character is displayed.

[timothycarambat]

Sounds like KTransformers does not return a stop reason - thus why the connection stream does not close. Can you confirm if the last chunk from KTransformer returns a stop reason?

[jiazhen-code]

There is a stop reason like this.

[jiazhen-code]

I'm sure it's because there is no finish_reason in the last chunk. I simply fix it by adding the reason in the last chunk as follows (in ktransformers/server/schemas/assistants/streaming.py):

[zhugh2333]

does it work?

[timothycarambat]

That should work finish_reason is different from [DONE]. [DONE] is for Server side event streaming completion while finish_reason is for response streaming conclusion.

anything-llm/server/utils/AiProviders/genericOpenAi/index.js

Lines 312 to 316 in fab7403

	if (
	message?.hasOwnProperty("finish_reason") && // Got valid message and it is an object with finish_reason
	message.finish_reason !== "" &&
	message.finish_reason !== null
	) {