Please add option for a local reranking model!

[exito100]

I would think many people would want to use a fully private non-naive RAG setup. So why not add teh option to use Ollama for the reranking model too?

There is at least on GGUF reranker that Ollama supports: epoch/rank-zephyr. I have tried many platforms to get a quality RAG system working and Kotaemonm comes closest so far but is lacking the local reranker. Perhaps another model that is smaller would also be a good option for people with resource issues.

[taprosoft]

Temporary done by utilizing TEI endpoints.

[tknrych]

FYI. Thank you very much. I tried the TEI reranker feature yesterday. The reranker server is working on another server (actually, another container). The new feature seems working fine, but is_truncated option doesn't work at this moment. As a workaround, I submitted one pull-request that can truncate string lenght according to max_tokens parameter.

[taprosoft]

Waiting for ollama/ollama#7219

[vap0rtranz]

Given the flurry of issues after Ollama team released v4, I worry that a private/local option may take awhile.

When you say TEI endpoints, are there Cohere compliant APIs available? The only way I see to do this today is to point the Cohere Reranker setup in the app to an API ...