What would you like to see?
Hello AnythingLLM Team,
I would like to request support for using locally deployed reranker models through Ollama.
Motivation:
Improve Chinese Retrieval Quality: Many users, especially in the Chinese-speaking community, rely on powerful models like Qwen for better contextual understanding. The latest Qwen3 models have excellent reranker versions that can significantly improve RAG performance.
Simplify Local Deployment: Ollama has become an extremely popular and simple way to run these models locally. Integrating it for reranking would lower the barrier for users to significantly improve their RAG quality.
Proposed Solution:
In the workspace settings, under a "Reranking" section, please add an option to select "Ollama" or a generic "Custom Endpoint".
When selected, this would allow users to enter:
The Ollama API URL (e.g., http://localhost:11434)
The Model Name (e.g., Qwen3-Reranker-0.6B)
This would enable AnythingLLM to send the document chunks to the local Ollama instance for reranking before feeding them to the LLM.
This feature would greatly enhance the flexibility and performance of AnythingLLM for a global user base. Thank you for your great work and for considering this request.
What would you like to see?
Hello AnythingLLM Team,
I would like to request support for using locally deployed reranker models through Ollama.
Motivation:
Improve Chinese Retrieval Quality: Many users, especially in the Chinese-speaking community, rely on powerful models like Qwen for better contextual understanding. The latest Qwen3 models have excellent reranker versions that can significantly improve RAG performance.
Simplify Local Deployment: Ollama has become an extremely popular and simple way to run these models locally. Integrating it for reranking would lower the barrier for users to significantly improve their RAG quality.
Proposed Solution:
In the workspace settings, under a "Reranking" section, please add an option to select "Ollama" or a generic "Custom Endpoint".
When selected, this would allow users to enter:
The Ollama API URL (e.g., http://localhost:11434)
The Model Name (e.g., Qwen3-Reranker-0.6B)
This would enable AnythingLLM to send the document chunks to the local Ollama instance for reranking before feeding them to the LLM.
This feature would greatly enhance the flexibility and performance of AnythingLLM for a global user base. Thank you for your great work and for considering this request.