Add support for Simple ReRanker or similar locally hosted version

[kalaspuffar]

I've been playing around with locally hosting a version of LibreChat, and to get it really useful, a web search feature was a requirement. I've hosted SearXNG and FireCrawl locally. Also ran Jina to do the reranking. But I could not find a locally hosted open source solution for reranking. So I set out to create one.

I found this repository:
https://github.com/AnswerDotAI/rerankers

And created a small wrapper Docker image api for it:
https://github.com/kalaspuffar/simple-reranker

It should pretty much be a slot-in solution alongside Jina and Cohere.

This was the code I tried with and never got to work:

export class SimpleReranker extends BaseReranker {
  constructor({
    instanceUrl = process.env.SIMPLE_RERANKER_HOST,
    logger,
  }: {
    instanceUrl?: string;
    logger?: t.Logger;
  }) {
    super(logger);
    this.instanceUrl = instanceUrl;
  }

  async rerank(
    query: string,
    documents: string[],
    topK: number = 5
  ): Promise<t.Highlight[]> {
    this.logger.debug(`Reranking ${documents.length} chunks with SimpleReranker`);

    if (this.instanceUrl == null || this.instanceUrl === '') {
      this.logger.warn('SIMPLE_RERANKER_HOST is not set. Using default ranking.');
      return this.getDefaultRanking(documents, topK);
    }

    try {
      const requestData = {
        config: {
          model_name: process.env.SIMPLE_RERANKER_MODEL_NAME,
          model_type: process.env.SIMPLE_RERANKER_MODEL_TYPE,
          lang: process.env.SIMPLE_RERANKER_LANG,
          api_provider: process.env.SIMPLE_RERANKER_API_PROVIDER,
          api_key: process.env.SIMPLE_RERANKER_API_KEY
        },
        data: {
          query: query,
          docs: documents,
          top_k: topK,
        }
      };

      return await axios.post(
        this.instanceUrl,
        requestData,
        {
          headers: {
            'Content-Type': 'application/json',
          },
        }
      );

    } catch (error) {
      this.logger.error('Error using Simple reranker:', error);
      // Fallback to default ranking on error
      return this.getDefaultRanking(documents, topK);
    }
  }
}

I'm willing to help out in the development process and share any of my work with the community. What you have done so far is great, and I'm looking forward to working with you.

Best regards
Daniel

[danny-avila]

Thanks @kalaspuffar we were thinking of adding something like to the RAG API, feel free to make a PR there:
https://github.com/danny-avila/rag_api

[kalaspuffar]

@danny-avila I've now pushed 3 PRs with changes in rag_api, agents, and Librechat

LibreChat-AI/rag-api#190
LibreChat-AI/agents#33
#10574

Also had to do a strange change to get my respository to work

-    const toolInstance = toolFn(async (...args) => {
-      return tool['_call'](...args);
+    const toolInstance = toolFn(async (input, config) => {
+      // Use the public 'invoke' method instead of internal '_call'
+      return tool.invoke(input, config);
     }, toolDefinition);

Might be something that is solved later. Realized that I run everything in Node 22 so I'll look into switching to Node 20.

[kalaspuffar]

Did not matter, still 6k linting errors on a newly cloned repository.

npm run lint
.....

✖ 6234 problems (6083 errors, 151 warnings)
  5855 errors and 8 warnings potentially fixable with the `--fix` option.

node --version
v20.19.5

[danny-avila]

Will be reviewing soon, thanks for your patience

[kalaspuffar]

Will be reviewing soon, thanks for your patience

Very much appreciated, I understand that the project has a lot of work going on and I can wait my turn.

[JumpLink]

@kalaspuffar thank you very much for your work!

I would be very happy if the feature were merged, as we ourselves are interested in hosting as much as possible ourselves and are now at the same stage with regard to web search, where only the reranker is still missing.

Is assistance required here? For example, testing of the PRs for this?

[kalaspuffar]

Hi @JumpLink

Any testing you can provide is much appreciated. All critic or suggestions are welcome. I want to keep this simple so we can get it merged but I'm open to suggestions.

Best regards
Daniel

[JumpLink]

@kalaspuffar I was able to test it now and it works, but mixedbread-ai/mxbai-rerank-large-v1 was clearly too large for my local machine without an NVIDIA GPU..

Our goal is to use as much open source and as many locally hostable models and services as possible, but we will probably have to rely on an AI provider for the time being. We are currently using OpenRouter for testing purposes, but would prefer to use a provider from Europe and are therefore looking for a way to easily switch providers without having to implement the provider directly in LibreChat. And when I look at https://github.com/AnswerDotAI/rerankers/blob/main/rerankers/reranker.py#L7, it seems that the language parameter can certainly be applied to other providers.

Therefore, it would be good for us to be able to specify a provider for reranking, but I understand that you want to keep the PR as simple as possible for now.

You can find the configuration I used to test the PRs here: faktenforum/ai-chat-interface#1

I may test smaller models as well. Do you have any suggestions for achieving good results without a GPU?

[JumpLink]

@kalaspuffar The FlashRank models seem to work quite well for me, so perhaps these would be more suitable default models that should work in many setups?

[kalaspuffar]

Hi @JumpLink

Sounds good, I'll try something like

- `SIMPLE_RERANKER_MODEL_NAME` (Optional) defaults to `ms-marco-MiniLM-L-12-v2`, more options at (https://github.com/AnswerDotAI/rerankers)
- `SIMPLE_RERANKER_MODEL_TYPE` (Optional) defaults to `flashrank`, more options at (https://github.com/AnswerDotAI/rerankers)

reranker_instance = Reranker(
    model_name=os.getenv("SIMPLE_RERANKER_MODEL_NAME", "ms-marco-MiniLM-L-12-v2"),
    model_type=os.getenv("SIMPLE_RERANKER_MODEL_TYPE", "flashrank"),
)

Best regards
Daniel

[kalaspuffar]

Defaults are now changed.

[rio100]

+1
I also see the need to be able to provide a local reranker of our choice to avoid vendor lock-in and an easier way to manage preferred rerankers. Would be nice to have a local option one as done with searxng and firecrawl. Of importance is to provide an agnostic approach of which @kalaspuffar has done with integrating community supported AnswerDotAI's reranker libraries.

AnswerDotAI's rerankers library which provides:

1.)Unified API for multiple backends
FlashRank (local, ONNX-optimized, CPU-friendly)
pypi.org
BGE Reranker (multilingual, open-source)
Jina API
Cohere API
Cross-encoders

2.)Lightweight dependencies and easy installation

3.)Community maintained with active development

[ucodia]

Search reranking can be a fairly low-compute task and CPU-friendly, so adding local reranker support would make LibreChat an even stronger open-source stack by removing the need for an external provider. In my case, my stack already uses a cross-encoder reranker for a custom RAG pipeline and it runs without any trouble on CPU alone with a dozen users.