Optional retrieval rerank (off by default): generic hook, Jev as one possible backend

[WiktorB2004]

Opening this as an Idea, not an issue. I will not open a PR unless you want this on the roadmap.

Summary

Would you consider an optional rerank step after vector retrieval, off by default? RAG chat could keep a larger retrieval.top_k and only send the most relevant chunks to the LLM.

I maintain llama-index-postprocessor-jev (JevRerank), a LlamaIndex BaseNodePostprocessor. I am saying that so it is clear I am the package author. This is not a request to make TypeSafe required.

Why

Embedding similarity often returns chunks that are about the topic but do not answer the question. Rerank is retrieve-many, keep-few.

PrivateGPT already retrieves with retrieval.top_k (default 32). Older settings had optional local rerank:

rag:
  rerank:
    enabled: false
    model: cross-encoder/ms-marco-MiniLM-L-2-v2
    top_n: 1

That block is gone on current main. Contextual chat still has a node_postprocessors list (for example MetadataReplacementPostProcessor), so a reranker can be appended when enabled.

What I have in mind

Not a required dependency. Not on by default.

A generic rerank hook: after retrieval, optionally run a LlamaIndex postprocessor, then continue. Settings pick the backend. Default is none. Jev would be one implementation of that slot, not the feature itself.

retrieval:
  top_k: 32
  rerank:
    enabled: false
    backend: none           # none | local | jev
    top_n: 5

backend	What runs
none	Nothing (today)
local	On-machine cross-encoder (old rag.rerank)
jev	JevRerank, extra/optional package, API key from env, never committed

If rerank errors, keep the original retrieval order (truncated). Do not drop context. Fail open.

Default install and air-gapped / offline_mode stay unchanged.

If you only want none and local, that works. Hosted Jev can stay out of this repo.

Local path on the package (not this discussion)

Jev is hosted today (provider="typesafe" or "openrouter"). The package already switches transport with provider=... and the same typed answers (Score / Choice / Noul).

If a suitable open decision model appears (same return types, not necessarily TypeSafe weights), later package releases will add provider="local" so rerank can run on-machine with no TypeSafe or OpenRouter call.

For PrivateGPT, the same settings flag could later point at local inference. I am not asking you to wait for that. There is no local model I would ship yet as matching hosted Jev calibration.

Out of scope

• Making TypeSafe or OpenRouter required
• Changing default retrieval
• Router / tool-selection
• Asking you to wait for provider="local"
• Treating this as a bug or a commitment to implement

What I need from you

PrivateGPT is built for local/private models. Hosted Jev may be a no. Useful answers:

1. Yes to a generic hook (local and/or jev). I can PR.
2. Local rerank only. I can drop hosted Jev from the proposal.
3. Not on the roadmap. I will leave it (fork if I still want it).

Offer

If you want it, I can PR: settings, optional extra, tests with a mocked client (no live API in CI), short docs note. I will wait for a yes or no.

References

• Package: https://pypi.org/project/llama-index-postprocessor-jev/
• Docs: https://wiktorb2004.github.io/llama-index-jev/