[Task] text-ranking model support

[DingmaomaoBJTU]

Overview

Text ranking (a.k.a. cross-encoder reranking) takes a query and a set of candidate passages and scores their relevance. Unlike bi-encoder retrieval which embeds query and documents independently, a cross-encoder jointly encodes the pair for higher accuracy. This is typically the final scoring step in a retrieval-augmented generation (RAG) pipeline.

This is primarily a pipeline feature: cross-encoder reranking models can be built on top of existing text feature extraction support. The implementation involves adding a text-ranking pipeline class that wraps a sequence-classification model and accepts (query, [passages]) as input.

Agent Scenarios

• RAG reranking agent: after BM25 or vector retrieval returns a candidate pool, rerank with a cross-encoder before passing top-k to an LLM for generation
• Enterprise search agent: improve precision of document retrieval over internal wikis or code bases by reranking BM25 hits
• Question answering agent: select the most relevant passage from a retrieved set before extracting or generating an answer
• Recommendation agent: score user query against candidate items (product descriptions, articles) and surface the highest-relevance results

ModelKit Integration

wmk config → wmk build (ONNX export) → wmk perf → wmk eval

Requires implementing the text-ranking pipeline in ModelKit before model onboarding can begin.

Acceptance Criteria

• Implement text-ranking pipeline (cross-encoder scoring of query–passage pairs)
• Validate with top text-ranking models (>2k downloads on HuggingFace)

[xieofxie]

Mostly qwen https://huggingface.co/models?other=text-ranking