Repository navigation
Exploring a Lightweight Multi-Stage Ranking Pipeline for DonSeTch #170
adaaaaaaaaaaaaaaaaaaaaaa
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi, I’ve been experimenting with a different ranking pipeline for DonSeTch and wanted to share the direction before pushing it further.
The idea came from the current ranking path. I think the existing approach makes sense, but I’m not completely satisfied with relying so heavily on the ONNX reranker, especially when it has to evaluate a relatively large candidate set. It works, but I think there may be room to go one step further both in ranking quality and in how we spend that compute.
The direction I’m testing is roughly:
high-recall candidate union → LambdaMART learning-to-rank → set-aware evidence selection
The candidate stage combines several cheap retrieval signals so useful passages are less likely to be discarded too early.
LambdaMART then learns how to combine lexical, structural and document-level signals instead of relying on one fixed relevance formula.
The final set-selection step is meant to solve another problem I’ve been looking at: the highest-ranked passages are often individually relevant but redundant with each other, while a slightly lower-ranked passage may contain an important complementary condition or exception. So the selector considers the evidence already chosen rather than ranking every passage completely independently.
The results so far are promising enough that I think the direction is worth exploring further.
I’m also looking at what this could mean for the existing ONNX reranker.
One possibility is replacing it if the lightweight pipeline eventually proves robust enough. But the option I currently find more interesting is using both together: let the cheaper ranking pipeline reduce and organize the candidate set first, then run the ONNX reranker only on the smaller subset where its semantic capabilities are actually useful.
So instead of asking the neural reranker to process everything, we could potentially use something like:
candidate retrieval → lightweight learned ranking / selection → ONNX reranking on a much smaller difficult subset
That introduces a more sophisticated ranking stage, but at the same time could reduce the pressure on the more expensive existing reranker when it is enabled.
I’m still validating this carefully, especially around regressions where an important condition or exception can disappear even if the average ranking score improves. I’m also keeping source-grouped held-out evaluation and anti-overfitting constraints around the experiments rather than tuning rules around individual benchmark failures.
I don’t think the current ranker is fundamentally the wrong approach. My feeling is more that we can build on it and make the overall pipeline smarter about what actually deserves neural reranking, what can be handled cheaply, and which pieces of evidence should be delivered together.
Curious what you think about this direction, especially whether you’d see this fitting better as a replacement for the current reranker or as a lightweight stage in front of it.
All reactions