feat(python): Hybrid search & Reranker API

[AyushExel]

based on #713

• The Reranker api can be plugged into vector only or fts only search but this PR doesn't do that (see example - https://txt.cohere.com/rerank/)

Default reranker -- LinearCombinationReranker(weight=0.7, fill=1.0)

table.search("hello", query_type="hybrid").rerank(normalize="score").to_pandas()

Available rerankers

LinearCombinationReranker

from lancedb.rerankers import LinearCombinationReranker

# Same as default 
table.search("hello", query_type="hybrid").rerank(
                                      normalize="score", 
                                      reranker=LinearCombinationReranker()
                                     ).to_pandas()

# with custom params
reranker = LinearCombinationReranker(weight=0.3, fill=1.0)
table.search("hello", query_type="hybrid").rerank(
                                      normalize="score", 
                                      reranker=reranker
                                     ).to_pandas()

Cohere Reranker

from lancedb.rerankers import CohereReranker

# default model.. English and multi-lingual supported. See docstring for available custom params
table.search("hello", query_type="hybrid").rerank(
                                      normalize="rank",  # score or rank
                                      reranker=CohereReranker()
                                     ).to_pandas()

CrossEncoderReranker

from lancedb.rerankers import CrossEncoderReranker

table.search("hello", query_type="hybrid").rerank(
                                      normalize="rank", 
                                      reranker=CrossEncoderReranker()
                                     ).to_pandas()

Using custom Reranker

from lancedb.reranker import Reranker

class CustomReranker(Reranker):
    def rerank_hybrid(self, vector_result, fts_result):
           combined_res = self.merge_results(vector_results, fts_results) # or use custom combination logic
           # Custom rerank logic here
           
           return combined_res

• Expand testing
• Make sure usage makes sense
• Run simple benchmarks for correctness (Seeing weird result from cohere reranker in the toy example)
• Support diverse rerankers by default:
• Cross encoding
• Cohere
• Reciprocal Rank Fusion

[changhiskhan]

• Please also add tests for the rerankers (including the simple linear combo one).
• Curious to see the real world impact on search results

[prrao87]

The options for linear combination, cross encoder and cohere reranker make sense from a variety standpoint, and covers all the bases. I've seen others go down the road of Rank Fusion methods, but we can revisit that based on community feedback (though most people will likely go with cohere because it's really good quality).

The weight of 0.7 for the linear combination is a sensible default (I presume the 0.7 is for the vector component). Would love to give this a spin on some larger data and see how it fares on search relevance tasks! Performance is a whole other matter and could be improved with more efficient numpy/pyarrow ops down the line, hopefully.

[changhiskhan]

i think you need to do combine_results.take(indices[np.argsort(scores)]) ?

[AyushExel]

Oh results are already sorted in descending order of score by cohere api. Sorting is not needed. removed it

[changhiskhan]

this looks good overall but the score inversion and fill values needs to be corrected i think

[changhiskhan]

what does this do with the _distance and _score columns? does it just add nulls?

[AyushExel]

Yeah if the values are not present it adds null

[AyushExel]

For example:
fts

                                text                                             vector  score  _rowid
0  I've got a bad feeling about this  [0.016979132, -0.009463778, -0.0041340766, -0....    0.0       4

vector

          text                                             vector  _distance  _rowid
0  hello world  [-0.014907048, 0.0013432145, -0.01851529, -0.0...        0.0      10

Will result in combined:

                                text                                             vector  _distance  _rowid  score
0                        hello world  [-0.014907048, 0.0013432145, -0.01851529, -0.0...        0.0      10    NaN
1  I've got a bad feeling about this  [0.016979132, -0.009463778, -0.0041340766, -0....        NaN       4    0.0

[changhiskhan]

actually i think this won't work if the user chose to use ranks rather than scores. inverting ranks should be something like rank_inverted(i) = max(ranks) - rank(i) ?

[AyushExel]

Discussed..

[changhiskhan]

i think the fill values for vector results and fts results will be different, especially if the scores are not pre-normalized?

[AyushExel]

As discussed async the scores are always normalized currently

[AyushExel]

I think I responded to all comments.. please verify
(docs test failing on installing so shouldn't be related to this)

[changhiskhan]

yolo

[AyushExel]

okay I'm gonna bite the bullet and merge this. Ran some manual tests on open reranking datasets and the results seem sane. If you guys know some in-depth re-ranking benchmark scrips lmk and I'll run those for blog announcement.