Add Function Score Query

[alex-au-922]

The implementation of FunctionScoreQuery which accepts an existing query and a closure for applying new scoring. You may consider this query as a .map() function which does not filter the documents out, but rather change their score.

Closes #2395

[adamreichold]

How you would say this relates to the existing tweak_score? Would that also suffice to handle your use case?

[adamreichold]

Ah sorry, you wrote that in the linked issue...

[alex-au-922]

Ah sorry, you wrote that in the linked issue...

Sorry that I'm not quite used to contribute to large OSS projects so I'm not sure the exact practices on linking Issues and PR. Will add the issue link in the PR next time

[adamreichold]

Regarding the formatting errors, note that you'll need to use a nightly toolchain to apply the correct formatting configuration.

[fulmicoton]

Can you implement the other methods too? seek for instance, has a default impemnetation but it will be catastrophically slow.

[fulmicoton]

If your fucntion needs to access a column from a fast field, you don't want to access it at each call.

The function should be a factory that takes a SegmentReader and returns a scoring function

[alex-au-922]

I have removed the use of SegmentReader in the current version of FunctionScoreQuery as the query could be too versatile that users cannot optimize their query performance with the wrong use of SegmentReader.

The use of SegmentReader should be thoroughly discussed. We can even just limit them with the use of fastfield only.

The current signature should only be a simple Fn(Score) -> Score which can simplify complicated nested boosting cases and some impossible maths like reciprocal, logarithm etc..

[adamreichold]

The use of SegmentReader should be thoroughly discussed. We can even just limit them with the use of fastfield only.

I think @fulmicoton's proposed approach is very right on target and has precedence in other Tantivy API, i.e. make the user pass a fallible factory function which given a segment reader will produce an infallible per-document scoring function.

[alex-au-922]

I think @fulmicoton's proposed approach is very right on target and has precedence in other Tantivy API, i.e. make the user pass a fallible factory function which given a segment reader will produce an infallible per-document scoring function.

What comes up in my mind is a new enum like

pub enum ScoreFunction {
    ScoreOnly(Arc<dyn Fn(Score) -> Score + Send + Sync + 'static>),
    ScoreWithDetails(Arc<dyn Fn(&SegmentReader, Score, DocId) -> Score + Send + Sync + 'static>),
}

pub struct FunctionScoreQuery {
    query: Box<dyn Query>,
    score_function: ScoreFunction,
}

so that we can match different types of ScoreFunction. However, this cannot prevent users from using the ScoreWithDetails when they don't need the SegmentReader at all.

If we accept users to dynamically use the segment reader based on different conditions, we have to include it in the closure input parameter and in the end each call will still calling the segment reader. Or do you have any suggestions?

[adamreichold]

The idea is not allow the scoring function access to the segment reader but only to the relevant column itself, because after that column has been loaded, access to the values becomes essentially infallible.

type ScoreFunction = Arc<dyn Fn(DocId, Score) -> Score + Send + Sync + 'static>;

type ScoreFunctionFactory = Arc<dyn Fn(&SegmentReader) -> crate::Result<ScoreFunction> + Send + Sync + 'static>;

(The score function will access the column retrieved from the segment reader in the factory by capturing it.)

[alex-au-922]

Yes, that makes sense by utilizing the type checker system. However, I'm still thinking how can we be benefited with not accessing the segment reader by every call, as @fulmicoton mentioned:

If your fucntion needs to access a column from a fast field, you don't want to access it at each call.

I think it's better to separate two structs as FunctionScoreQuery (only dealing with the score) and a FunctionSegmentQuery (accessing the segment reader)?