Repository navigation
Conversation
Fixed tests with dummy `|_| 1.0` scorer
# Conflicts: # src/query/automaton_weight.rs # src/query/fuzzy_query.rs # src/termdict/fst_termdict/termdict.rs
Codecov Report
@@ Coverage Diff @@
## main #1244 +/- ##
=======================================
Coverage ? 94.10%
=======================================
Files ? 206
Lines ? 35072
Branches ? 0
=======================================
Hits ? 33003
Misses ? 2069
Partials ? 0 Continue to review full report at Codecov.
|
|
I'll run rustfmt on this in a moment |
|
|
||
| /// A weight struct for Fuzzy Term and Regex Queries | ||
| pub struct AutomatonWeight<A> { | ||
| pub struct AutomatonWeight<A, F> { |
There was a problem hiding this comment.
This approach is not good...
The automaton weight is for regex and fuzzy queries, as mentionned above.
F has no meaning for regex queries.
Let's try something nicer.
We can have F match F: Fn(A::State) -> Score .
For better usability, we can also offer a default to the Const function that returns one.
The tricky part then, is that we want the user to pass an array over the distance. Not a function of the state.
Well, that's a job for the FuzzyQuery itself.
The FuzzyQuery has access to the automaton. It can therefore build a function that picks a state, computes the distance, and return a score.
There was a problem hiding this comment.
F has no meaning for regex queries.
In this case, it can default to some no-op function like |_| 1.0
We can have F match F: Fn(A::State) -> Score .
I don't really understand this because there would still be an F for a regex query then, wouldn't there?
For better usability, we can also offer a default to the Const function that returns one.
If F is a Fn then we cannot provide struct AutomatonWeight<A, F = <some default>> because we can't name |_| 1.0, nor implement it for a struct DefaultScorer; because that's unstable. Otherwise, a scorer trait could be added, which can then have a named default implementation, and be blanket implemented for Fn(A::State) -> Score.
There was a problem hiding this comment.
I also don't quite understand what you mean by "we want the user to pass an array over the distance, not a function of the state", if you are also suggesting Fn(A::State) -> Score. Apologies if I am misinterpreting this
There was a problem hiding this comment.
I don't really understand this because there would still be an F for a regex query then, wouldn't there?
Ah yes... :)
But there is one big notable difference.
The role of F can then be explained without necessary talking about the fuzzy term query.
because we can't name |_| 1.0
nor implement it for a struct DefaultScorer; because that's unstable
Which feature is unstable exactly?
I also don't quite understand what you mean by "we want the user to pass an array over the distance, not a function of the state", if you are also suggesting Fn(A::State) -> Score. Apologies if I am misinterpreting this
We can keep the function.
The user would give the FuzzyTermQuery an array over the distance rather than a function. It's tiny anyway.
user gives [f32; max_distance + 1]
FuzzyTermQuery takes that an transforms into a Fn(u32) -> f32,
The implementation take a state, uses the automaton to get a distance, and do a lookup in the user array.
There was a problem hiding this comment.
Which feature is unstable exactly?}
Implementing a Fn trait is unstable: rust-lang/rust#29625. This can be mitigated by making trait AutomatonScorer { fn score(A::State) -> f32 } or similar.
The user would give the FuzzyTermQuery an array over the distance rather than a function. It's tiny anyway.
So, in FuzzyTermQuery the user provides a function, and then at the AutomatonWeight boundary it is converted to a suitable F?
|
Another question: is there any way to factor in the length of the suffix of the term (when using the query as the prefix) to the score? If possible it seems like something that should be exposed. |
| A::State: Clone, | ||
| F: Fn(u8) -> f32, | ||
| { | ||
| if TypeId::of::<DfaWrapper>() == automaton.type_id() && TypeId::of::<u32>() == state.type_id() { |
There was a problem hiding this comment.
this it the code we want to get rid of.
Great question... Factoring in the length of the suffix is harder, but we can probably cook something if we dig deep in the levenshtein automata crate. I suggest we use the One case where that will give us different result is -A: target: elephant, query: eleh In this case, B will get a better score. But A could be the deletion of a character bringing us to the actual same suffix size, same levenshtein distance. |
This sounds good to me. How can we expose this to the user in such a way that they can factor it in? |
|
@Restioson a funciton is good I think.
We can cache the result for |
|
As far as caching goes, would it maybe be cleaner to let the user do their own caching? They could always wrap it with a memoisation function if they wanted to, perhaps? I'm just wondering if caching might be extraneous in a situation of just reading off a predefined const value table. |
I agree that caching should be left to the user, I imagine in most cases anyway the calculations are going to be pretty simple and easily optimised by the compiler. |
e550a98 to
84e0c75
Compare
Hi, I've tried to implement the ability to use a custom score function (
Fn(u8) -> f32) for fuzzy queries, which take the DFA distance as the input and return the score for the term. This is a proof of concept for #998, and I am opening this as a draft so that people can more easily see the diff and comment on it. If the solution is good, it could perhaps be merged into @lerouxrgd's branch first, and then #998 could be merged, or whatever people see fit.