Add score threshold into Scorer.score() [LUCENE-3364]

[asfimport]

This is an optimization for scoring.
Given a Scorer.score() implementation, where features are gathered to calculate a score.

Proposal, add a parameter to score, e.g. score(float threshold)

This threshold is the minimum score to "beat" to make it to the current PriorityQueue. This could potential save a great deal of wasted calculation in the cases where recall is large.

In our case specifically, some of the features needed to do calculation can be expensive to obtain, it would be nice to have a place to decide whether or not even fetching these features are necessary.

Also, if we know the score would be low, simply threshold can be returned.

Let me know if this makes sense and I can work on a patch.

---

Migrated from LUCENE-3364 by John Wang, updated Oct 13 2011

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Perhaps it would be easiest to just create a Collector that cut off based on score?

[asfimport]

John Wang (migrated from JIRA)

Hi Yonik:

In Collector, the decision time to cut of a score would be too late, e.g.

float score = Scorer.score(); <--- this is where the cost would occur.
boolean cutOff = decide(score)

In my example, my score impl:

float s1 = cheapCalc(docid);
float s2 = expensiveCalc(docid);

return s1+s2;

So now if I know expensiveCalc is bounded by N, and cheapCalc returns a very small number, I can simply skip s2 calculation because this doc would be thrown out anyway.

Hope I am making sense :)

-John

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Ah, gotcha - I see what you're saying now.

[asfimport]

Paul Elschot (migrated from JIRA)

To have more more advantages the minimum score could be given as a parameter to the nextDoc() and advance() methods.

[asfimport]

John Wang (migrated from JIRA)

Hi Paul:
I think adding score threshold to the nextDoc()/advance() apis would pollute the filtering part of the API which has nothing to do with scoring. I do see the spirit to allow for further skipping of docs given the score, however, a simply if check in that loop may be a hinderance to performance. So I am not sure if it is worth it.

Just my $0.02.

-John

[asfimport]

John Wang (migrated from JIRA)

Any thoughts on this whether this is a reasonable thing to add to Lucene?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

can't you override the range methods like score(Collector, ... (see termscorer for one scorer that overrides this method)

your scorer can even do this inside the original booleanscorer (it will call this range method).

otherwise, we are adding per-document checks for what is a really special case.

[asfimport]

John Wang (migrated from JIRA)

Thanks Robert for the pointer!
I don't see how this would work (help appreciated!), the problem I am trying to get around is for the collector to provide a threshold score and my scoring function would decide based on this threshold. How would I obtain such threshold from the range?

Thanks

-John

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

checkout the TermScorer method for an example:
note it overrides score(Collector) too and just passes to Integer.MAX_VALUE for end.

  public boolean score(Collector c, int end, int firstDocID) throws IOException {
    c.setScorer(this);
    while (doc < end) {                           // for docs in window
      //System.out.println("TS: collect doc=" + doc);
      c.collect(doc);                      // collect score
      if (++pointer >= pointerMax) {
        pointerMax = docsEnum.read();  // refill
        if (pointerMax != 0) {
          pointer = 0;
        } else {
          doc = NO_MORE_DOCS;                // set to sentinel value
          return false;
        }
      } 
      doc = docs[pointer];
      freq = freqs[pointer];
    }
    return true;
  }

so your scorer and collector can totally work in tandem here...

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

and just fyi, this method is only called if you are a top-level scorer (or inside a BooleanScorer, but in that case the collector won't be "your collector")

it also currently wont be called if you do IS.search() with a filter, but there are some efforts to make it so you can control that (#2610)

[asfimport]

John Wang (migrated from JIRA)

Hi Robert:

I see what you mean! This works well and would require no api change, sweet! I think all I am waiting for is LUCENE01536 because I would have a filter.

Thanks

-John

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

#2610 would only help if your filter supports random access and is not too sparse (unless you override FilteredQuery.useRandomAccess() to always return true). In all other cases, filters are applied iterative like in previous versions. I just want to mention this.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

or as a workaround, i think you pass the filter to your query and filter inside your scorer whichever way you want (just saying, it would allow for your heuristic and for you to be top-level always)