Support highlighting against queries other than the search query

[nik9000]

This issue was originally about rescore queryies and I've left it intact below so the discussion makes sense. I've changed the issue to be more general. Now this is a feature request to let the user specify a query against which highlighting will run in the highlight element. Like this:

curl -XPOST http://localhost:9200/test/test/_search?pretty -d'{
  "fields": [
    "_id"
  ],
  "highlight": {
    "order": "score",
    "fields": {
      "text": {
        "number_of_fragments": 1,
        "score_query": {
          "bool": {
            "must": {
              "match": {
                "text": {
                  "query": "central iraq"
                }
              }
            },
            "should": {
              "match_phrase": {
                "text": {
                  "query": "central iraq",
                  "phrase_slop": 3,
                  "boost": 10.0
                }
              }
            },
            "minimum_should_match": 0
          }
        }
      }
    }
  },
  "query": {
    "match": {
      "text": {
        "query": "central iraq"
      }
    }
  }
}'

This is the old issue:
When sorting highlights by score the rescore query should be used if there is one. At least, I think it should be.

This gist: https://gist.github.com/nik9000/6451307
Should spit out: "Mesopotamia, while Mithna ibn Haris withdraw from <em>central Iraq</em> to the region near the Arabian desert to delay"
Rather than: "of <em>Iraq</em> fell to the Muslims after initial resistance in the Battle of Hira. File:Ctesiphon, <em>Iraq</em> (2117465493)"

I have a pretty simplistic solution for this I'll submit once I'm done running it through tests.

[s1monw]

hmm I am not sure if we should use rescore for this but maybe have a dedicated highlight query?

[nik9000]

I see how having a highlight query might be nice but using the rescore query feels right to me - when you sort highlights by score I expected it to be the same query that orders your results.

It wouldn't be difficult to modify the pull request to create an actual highlight query element in the api that defaults to the rescore query if there is one and the main query otherwise. That'd make me happy.

[nik9000]

Alternately I could make the highlight query default to the query and have an option to use the rescore query. That'd work for me too.

[jpountz]

I think using the rescore query by default for highlighting is risky since not every match is required to match it (on the contrary to the main query). But I definitely see the point of using the rescore query to highlight matches to have better snippets.

So I agree it would be nice to allow to use the rescore query to highlight matches, or maybe even an arbitrary query, probably with a fallback mechanism so that the main query gets used in case the highlighter couldn't find a snippet to highlight with the rescore query.

[s1monw]

I agree with @jpountz I think that makes most sense to have a dedicated query.

[nik9000]

Dedicated query it is then.

Since we're letting the user specify the query themselves I'm included to let the user deal with fallback highlighting with a bool query and boosts or something.

[nik9000]

If anyone has time to comment on this I'd be glad to make any changes you need. I'm happy to rebase it as well I just don't want to rebase it now only to have to rebase it again in a week when folks have time to look at it.

[nik9000]

Removed change that wasn't relevant now that we don't explicitly consider rescore queries and rebased.

[s1monw]

thanks nik!

[kimchy]

one quick small suggestion, might make sense to change the highlighter_query to just be query, because its already within the highlighting context, so we kind of repeating info? Make sense?

[nik9000]

one quick small suggestion, might make sense to change the highlighter_query to just be query, because its already within the highlighting context, so we kind of repeating info? Make sense?

We had that exact conversation in the pull request. I like calling it query for the same reason you mention I'm worried that might look vague. The first name I came up with, score_query, was just horrible.