Support multiple rescores

[nik9000]

Support multiple rescores

Detects if rescores arrive as an array instead of a plain object. If so
then parse each element of the array as a separate rescore to be executed
one after another. It looks like this:

   "rescore" : [ {
      "window_size" : 100,
      "query" : {
         "rescore_query" : {
            "match" : {
               "field1" : {
                  "query" : "the quick brown",
                  "type" : "phrase",
                  "slop" : 2
               }
            }
         },
         "query_weight" : 0.7,
         "rescore_query_weight" : 1.2
      }
   }, {
      "window_size" : 10,
      "query" : {
         "score_mode": "multiply",
         "rescore_query" : {
            "function_score" : {
               "script_score": {
                  "script": "log10(doc['numeric'].value + 2)"
               }
            }
         }
      }
   } ]

Rescores as a single object are still supported.

Also add documentation on score_mode when adding documentation about multiple
rescores.

Closes #4748
Closes #4742

[nik9000]

This isn't quite ready but it is worth reviewing I think. TODO:

1. Test the explanation of multiple rescores.
2. Documentation.
3. Rest testing?

[nik9000]

Added test for explanation. It caught that I was building the explanations backwards so the first rescore looked like it processed output from the second when in fact the opposite is true.

[nik9000]

Wait! I had it backwards! The code was right and the test was wrong. Fixed.

[nik9000]

Added documentation and while I was in there documentation for score_mode.

[nik9000]

I don't think I have anything to do for the rest testing/rest spec because it just describes the body of the search request as "The search definition using the Query DSL" which I think the asciidoc covers.

[jpountz]

space missing between 'default' and 'the'

[jpountz]

This looks good in general. However it seems to me that the second rescorer would be applied to all top docs but not only the first 10? (QueryRescorer.rescore rescores all top docs which was probably ok when there could be a single rescorer but now that there should be several ones, I think the TopDocsFilter should take the window size into account?)

[s1monw]

cool stuff I like the feature!

[nik9000]

@jpountz I'll have a look at that in a bit. I thought I had a test that checked that if the second window is smaller then the first and the first doesn't pull the match into the window then the second one doesn't see it.

[nik9000]

@jpountz you were right of course. My test was actually backwards. It was making sure that the second rescore took effect when I wanted the opposite.

I've pushed a fix. I also did some reworking on QueryRescorer#rescore because it was a little twisted. The only real change is that TopDocsFilter now takes a maximum number of docs to filter and I set it to the rescore window. I also set the maximum number of docs returned by the searcher to the rescore window rather than the size of topdocs.

[nik9000]

Rebased. Is there anything else I should change in this?

[jpountz]

This looks very good . My only concern right now is about the client API. Now that it is possible to have several rescorers per request, it feels wrong to me to have the setRescoreWindow method on SearchRequestBuilder. @s1monw what do you think?

Something that would be nice also would be to validate that rescore window sizes are in strictly descending order. Otherwise applying a rescorer that is followed by a rescorer with a greater window size would be useless I think?

[s1monw]

IMO the rescore window should be max(default_window_size, max_rescorer_window where we can setRescoreWindow as a default so we don't need to specify it everywhere? and I agree we should sort the rescorer by windows size!

[nik9000]

Something that would be nice also would be to validate that rescore window sizes are in strictly descending order. Otherwise applying a rescorer that is followed by a rescorer with a greater window size would be useless I think?

I was thinking it might be nice to have a multiply rescore with a big window after a total rescore with a smaller window. The multiply must come after so you multiply the totalled score. The total has a smaller window because it is more expensive then the multiply.

setRescoreWindow as a default

I'll make this change and see what it looks like.

[jpountz]

I was thinking it might be nice to have a multiply rescore with a big window after a total rescore with a smaller window. The multiply must come after so you multiply the totalled score. The total has a smaller window because it is more expensive then the multiply.

Agreed, let's not check the window sizes in order to allow for this kind of usage.

[nik9000]

Added another commit to make setRescoreWindow set a default and to use set/add instead of set/next. If you like it I'll squash the changes together. I didn't want to have to do git backflips to get back to the old api in case this one doesn't make sense.

[jpountz]

Looks good to me. I'm going to merge this PR if there are no objections.

[nik9000]

Cool. Want me to squash the commits or will you handle it?

[s1monw]

LGTM - would be awesome if we could have an issue for this as well to mark the versions etc. otherwise +1 to the feature thanks nik!

[nik9000]

awesome if we could have an issue for this as well to mark the versions

Is #4748 what you need?

[s1monw]

yeah @jpountz made see it too :) sorry for the noise! ;)

[jpountz]

Want me to squash the commits or will you handle it?

Actually what would be nice would be to split the change into one commit for documentation of score mode that I'll merge into 1.0,1.x and master and the rest that I'll merge into 1.x and master.

[nik9000]

Done.

[jpountz]

Merged, thanks again Nik!