NDCG@10 doesn't include documents rated from Explain Other

[epugh]

The ndcg@10 appears to only look at the first 10 search results, which I think is the NDCG Local implementation. We notice that for a query that has only a single result, because of how NDCG works, no matter the rating, it scores 100. This makes sense.

However, when we use the Explain Other to find other documents that are relevant and score them, because they don't show up in the search results, the score for the 1 doc result remains 100. We think we should look at the explain other rated documents as well if we don't either have 10 results, or we should use all the explain other results (and make it easy to find them in the UI).

[epugh]

[epugh]

In looking at the ratings table, for the above screenshot, we do see the four documents, 1 returned by the query, and three others by the explain other, are rated.

[epugh]

In digging through the chain, it appears that the NDCG@10 uses the eachDoc scorer, which only looks at the documents returned by the search engine. However, in the version that @brigaldies wrote (which isn't this default one) we use the getBestDocs which appears to use all the rated documents. See the gist here https://gist.github.com/epugh/2e0a591e2d06b5c472bdfa704a27b142

[TheSench]

I can't speak to the exactness of the numbers, but overall, the scores from the linked gist look closer to what I would expect.

[TheSench]

What is the expected behavior when the query returns no results, and there are valid results in the "Missing documents"?

[epugh]

With this fix in #90 the scoring will include all the rated documents. Going to mark this closed, and we are working on a release.