DuplicateFilter in _search

[alexis779]

Hi,

I have indexed documents that have a duplicate field. During the search I want to display only the document that has the highest score among its duplicate class (the group of documents that have the same field value).

There has been a discussion about deduplication in Lucene mailing-list:
http://www.mail-archive.com/java-user@lucene.apache.org/msg21437.html

Would it be possible to implement the Duplicate filter which is in contrib:
https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/contrib/sandbox/src/test/org/apache/lucene/sandbox/queries/DuplicateFilterTest.java

So the request would look like:

{
    "query": {
        "term": { "name": "alexis" }
    },
    "filter": {
         "duplicate": { "field": "name" }
    }
}

Let me know if that makes sense...

[kimchy]

duplicate filter in Lucene does not work really well since Lucene moved to segment based searching, and, its problematic in distributed search as well...

[alexis779]

Thanks for your quick reply.

We did not like to see documents that looked like duplicates in search results so we set up a process to detect whether a document is a duplicate or not at indexing time, discarding it from search results by seting _boost parameter as 0.

[dustinboswell]

I have a similar need for removing duplicates, but can't remove them at indexing time for various reasons. Is there another recommended approach for this problem?