Option to filter out duplicates in suggestions

[kelson42]

We should be able to avoid duplicates in suggestions search. Currently and most of the time, it returns many time the same article (partly via multiple redirects). This is not only useless but limit in addition the number of alternatives proposed.

I would propose to:

• Identify duplicates (via redirects)
• Choose the best candidate (based on levenstein distance between the search pattern and all the candidates).

[holta]

As an example — attempt a title-search for "mexico" in the search box/textfield in the top-right of...
https://library.kiwix.org/wikipedia_es_all_maxi_2019-09

Presents 2 UX challenges...

• The top 6 (of 10) results of the autocomplete are duplicates, all pointing (via redirects) to this very same page: https://library.kiwix.org/wikipedia_es_all_maxi_2019-09/A/M%C3%A9xico,_M%C3%A9xico
• This page (about an old FIFA song) isn't the most relevant search result for schools (:

[kelson42]

@maneeshpm My thoughs about this problem have evolved since I have written the ticket. Here is I believe a better approach to solve it:

• At the indexing time, if the article is a redirect, write the targeted article URL instead of the redirect URL in the Xapian title index
• At the searching time, keep track of the suggestions URL and if a URL appears a second time ( from the most relevant to the less) then just skip the suggestion.

That way we are almost sure to avoid suggestion duplicates and we trust Xapian ranking algorithm.

[mgautierfr]

At the indexing time, if the article is a redirect, write the targeted article URL instead of the redirect URL in the Xapian title index

I'm not against that, but two points to notice :

• The current implementation of kiwix-serve suggestions is based on title, so even if we store the targeted article url in xapian, we will have a redirection on the browser side (But 1. this can be change. 2. Not really important for the feature itself of removing duplicates)
• On xapian side we will lost the information about the fact this is a redirection. If we want to get this information back we would need to do a "classic" findByTitle and check the found dirent. (But do we really need this information ?)
• This may not work for chained redirection as we would store the "first" targeted url. (But do we have a lot of chained redirection ? Is it a acceptable limitation ?)

At the searching time, keep track of the suggestions URL and if a URL appears a second time ( from the most relevant to the less) then just skip the suggestion.

It is probably possible to ask xapian to regroup the result by url, or even just ensure that url is unique.

[kelson42]

At the indexing time, if the article is a redirect, write the targeted article URL instead of the redirect URL in the Xapian title index

I'm not against that, but two points to notice :

* The current implementation of kiwix-serve suggestions is based on title, so even if we store the targeted article url in xapian, we will have a redirection on the browser side (But 1. this can be change. 2. Not really important for the feature itself of removing duplicates)

I'm talking about suggestions in general and the system based on Xapian. The special case of Kiwix-serve is not much of interest for me at this stage. It's way of working is "wrong" and needs to be fixed, see kiwix/kiwix-tools#205.

* On xapian side we will lost the information about the fact this is a redirection. If we want to get this information back we would need to do a "classic" findByTitle and check the found dirent. (But do we really need this information ?)

Yes, I don't believe this is something necessary (but I'm not happy about that either).

* This may not work for chained redirection as we would store the "first" targeted url. (But do we have a lot of chained redirection ? Is it a acceptable limitation ?)

Yes, this will be inefficient in that case - so we might have two similar titles pointing ultimatively the same non-redirect article.

At the searching time, keep track of the suggestions URL and if a URL appears a second time ( from the most relevant to the less) then just skip the suggestion.

It is probably possible to ask xapian to regroup the result by url, or even just ensure that url is unique.

If Xapian can do that then I vote for this approach obviously. @maneeshpm this is on you ;)

[maneeshpm]

I agree with your approach @mgautierfr. The results are returned as a Xapian::MSet object, which cannot be modified directly like a list. So we will have to use a collapse_key at the search time which will do this task for us. For the indexing time, If we can use redirectPath attribute appropriately instead of directly using path in the index.

[maneeshpm]

@mgautierfr @kelson42 It seems that using a keymaker is supported only on sorting right now, but not on collapsing mset. Doing it directly using Xapian is not possible. So we need a custom solution to this.
The issue is we stores path as the data of the index whereas Xapain uses values for tasks such as collapsing. If we use the data as a sort or collapse parameter, we will need to check and compare for each entry which will affect the performance.

[kelson42]

@maneeshpm So if we store the path as value then it will work? If "yes", then we should do so and secure back compatibility.

[maneeshpm]

If we store the path as a value, it will be as easy as setting the collapse key to the valueslot of path. We should probably do that for future zims. I am figuring out a way to ensure backward compatibility as well.

[kelson42]

@maneeshpm I would associate a key metadata "value_store" to the database and act according to its presence/value. Secure that you access this metadata in lazy mode so you don't have to access the xapian file each time you need to check the value.