Improve speed of current search

[blackforestboi]

Currently we use PouchDB quick-search for...well....search :)

But PouchDB-QS is already very slow with just a few dozen pages.
Also it is limited in the amount of filter options we can build in without work-arounds.

The task would be to come up with a solution for searching, that is:

• fast
• scalable (to a limit, does not have to last 2 years worth of content to be indexed - if it could, great)
• runs in the browser,
• we can build custom rankings,
• compatible with PouchDB
• possible implementation a variety of custom filters (i.e. time, domain, source, or other fields that are saved)
• persists the index

@Treora any other requirements that come to mind?

[blackforestboi]

A possible idea is to copy/mimic how WorldBrain does it right now: https://github.com/WorldBrain/Research-Engine/blob/ed1e6bc4e2a0f3386105fb5249492560e9418d39/src/js/background.js

@Treora can you give a short description how the mechanism works?

[Treora]

Search should indeed be faster, pouchdb-quick-search was not built for the scale we use it for. In its creator's own words:

@Treora: that sounds quite hopeless to me.

We could try create an index that largely lives in memory, or do less indexing but keep recent documents in memory (such that searching is only slow for older documents), or possibly something else. Regarding less indexing:

@Treora can you give a short description how the mechanism works?

I have not dived the details of how the WorldBrain (or actually Falcon) implementation of full text search works, but what I understood is that it orders visited pages by time, pulls the pages of a certain time range (default 2 weeks) into memory, and then simply does an indexOf for the query words in each of these pages, then returning the pages that contain the given words. No text index at all, but quite fast for a reasonable amount of pages (20k pages appears fine still).

In the short term, we could try to do some index-less approach for quickly filtering recent documents containing particular words; in the longer term, I would love to find/build a scalable, proper search engine, but that is a project by itself.

[reficul31]

Please tell me if I am wrong. But I think instead of taking the whole text of the page into the db how about we filter out certain words such as "as","the","in","a" etc. These words would rarely come up in the searching query. How about we take the whole text of the page and just take the important words leaving out the common words. For example if the page says "The effects of global warming on different countries".
We just save the words "effects","global","warming","different","countries". Then we could use something like the levenshtein approximate string matching to show the results. I think this would decreases the speed of the search engine by some time.

[mangalutsav]

Can we use elastic search on keywords it's very fast and reliable

[hackboxlive]

I agree strongly with @reficul31 and would furthur suggest that we could do one of these things:

:- Instead of filtering out certain common words and saving the rest of the words, we could map each page with certain tags only using certain learning algortihms(something like auto-tagging). This would speed up the search multifold.

:- The other thing which could be done is the implementation of an already known search algorithm like Page Ranking or HITS algorithm. I say, we should go for the latter one.

[blackforestboi]

Has anyone of you worked with Lunr.js? Might that be an option to implement?

[Treora]

Thanks everybody for thinking along. I'll try to address the points quickly:
@reficul31: I did not mention removing stop words, but this is indeed be a preprocessing step in practically every text search algorithm. The pouchdb-quick-search we currently use does it, and so does Falcon (both fixed for English though, but language detection is a task for another time). Using an approximate string matching algorithm (e.g. Levenshtein) on the query words would not be making things faster, but is a feature that would be nice to be more lenient with typos and endings (word stemming helps too).
@mangalutsav: elasticsearch does not run in a browser, so that is not an option here.
@hackboxlive: extracting only a few topic words/tags from a page, instead of indexing all but the obvious stop words, may be interesting; it would not be full-text search as such. Something like LSA may be a good addition at some moment, but I don't see a ready-made approach to do this now. PageRank or HITS seems out of the question here.

Despite the validity of some of your suggestions, I would like to keep the scope of the current issue just to getting a quick solution to make a faster search. I'd love to see or create a good in-browser search engine in the long term, but that requires more skill and time than appears to be available here and now.

[Treora]

@oliversauter: lunr.js would indeed be a possible tool of choice to do stemming, stop word filtering, etcetera. It is also used by pouchdb-quick-search. The question is still how to persist and access the index, which will become too large to keep everything in memory. Here is some interesting example code using Lunr and Dexie (a wrapper for IndexedDB), by the developer of pouchdb(&-quick-search).

As indicated, this may be quite a task to do properly. I removed new-comer tag accordingly. If somebody feels comfortable with the tools I'd be happy to see experiments though, but we could start with doing something simple and limiting the scope of the problem to search e.g. only recently visited pages by default.

[mangalutsav]

@oliversauter there is a new plugin for browser
https://www.npmjs.com/package/elasticsearch-browser
so it can be taken into consideration

[Treora]

@mangalutsav: that is only a client that connects to a remote Elasticsearch server, not an in-browser search engine.