Mega Enhancement: Search

[sbrl]

The current search facility only directs the user to a specific page. I would be nice if it would actually search the site if it can't find an exact match. Perhaps an upgrade is in order?

Firstly we should do some research on how to create a search engine in PHP which doesn't take ages to find anything.

[sbrl]

Basically, we need to construct an 'index' of sorts that contains the most common words on every page. We could include this in the page index, but the worry here is that this will bloat the size of the page index, increasing the time it takes for the pageindex to be read in and parsed.

[sbrl]

We might want to look into an inverted index. This apparently allows for 'fast full text searches'.

Information here:

• https://www.elastic.co/guide/en/elasticsearch/guide/current/inverted-index.html
• https://xlinux.nist.gov/dads/HTML/invertedIndex.html

[sbrl]

In order to use an inverted index though, we will have to assign an id to every page. This is such that the inverted index remains small and easy to search.

We will have to come up with an algorithm to assign short unique id to every page.

[sbrl]

Perhaps we could have a single pageids.json file that maps ids onto page names and back again. That would avoid us having to touch the pageindex, and would also allow us to make the id <---> page name translation process completely transparent.

[sbrl]

We now have search! It isn't yet complete though. I have a few things that I think we really ought to deal with before closing this bug.

• Allow searching by tag
• Allow searching by title

It might also be smart to give tags and titles a weighting. What I mean by this is that occurrences of a query term in the title or tags should bump the rank by say 5 for titles and 2 or 3 for tags. This should be configurable, though.

• Give a rank weighting to query nterms found in the title / tags of a page

[sbrl]

Done! We have a fully integrated search function.

Since it's a hugely complex system (it has many more moving parts than other parts of Pepperminty WIki) it may will contain some bugs - please open an issue if you find one!