[Discussion] Solving Search in Ghost?

[ErisDS]

The issue to add Search to the Posts API, is literally the oldest open issue in this repository. It’s a specialist subject, so we’ve always hoped someone with specialist knowledge would come forward to help us solve it, but unfortunately that hasn’t happened.

To try to increase visibility of the problem we set up a page advertising contributor roles - detailing that we were looking for someone to help solve adding search to Ghost. That didn’t work either :(

Unfortunately, adding search is a particularly complex challenge. This is another attempt to try to get engagement from the wider OSS community on how to do this - we’d really appreciate the opinions and input of anyone who knows their search-fu.

---

Use cases

There are two key use cases for search inside of Ghost itself - one being in the admin panel to find a post, tag, user or perhaps even a setting that you want to change. This would make a significant improvement to the usability the admin interface.

The other is adding search to the frontend of a blog, so that it becomes easier to find content. The main things that need to be searchable are post titles and content, tag names and description, user names, and probably meta titles and descriptions for the different resources. There are plenty of use cases where other things would also need to be searchable, so this will need to be extensible, but these are the key fields.

Approaches

The best way to make both of these use cases possible, as well as making the search feature available to any apps or other extensions of Ghost in future, is to make it available through the API.

There are generally two approaches to solving search that would work in Ghost that I am aware of - the first being to use the FTS features of SQL and the second being to use a third party search tool of some description.

The upside to FTS is that, if we get it working well, it could work for all Ghost installs without any need to install additional (and likely complex) dependencies. The downside is that getting FTS to work for all three of sqlite, MySQL and PG is a sizeable challenge & the status of FTS in knex is unknown - but perhaps it can be done as a bookshelf/knex plugin, in a way that will benefit the wider community as well?

The upside to using a 3rd party tool, is that there are many modern and advanced systems that we could leverage to make an exceptional search feature. The downside is that most of them require additional complex dependencies* to be installed, which mean they aren’t viable options for Ghost core.

One of the tools that has been recommended is lunr.js, and there is a plugin for Ghost themes which combines lunr & the RSS feed to make search possible on the frontend. Supposedly lunr also works in node, not just the browser, but I’ve not found much information about it.

Other tools that are popular for search include lucene and elasticsearch and there are also numerous modules for node built on top of leveldb and redis databases. There are likely solutions out there that I haven’t heard about, so I’d be interested to hear ideas from others.

* complex dependencies includes binaries or external services - Ghost is currently installable on almost any platform without too much fiddling with compilers etc, and we want to keep it that way!

Moving forward

The ideal solution, in my mind, looks something like:

• Have a rudimentary version of search built into Ghost via the API - using FTS or something else - that will work for 90% of blogs (i.e. relatively small amounts of content).
• Implement this version of search to be easily extensible, so that it’s very easy to create an app that replaces the search with something more heavy-duty like elastic search or lucene, to support larger blogs.

The immediate future

As part of Zelda, we want to introduce a search bar to the admin interface which provides auto-complete style results for posts, tags and users. This should be possible using our existing API, some minor modifications (to allow us to fetch certain fields) and Ember. It’s not ideal, but I think something incredibly rudimentary is better than nothing at all! This will be spec’d in a separate issue.

Continuing this discussion

I’m really, really looking for the community to get involved and share their ideas here. This is a relatively specialist subject and I know there are tonnes of developers out there that have a great deal more experience implementing search than I do.

I’d like to hear thoughts, ideas and experiences on FTS - will we be able to make it work for all 3 databases without having to write too much custom code? Could a bookshelf plugin work? Is it an absolute nightmare not worth pursuing?

What about lunr? Is it a viable server-side option? Does it have too many requirements or not work for large data sets, or is it the perfect basic option provided it can easily be overridden with elastic search or something else?

Is there another solution that isn’t mentioned here? Remember we can’t have leveldb or redis or other complex dependencies in core and it has to work across sqlite3, mysql and postgres.

And what about the API? What would a good search API look like? Who out there has a great RESTful search API that we should take inspiration from?
If you know someone you think could answer any of these questions, please link them here and ask them to get involved!

Thanks :)

[riyadhalnur]

I haven't seen any viable option out there yet except lunr. This look promising norch but the documentation needs more work. Norch follows almost the same syntax as lunr; so for anyone to start with it should be fairly easy. My 2 cents

[ErisDS]

@riyadhalnur norch fails the requirement of not needing to install complex dependencies as it is based on levelDB.

[CWSpear]

@nichcurtis is a search master. He says Sphinx fits your bill and has an npm module, but he hasn't used it with node before.

Pinging him to try and get him involved!

[olivernn]

Lunr should work just as well in node as it does in the browser. It doesn't have any dependencies, so should be straightforward for ghost users to set up.

I don't know the scale of the indexing/searching you are trying to achieve but lunr can scale to fairly large corpus sizes, though it is never going to compete with the likes of Solr, ElasticSearch etc. I'm more than happy to try and help with any load testing you might want/need to do though.

Let me know if I can be of any help helping you decide, even if it is to say that lunr isn't the right fit.

[CWSpear]

Lunr's scalability would be directly related to memory, correct? Isn't it all stored in memory when it does the searches? 100MB worth of data is a decent sized blog, but not unfeasible, and since this app doesn't normally have a huge need of memory (I'm running my blog on a 500 MB VM), it could definitely be a problem, no?

[ErisDS]

I imagine that for very large blogs, the memory footprint could get problematic, but whether or not this is a serious issue for 'most blogs' would depend very much on the relationship between amount of text & the size of lunr's footprint. @olivernn some sort of testing around lunr.js memory footprints for an average blog would be really helpful here I think - I'm sure we could muster up some anonymised stats on what an average blog looks like if needed.

In terms of solving this elegantly, an idea we're working towards at the moment is to start delivering 'internal apps' - where bits of new functionality are delivered as an 'app' which is automatically installed and either enabled or disabled. Search could be delivered this way, as an app which is always present but disabled by default.

The benefit to this is that not everyone wants or needs search for their blog - so if you're happy without it, you can leave it disabled. If you want it you can enable it, and if it's too memory intensive on your blog you can (in theory) swap it out for the 'ElasticSearch' app instead. There are a lot of puzzle pieces missing for apps still, but it's worth knowing this is the way we intend to move forward.

[olivernn]

@CWSpear lunr is an in memory search index, and its memory usage is going to be related to the number and size of documents being indexed, so memory use may be an issue for larger blogs.

@ErisDS I can put together some benchmarks of lunr's memory footprint if you can provide some representative sample data. It will at least give you a data point to help you make the decision.

I don't know Ghost at all but from what I see it is using SQLite as a datastore? Doesn't SQLite have a full text search index that could be used here? Or am I completely misunderstanding the architecture of Ghost.

[halfdan]

@olivernn Ghost uses SQLite3 by default, but also supports PostgreSQL and MySQL. As @ErisDS described we would need a solution that can cover all three databases.

[kowsheek]

👍 on the ElasticSearch suggestion. It has a huge community which is super active. Lots of hosting support as well. ES would work nicely for other languages as well.

Ideally, ES should come packaged, but give the option to use a built in node or one that's started elsewhere (or none at all).

Ditto what @leonkyr & @ErisDS says, especially since PostgreSQL has a pretty awesome FTS, the searching shouldn't be limited to ES, but may be the default.

The good idea would be to implement a microservice for search that can (re)index documents and search them. Inside it can be ES or FTS. ES supports text highlighting and a lot of other things.

[ErisDS]

it platform independent (you only need java)

 

Ideally, ES should come packaged

ElasticSearch would be awesome, but I'm really struggling to see how we could use it in Ghost in a way that didn't make it impossible to install?

[CWSpear]

@ErisDS I think your only option is Lunr (or something similar to it), because everything else at the very least requires a persistence layer (i.e. https://github.com/tj/reds) or a backing process (i.e. elasticsearch) (or both). Or they are specific to a particular database (https://dev.mysql.com/doc/refman/5.0/en/fulltext-natural-language.html).

So we need something that wraps around the the different databases you support/plan on supporting (SQLite, Postgres, MySQL) with a common API (i.e. if Knex ends up getting full support for it...), or something that puts everything it in memory like Lunr.

[riyadhalnur]

ElasticSearch as good as it may be, IMHO I don't see it as a possible solution for Ghost. To have to install Java to make it run for a small application is a big hassle among other things.

why couldn't we just use the indexing capabilities of the DBs Ghost supports and write a high level API for it so developers and users (there are some who will) can use it however they see fit? in other words, a polyglot framework. it's a lot of work but the payoffs will be immense.

this post here talks about how lunr.js can be integrated into a node app http://matthewdaly.co.uk/blog/2015/04/18/how-i-added-search-to-my-site-with-lunr-dot-js/ but it looks meh but I guess for basic search in blogs, it'll work just fine

[ErisDS]

I agree - Java cannot be a dependency for Ghost.

I also agree about a high level API - the way I imagine it would be that Ghost would offer a /api/search?q= endpoint from it's API, but how that search gets performed would be configurable. The underlying structure would be ideally to have a default that is built in to Ghost and then be able to plug in ES or Solr or whatever takes your fancy using an app.

What we need (and the intention of looking for a Search Lead) is to figure out how that will work. Firstly, which is the default? I think either lunr or FTS could be the default, but it needs someone familiar with implementing this sort of thing to do a bit of a test and spec it out. Second of all, how does the default work. Thirdly, what does the fine details of the API endpoint/ look like and finally how can an app hook in and override the search functionality whilst keeping the API the same?

[riyadhalnur]

ES and Lunr.js use JSON stores. i would see the system as generating JSON stores which an app can use (hook into) to power the API

[nagug]

Just a wild thought..
Can the raw queries of knex be used and a fts performed? Some amount of overhead would be there to form the query depending on the db used.

Not sure an app approach would help.. But app approach might introduce security issues.
Thanks

[codeincarnate]

This thread has died down, but I wanted to hop in because it's crucial that Ghost gets this right. I'm going to talk about why, and then outline an approach that Ghost can take to avoid this.

I've implemented search on many Drupal based sites and it's easy to mess up. Way back, Drupal devs decided to implement a search solution on top of MySql and it was plagued with problems. The search results were ineffective and performance was terrible. It became common knowledge to simply disable it altogether. It was so bad that it was effectively unusable and the community adopted the use of Solr in droves.

This is definitely undesirable, and I believe this outlines the following requirements:

• Search should be a pluggable backend. The search requirements for a blog with 100 articles and one with thousands are very different.
• Search should be performant and relevant out of the box. It's not enough to have something that kind of works, it should work well.
• Given the above, it should not be complicated to setup search. By default it should require no setup.
• It should be a fairly straightforward to move from one search backend to another as needs change.

Backends

In regards to backends, there are a number that are available as a default choice:

• Lunr.js
• SQL based Full Text Search
• search-index
• levi

Lunr.js is nice in that it has a relatively active community. However, Lunr has a couple of downsides. First, it's purely Javascript which could negatively effect performance. Second, Lunr stores it's index by serializing it to JSON. This would likely be slow, but also makes managing the index tricky.

Full Text Search (FTS) features of various databases could definitely be used and would likely have fairly solid performance. The downside of this is that it would be relatively difficult to code and maintain. It would effectively have to be implemented for MySql, Postgres, and Sqlite. Another more subtle difference is that this also means that the database begins storing metadata of the blog in addition to the blog data directly. This can make migrations and other tasks trickier since there's a portion of the database that could effectively be regenerated at any time.

Finally, search-index and levi. Both are implemented on top of LevelUp which is a LevelDB interface. At first glance, this would appear to introduce a node-gyp dependency, however LevelUp has pluggable persistence. This means that a solution built on top of LevelUp could use something like MemDOWN by default, and switch to something like LevelDown as performance needs increase.

It could also be possible to do some work to use node-pre-gyp for LevelDown so that there's no need for users to build binaries at all. In fact, this is already the case for Sqlite support in Ghost! This would also have the added advantage that all index data would be separate from the blog data itself.

Personally, I feel the LevelUp backed path is the way to go. It's an approach that allows for very straight-forward setup, is likely to have solid performance, is pretty full-featured, and avoids maintaining separate search backends (in the SQL FTS case).

Edit: Another nice feature of Levi is that it returns search results in streams which allows for all node stream related tools to be used for this.

Indexing

Thanks to bookshelf, it should be straightforward to hook into events that are occurring at the model layer. This is already being done and could be extended to meet indexing needs. There would likely need to be extra added to do index clearing and re-indexing as necessary.

Pluggable Backends

I'm not aware of everything that goes into Ghost apps, but it would be a good avenue for other backends. I'm sure this would be important for Ghost Pro and larger blogs so that a solution like Solr or Elastic can be used in that case.

I'm part way through doing a very barebones implementation of the above. Regardless of direction, I'd love to hear some feedback and thoughts.

[codeincarnate]

An update to my last comment, I have a super, super basic implementation here. This doesn't have pluggable backends or anything, just some indexing and the ability to pass in queries at the moment.

[CWSpear]

@codeincarnate looks like that's a private repo (or maybe just a bad link?) as I got a 404 when trying to visit it.

[codeincarnate]

@CWSpear Missing an "s" at the end, fixed now!

[gergelyke]

@ErisDS any update on this?

[codeincarnate]

@ErisDS would love to have some feedback. My business partner is getting married this weekend, but after that I'm going to have more time to really take a crack at this.

[sethbrasile]

@codeincarnate if you get into this, feel free to throw some tasks my way if you'd like.

[codeincarnate]

@sethbrasile That's a very generous offer and I'll definitely do so. Hoping to get some feedback from @ErisDS before taking a deep dive into this.

[ErisDS]

@codeincarnate thanks for the enormous writeup, and apologies for taking a while to digest it.

It seems to me that, providing that we can have a non-binary/in-memory dependency as the default, this approach looks like it may be a great solution.

Having a node-pre-gyp wrapper for the binary version is very desirable, even if it's not the default. I would recommend that we avoid having another default dependency which requires a binary, even with node-pre-gyp, because it does still add overhead both to users installing and to developers (switching between node versions, which is becoming commonplace during testing, is harder with binary deps).

I'd be interested to see some tests to show at roughly what point in-memory search becomes untenable. E.g. how many 500 word posts is the limit, assuming a DO $5 droplet / 512MB ram? This could help us to provide useful documentation on how and when to upgrade to 'advanced' search, as well as perhaps even add warnings inside Ghost itself.

Not sure if you've seen this, but here's the documentation for how to configure a custom storage module, which is a similar 'pluggable' extension to Ghost: https://github.com/TryGhost/Ghost/wiki/Using-a-custom-storage-module. I'd imagine the extra config for search would be similar - an npm install & a config.js update. This could be done in a more managed way maybe using the app platform in future, but for the time being this approach works.

The only major question I'm left with atm is why levi over search-index?

[jberkus]

PostgreSQL Expert, JS neophyte here, just putting in:

For Postgres, at least, full text search can be added in ways which are invisible to the user, and which work fine with backup/restore. Further, if you only support simple boolean operations (i.e. AND/OR words) and don't support more complex operators, it's trivially easy to write a library which would convert user-obvious syntax into the syntax supported by each FTS engine. The one user-visible operation would be requiring the user to choose a language, as FTS dictionaries are language-specific, and the dictionary would become an optional installation requirement.

MySQL is more complicated, especially if you need to support core MySQL and not require MariaDB or Percona. Core MySQL, AFAIK, does not support FTS for InnoDB tables, forcing you to make a rather substantial user-visible change which affects reliability. This is why prior to Maria/Percona, most MySQL users used Spinx search instead.

[jberkus]

The main reason to support SQL-based FTS would be in order to support large websites with several GB of text to search. Does anyone even use Ghost for that?

[codeincarnate]

@jberkus Awesome run down in terms of Postgres. For the most part, many people are likely to be running Ghost on a small blog and there's probably not going to be more than 100mb of text or so and even that is probably an over-estimation.

From the perspective of ghost, it can run on top of MySql, Postgres, or Sqlite. In terms of search, MySql is likely the one to hold it back. More than that however, dealing with a wide array of potential database configurations is not ideal for a default search function across small blogs.

As I mentioned above, this is the point of having search be a completely pluggable interface. If you're running Ghost on top of Postgres it makes a lot of sense to also have search in Postgres as well. For a service like Ghost Pro, running search on top of Elastic or Solr makes sense. For the easy case, something that is zero configuration and no worries is definitely for the best.

[paambaati]

Just in case someone landed here looking for a quick-and-dirty solution, I've written about it here — https://httgp.com/adding-search-to-ghost/

tl;dr
Fetch all post content on page load via ServiceWorker and build search index via Fuse.js.

[pascalandy]

@paambaati Do you have a PR somewhere with the solution's implementation you are describing in your post?

[paambaati]

@pascalandy Not really, because the solution will work for only small blogs, and wouldn't scale for larger blogs. This discussion brings up some other interesting and yet valid use cases, and hence the lack of a PR.

[kevinansfield]

@pascalandy, @paambaati this is essentially what GhostHunter (most common Ghost search plugin) does except that uses lunr.js instead of fuse.js. It does scale to large blogs (100s of posts) reasonably well but very large blogs (1000s of posts) can struggle, it requires a fair bit of server-side processing to generate the API response and then you have the download time as well although that generally won't be much bigger than a large image.

There is also an algolia integration but bear in mind it currently requires core to be patched every time you upgrade.

We have an Ideas topic on our forum now, it's worth adding your vote to the Search idea and joining the discussion there.

[pascalandy]

@paambaati

Not really, because the solution will work for only small blogs, and wouldn't scale for larger blogs.

Just to make sure, I meant any PR from your example into your own git repo (not a PR into the official ghost project.)

I wanted to try fuse.js as I saw many times that GhostHunter was not working properly. Thanks!

[jozic]

I use twitter's typeahed, it's far from ideal, but works
see it live here https://onlydarkchocolate.com/
and here's PR to my Casper fork jozic/Casper@bde7c79

[pascalandy]

It works very well @jozic. What makes it far from ideal?
Thanks for the PR !!

[jozic]

@pascalandy
things i don't like

• when you select an item from the list, before opening the post it puts selected item as json in search field (minor)
• sometimes it will not open an item, not very often, but it's not great experience for sure

also it loads all posts and does search on client side, which won't scale (but I'm not at that scale yet)

otherwise, it works fine

[aiden-leong]

3 years passed

[vikaspotluri123]

@andy1247008998 The same sentiment in #4453 (comment) applies here.

[juan-g]

This feature can be voted at Search for Ghost - Ideas - Ghost Forum, as Kevin suggested.

[lengerrong]

just an idea.

use sqlite3 as database for your ghost.
then there will be a database file.
open the database file via sqlite3.

$sqlite3 content/data/ghost-dev.db
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> .tables
accesstokens migrations refreshtokens
api_keys migrations_lock roles
app_fields mobiledoc_revisions roles_users
app_settings permissions sessions
apps permissions_apps settings
brute permissions_roles subscribers
client_trusted_domains permissions_users tags
clients posts users
integrations posts_authors webhooks
invites posts_tags

As you can see, all posts definitely stored in the table named 'posts'.

sqlite> select * from posts;
id|uuid|title|slug|mobiledoc|html|comment_id|plaintext|feature_image|featured|page|status|locale|visibility|meta_title|meta_description|author_id|created_at|created_by|updated_at|updated_by|published_at|published_by|custom_excerpt|codeinjection_head|codeinjection_foot|og_image|og_title|og_description|twitter_image|twitter_title|twitter_description|custom_template

Then we got the structure of the 'posts' table.
As I tested, 'slug' is the route path of a recorded post.

let's search posts contains 'welcome' as an example.
sqlite> select slug from posts where html like '%welcome%';
slug
apps-integrations
welcome

then we can check the real post in browser.
http://localhost:2368/welcome
http://localhost:2368/apps-integrations

see, search things goes well in a shell bash.
all we need to do write js codes to do above operations and display the result correctly in ghost core server module.

anyone who familiar with ghost developing work can have a try for that.
any suggestion please guide.
I would like to implement it at part time.
Hope I can finish it soon.

[zicklag]

GhostSearch by HauntedThemes works well for my small blog. I've just got to work it into the theme.

[Windyo]

Quick ping here : does it make sense to continue maintaining GhostHunter ? With the V2 of ghost having been just released, I'm wondering if you guys are closer to releasing the Apps API, or maybe your own search, or if I should keep maintaining the package.

[ErisDS]

@Windyo we are not really closer to implementing search or any sort of apps into Ghost itself, instead we are focusing on providing extensibility through the API & Webhooks.

The new v2 API will offer two relevant things:

1. an easier integration point with something like Algolia through webhooks, although it will still require a separate service to be running to, for example process the webhooks & send data to algolia in the right format
2. the v2 API is designed to be cached, which means that it is much better at handling the kind of heavy usage that busy sites might run through GhostHunter.

GhostHunter is currently one of the most commonly used extensions for Ghost and I think it will stay that way for a little while longer yet 😊

[Windyo]

Alright @ErisDS I'll keep GH up to date at least then - May have ideas for minor improvements yet.
Let me know if the latest changes make it better for your server load.

Thanks for the update!

[r3wt]

I am building a route in our node.js proxy today which will query the ghost api at startup, build an index, and then periodically query and update the index. this will allow me to add search capabilities for our ghost blog, which is headless (we have a react webapp, and only use ghost api). when i finish, i'll share the code for you here, and someone can port it into ghost if the team wants.

I've built several simple full text search engines in node.js, so its not something i'm new to at all. i'm happy to share the solution, so that the community can take it and then all you have to do is add the UI part of the equation so it can ship with ghost by default and make it available in the api and api client libraries. hope it helps.

its amazing to me that in 6 years time the support for this feature hasn't been added. truly mind boggling, but hey atleast its still better than wp

[r3wt]

as promised, i've written a basic implementation of a search engine for ghost. now, this utilizes the tryghost/content-api library for fetching the posts, and it uses elasticlunr.js for the index and search implementation, and it runs as a route in express app. i was hoping to throw up a gist for this, but working in a startup and the need to move fast, i really don't have time. i'm just gonna drop this mvp right here for those who might wonder how to do this easily.

EDIT: code has been updated to fix a deadlock, and an archive feature (browse post by month) has been added. code has also now been moved to a gist, which you can find here

I'll try and update this later as i have time. hope you all can find some use from this.

[vikaspotluri123]

@r3wt the implementation seems similar to jamalneufeld/ghostHunter and the serverside implementation (I've done quite a bit of work on the serverside version). There's also HauntedThemes/ghost-search which is a frontend-only implementation

[r3wt]

@vikaspotluri123 not sure why you would say that, but ok. my implementation uses elasticlunr rather than lunr, and also works with the v2 ghost api, leveraging the content-api package from GhostSDK. The only real similarities i see are that they provide a simple search endpoint for use with a ghost blog. I think the advantage of mine is that you make no changes to ghost. you just go into the admin interface and add an integration and grab the key. pretty simple. the code above is unlicensed, anyone can have it and freely modify it. at some point i will probably release it as an MIT license module and make it more configurable. the advantage of elasticlunr is that you can control the scoring pipeline and also change the weight of fields, so you can tune the search results as needed for your application. that's why i chose it over lunr.

best of luck.

r3

[vikaspotluri123]

@r3wt great points! GH-Server hasn't been updated for the V2 API, but basically does the same thing, just using the v0.1 public API 😉 GH-Server is also designed to run in parallel to Ghost, and also has frontend integration because GH does 😂 I don't want to pollute this thread, so if you want to talk specifics or collaborate, feel free to drop a line 😊

[r3wt]

@vikaspotluri123 Sounds cool, if i take a larger interest in ghost i will definitely take you up on on that. Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aiden-leong]

This issue should not be closed until solved.