search still not fully working?

[mooreniemi]

Compare these two results:

http://www.transbucket.com/pins?surgeon=5
http://www.transbucket.com/pins?utf8=%E2%9C%93&query=Fischer%2CBeverly&commit=Go

Seems like search is not returning all, or we're not paginating?

[mooreniemi]

I'm kind of split on whether to fix this or just try to jump out to something else like Elasticsearch. ES is overkill in a way but I think we could do the free tier on AWS and thus kill the insanely over-priced Thinking Sphinx monthly charge.

[mooreniemi]

I am pretty far along now, and have an ES domain on AWS set up which I can hit as well as a local instance working. I think what I will do is try to get this up on staging tonight or tomorrow. If it works there I will roll it out and cancel Thinking Sphinx.

[mooreniemi]

Hmm, the instance seems stuck/dead.

[mooreniemi]

I tried changing EBS amount to force it to BG but still stuck.

[mooreniemi]

Trying to up it to 2 nodes. If it has to be on 2 just to run we're pretty much back to square one wrt the budget.

[mooreniemi]

That brought it back up. (Or it finally just recovered...) I put it back down to 1 node, watching to see if it goes out of Processing.

I am wondering if I should kill this experiment though and just do postgres full text search...

[mooreniemi]

Went down overnight at some point, which doesn't scare me as it came back up. But! The index was deleted, too!

We also need better client handling by the service to deal with getting a 404.

I've never used the burstable instances. It must've dropped the node and without a replica not been able to restore the index. This suggests we'll need to carefully monitor it to keep it up or add another node.

[mooreniemi]

Again a node went down, and despite going to 2 nodes still in Processing. I think it's worth trying postgres instead.

https://www.postgresql.org/docs/9.5/textsearch-tables.html
https://stackoverflow.com/questions/10875674/any-reason-not-use-postgresqls-built-in-full-text-search-on-heroku

[mooreniemi]

I think I will merge this and leave it up on 2 nodes, hoping that that can stay up "mostly" until replaced. I would like to at least implement better error handling on 400/500 codes from ES though.

Support post: https://forums.aws.amazon.com/thread.jspa?threadID=335962

[mooreniemi]

Things to try:

https://github.com/Casecommons/pg_search
https://github.com/textacular/textacular

https://github.com/sunspot/sunspot

Is used by websolr which is $20 a month on heroku.

https://elements.heroku.com/addons/bonsai has a free tier which is plenty big enough for us I think. This might be the best starting option?

I think I will try the Bonsai free tier for now. If it can stay up under load, that might work for now. Gives us a bit of room to try pg search as well.

https://docs.bonsai.io/article/97-ruby-on-rails

[mooreniemi]

Bonsai is up now, we'll see.

[mooreniemi]

I'm calling this all set now. There's more improvements to do, but Bonsai has held up with no issue at all.