Pull request Compare This branch is 1184 commits behind ankane:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib
test
.gitignore
.travis.yml
CHANGELOG.md
Gemfile
LICENSE.txt
README.md
Rakefile
searchkick.gemspec

README.md

Searchkick

🚀 Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

  • stemming - tomatoes matches tomato
  • special characters - jalapeno matches jalapeño
  • extra whitespace - dishwasher matches dish washer
  • misspellings - zuchini matches zucchini
  • custom synonyms - qtip matches cotton swab

Plus:

  • query like SQL - no need to learn a new query language
  • reindex without downtime
  • easily personalize results for each user
  • autocomplete
  • “Did you mean” suggestions

🍊 Battle-tested at Instacart

Build Status

Get Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch

Add this line to your application’s Gemfile:

gem "searchkick"

Add searchkick to models you want to search.

class Product < ActiveRecord::Base
  searchkick
end

Add data to the search index.

Product.reindex

And to query, use:

products = Product.search "2% Milk"
products.each do |product|
  puts product.name
end

Queries

Query like SQL

Product.search "2% Milk", where: {in_stock: true}, limit: 10, offset: 50

Search specific fields

fields: [:name, :brand]

Where

where: {
  expires_at: {gt: Time.now}, # lt, gte, lte also available
  orders_count: 1..10,        # equivalent to {gte: 1, lte: 10}
  aisle_id: [25, 30],         # in
  store_id: {not: 2},         # not
  aisle_id: {not: [25, 30]},  # not in
  or: [
    [{in_stock: true}, {backordered: true}]
  ]
}

Order

order: {_score: :desc} # most relevant first - default

Limit / offset

limit: 20, offset: 40

Boost by a field

boost: "orders_count" # give popular documents a little boost

Pagination

Plays nicely with kaminari and will_paginate.

# controller
@products = Product.search "milk", page: params[:page], per_page: 20

# view
<%= paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search "fresh honey" # fresh AND honey

To change this, use:

Product.search "fresh honey", partial: true # fresh OR honey

Synonyms

class Product < ActiveRecord::Base
  searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
end

Call Product.reindex after changing synonyms.

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

class Product < ActiveRecord::Base
  def search_data
    as_json only: [:name, :active]
    # or equivalently
    {
      name: name,
      active: active
    }
  end
end

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

class Product < ActiveRecord::Base
  scope :search_import, includes(:searches)
end

To Reindex, or Not to Reindex

Reindex

  • when you install or upgrade searchkick
  • change the search_data method
  • change the searchkick method

No need to reindex

  • App starts
  • Records are inserted, updated or deleted (syncs automatically)

Keep Getting Better

Searchkick uses conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.

class Search < ActiveRecord::Base
  belongs_to :product
  # fields: id, query, searched_at, converted_at, product_id
end

You do not need to clean up the search queries. Searchkick automatically treats apple and APPLES the same.

Next, add conversions to the index. You must specify the conversions field as of version 0.2.0.

class Product < ActiveRecord::Base
  has_many :searches

  searchkick conversions: "conversions" # name of field

  def search_data
    {
      name: name,
      conversions: searches.group("query").count
      # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
    }
  end
end

Reindex and set up a cron job to add new conversions daily.

rake searchkick:reindex CLASS=Product

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

class Product < ActiveRecord::Base
  searchkick personalize: "user_ids"

  def search_data
    {
      name: name,
      user_ids: orders.pluck(:user_id) # boost this product for these users
      # [4, 8, 15, 16, 23, 42]
    }
  end
end

Reindex and search with:

Product.search "milk", user_id: 8

Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Autocomplete

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.

class City < ActiveRecord::Base
  searchkick autocomplete: ["name"]
end

Reindex and search with:

City.search "where", autocomplete: true

Typically, you want to use a Javascript library like typeahead.js or jQuery UI.

Here’s how to make it work with Rails

First, add a controller action.

# app/controllers/cities_controller.rb
class CitiesController < ApplicationController

  def autocomplete
    render json: City.search(params[:q], autocomplete: true).map(&:name)
  end

end

Then add the search box and Javascript code to a view.

<input type="text" id="q" name="q" />

<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/typeahead.js/0.9.3/typeahead.min.js"></script>
<script>
  $("#q").typeahead({
    name: "city",
    remote: "/cities/autocomplete?q=%QUERY"
  });
</script>

Suggestions

Suggest

class Product < ActiveRecord::Base
  searchkick suggest: ["name"] # fields to generate suggestions
end

Reindex and search with:

products = Product.search "peantu butta", suggest: true
products.suggestions # ["peanut butter"]

Facets

Facets provide aggregated search data.

Facets

products = Product.search "chuck taylor", facets: [:product_type, :gender, :brand]
p products.facets

Advanced

Product.search "2% Milk", facets: {store_id: {where: {in_stock: true}}}

Similar Items

Find similar items.

product = Product.first
product.similar(fields: ["name"])

Deployment

Searchkick uses ENV["ELASTICSEARCH_URL"] for the Elasticsearch server. This defaults to http://localhost:9200.

Heroku

Choose an add-on: SearchBox, Bonsai, or Found.

# SearchBox
heroku addons:add searchbox:starter
heroku config:add ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`

# Bonsai
heroku addons:add bonsai
heroku config:add ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`

# Found
heroku addons:add foundelasticsearch
heroku config:add ELASTICSEARCH_URL=`heroku config:get FOUNDELASTICSEARCH_URL`

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Other

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"] = "http://username:password@api.searchbox.io"

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Reference

Searchkick requires Elasticsearch 0.90.0 or higher.

Reindex one record

product = Product.find 10
product.reindex

Remove old indices

Product.clean_indices

Use a different index name

class Product < ActiveRecord::Base
  searchkick index_name: "products_v2"
end

Eagar load associations

Product.search "milk", include: [:brand, :stores]

Do not load models

Product.search "milk", load: false

Reindex all models (Rails only)

rake searchkick:reindex:all

Migrating from Tire

  1. Change search methods to tire.search and add index name in existing search calls
Product.search "fruit"

should be replaced with

Product.tire.search "fruit", index: "products"
  1. Replace tire mapping w/ searchkick method
class Product < ActiveRecord::Base
    searchkick
end
  1. Deploy and reindex
rake searchkick:reindex CLASS=Product # or Product.reindex in the console
  1. Once it finishes, replace search calls w/ searchkick calls

Elasticsearch Gotchas

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

class Product < ActiveRecord::Base
  searchkick settings: {number_of_shards: 1}
end

Thanks

Thanks to Karel Minarik for Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

TODO

  • Make Searchkick work with any language

History

View the changelog

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request