Active Search

Active Search implements search for Rails. It provides a common interface for searching Active Record models across multiple search backends.

This is alpha software. The API is not fixed and may have backward incompatible changes in future.

Supported adapters:

• Database: SQLite FTS5, PostgreSQL tsvector, MySQL FULLTEXT
• Search engines: Elasticsearch, OpenSearch, Solr, Meilisearch, Typesense, Manticore, Redis Search

• Installation
• Quick start
• Configuration
  • Named adapters
  • Index prefix
  • Database adapters
  • Search engine adapters
• Indexing
  • Index definitions
  • Collection fields
  • Polymorphic indexes
  • Models
  • Manual indexing
• Searching
  • Queries
  • Filtering
  • Highlighting
  • Results
  • Loading records
  • Routing
  • Native queries
• Adapters
  • Capabilities
  • Documented divergences
  • Verified versions
  • Custom adapters
• Commands
• Instrumentation
• Limitations
• License

Add the gem:

gem "rails-active_search"

For a search engine, add its client gem:

gem "elasticsearch", "~> 8.0"  # match your server's major version
gem "opensearch-ruby"
gem "rsolr"                    # Solr
gem "meilisearch"
gem "typesense"
gem "redis"                    # Redis Search
# Manticore needs no gem

The Elasticsearch client must match the server's major version. See Verified versions.

1. Configure the adapter. Create config/search.yml:

test:
  adapter: sqlite

production:
  adapter: elasticsearch
  hosts:
    - host: search.example.com
      port: 9200

2. Define an index. Declare it in config/search.rb:

ActiveSearch.define_index(:articles) do
  text :title
  text :content
  integer :account_id
  string :status
  datetime :published_at
  boolean :featured
end

3. Connect the model. Add has_search:

class Article < ApplicationRecord
  has_search
end

See Models.

4. Search.

results = Article.search("rails").results

results.each do |article|
  article.title      # the record
  article.hit.score
end

config/search.yml sets the adapter for each environment, as in the Quick start. The migrations and connection settings for each adapter follow.

An environment can have several stores, one per key. Set store_name: on an index to use one of them. Without it, the index uses default.

production:
  default:
    adapter: elasticsearch
    hosts:
      - host: search.example.com
        port: 9200
  legacy:
    adapter: mysql

ActiveSearch.define_index(:legacy_articles, store_name: :legacy) { ... }

index_prefix is added to the front of every index name on that store. Use it when two environments share one search engine. The install generator sets development_ and test_ for the search engine adapters.

development:
  adapter: elasticsearch
  index_prefix: development_

A database adapter stores each index in a table with a column per filterable field and a full-text index over the text fields (see Index definitions). A multiple: field is a JSON column, jsonb on PostgreSQL. The application owns the migration. The document generator writes it.

SQLite uses a table for the fields and an FTS5 virtual table for the text:

class CreateArticleSearch < ActiveRecord::Migration[8.0]
  def change
    create_table :article_documents do |t|
      t.string :article_id, null: false
      t.integer :account_id
      t.string :status
    end
    add_index :article_documents, :article_id, unique: true
    create_virtual_table :article_documents_fts, :fts5, [:title, :content]
  end
end

FTS5 does not stem by default, so company does not find "companies". Add the porter tokenizer to stem:

create_virtual_table :article_documents_fts, :fts5,
  [ :title, :content, "tokenize='porter'" ]

PostgreSQL uses tsvector columns with GIN indexes:

create_table :article_documents do |t|
  t.string :article_id, null: false
  t.text :title, :content
  t.tsvector :title_vector, :content_vector
  t.integer :account_id
end
add_index :article_documents, :title_vector, using: :gin

MySQL uses a FULLTEXT index:

add_index :article_documents, [:title, :content], type: :fulltext

MySQL matches a FULLTEXT index by its whole column list. A search that names a subset of the fields (see Queries) needs its own index over exactly those columns.

Each adapter has its own connection settings:

# Elasticsearch or OpenSearch
production:
  adapter: elasticsearch
  hosts:
    - host: localhost
      port: 9200

# Meilisearch
production:
  adapter: meilisearch
  url: http://localhost:7700
  api_key: your_master_key

# Typesense
production:
  adapter: typesense
  nodes:
    - host: localhost
      port: 8108
      protocol: http
  api_key: your_api_key

# Solr
production:
  adapter: solr
  url: http://localhost:8983/solr

# Manticore
production:
  adapter: manticore
  host: localhost
  port: 9308

# Redis Search
production:
  adapter: redis_search
  host: localhost
  port: 6379

Then create each index. See Commands.

The Elasticsearch and OpenSearch clients log to Rails.logger. Set config.active_search.logger to send their logging elsewhere.

Declare each index in config/search.rb. A text field is searched. The other types are filtered and sorted, and a string field holds one exact value. Declare the type your application writes. ActiveSearch casts each value to it when it builds a document.

ActiveSearch.define_index(:articles) do
  text :title
  text :content
  string :status
  integer :account_id
  float :price
  boolean :featured
  datetime :published_at
  date :published_on
end

Each index loads its results through a source. By default that is the class named by the singular of the index name. Set source: to name another class, or polymorphic: true to load several. On a database adapter, set document_class: to use your own model for the index table in place of the generated one.

ActiveSearch.define_index(:articles) { ... }                              # Article
ActiveSearch.define_index(:namespaced_tests, source: "Article") { ... }   # Article
ActiveSearch.define_index(:records, polymorphic: true) { ... }            # several classes
ActiveSearch.define_index(:records, document_class: "Search::RecordDocument") { ... }

Set multiple: true on an integer, string or datetime field to store a list of values:

ActiveSearch.define_index(:topics) do
  text :subject
  integer :folder_ids, multiple: true
  string :labels, multiple: true
end

A filter matches documents whose list contains any of the given values, the same as IN on a single value:

# topics in folder 4, folder 9, or both
ActiveSearch.index(:topics).filter(folder_ids: [ 4, 9 ])

A Range filter matches when any element is in the range. Check supports_collection_ranges? first. A collection field cannot be sorted on.

To index several models in one index, declare it polymorphic: true:

# config/search.rb
ActiveSearch.define_index(:records, polymorphic: true) do
  text :title
  text :body
  integer :account_id
end

Two extra fields, record_type and record_id, record each document's class and id. They are named after the singular of the index name. To use another name, pass it as the option:

ActiveSearch.define_index(:records, polymorphic: :searchable) { ... }   # searchable_type, searchable_id

You can declare these two fields yourself, as single-value string fields only. The id is stored as text so that models with different id types can share the index.

Post.search on this index filters on record_type: "Post". A filter you add on record_type narrows this and cannot widen it. ActiveSearch.index(:records) searches every model.

Connect each model with has_search index::

class Post < ApplicationRecord
  has_search index: :records, serializer: :to_content_document

  def to_content_document
    { title: headline, body: body, account_id: account_id }
  end
end

class Page < ApplicationRecord
  has_search index: :records,
    serializer: ->(page) { { title: page.title, body: page.content, account_id: page.account_id } }
end

has_search connects a model to an index. Without index:, the index is the one named after the table. It adds callbacks that reindex the record after every create and update, whatever changed, and remove it after destroy. It adds three methods to the model:

• .search("query") returns a chainable Query. Call it without text for a filter-only query
• .suppress_indexing and .indexing_suppressed?
• #hit, the search metadata of a loaded result

class Article < ApplicationRecord
  has_search
end

Article.search("query")
ActiveSearch.index(:articles).filter(status: "published")

suppress_indexing turns the callbacks off for saves inside the block:

articles = Article.where(status: :draft).to_a
Article.suppress_indexing do
  articles.each { |article| article.update!(status: :archived) }
end

When the index fields do not match the model's attributes, give has_search a serializer that returns the document hash:

class Product < ApplicationRecord
  has_search serializer: ->(r) {
    { name: r.name, description: r.description, price: r.price&.to_f,
      category: r.category, author_name: r.author&.name }
  }
end

Or name a method:

has_search serializer: :to_search_document

A model can be in several indexes. .search uses the first one declared, or the one with default: true. Pass index: to search another:

class Article < ApplicationRecord
  has_search index: :articles
  has_search index: :records, serializer: :to_content_document
end

Article.search("query")                          # :articles
Article.search("query", index: :records)         # :records

touch does not reindex unless reindex_on_touch is set on the index:

class Topic < ApplicationRecord
  has_search reindex_on_touch: true
end

touch_all and increment!(touch: true) do not run commit callbacks, so they do not reindex. Call reindex after them.

The other has_search options:

• async: false -- write inline instead of through Active Job
• Guard options: if, unless, add_if, add_unless, remove_if, remove_unless
• scope: -- the relation used to load results. See Loading records

if and unless apply to both the add on save and the remove on destroy. add_if and add_unless replace them for the add. remove_if and remove_unless replace them for the remove. A saved record that no longer passes its add guard is removed, and only remove_if and remove_unless are checked for that removal.

add and remove write at once. reindex_later and remove_later enqueue an Active Job. remove_later reads the id before the job runs, so it works after the record is destroyed.

index = ActiveSearch.index(:articles)
index.add(article)
index.remove(article)
index.reindex_later(article)
index.remove_later(article)

has_search async: false  # write inline for this model

reindex_later checks the guards when the job runs, so it can remove the document instead of adding it.

reindex applies every has_search declaration on the record. It adds the record where the guards pass and removes it where they fail. The after_commit callback calls it, and you can call it directly:

card.reindex

Card.find_each(&:reindex)     # repair after an index loss

ActiveSearch.index(:cards).add(card) ignores the guards. After a change to what qualifies, use reindex, so that records that no longer qualify are removed.

reindex uses each declaration's async: setting. An asynchronous index gets a job, and the others are written inline.

batch collects writes and sends them to the store in groups of at most max_size:

ActiveSearch.index(:articles).batch(max_size: 500) do |batch|
  Article.find_each { |article| batch.add(article) }
end

remove_by_filter removes every document that matches a filter and returns the count. Use it after a delete_all, which runs no callbacks and so removes nothing from the index:

ActiveSearch.index(:searchable).remove_by_filter(account_id: 1)   # => 42
ActiveSearch.index(:records).remove_by_filter(account_id: 1, record_type: "Post")

It takes the same filter arguments as filter. It has no model type filter, so on a polymorphic index add the type field, as in the second example. An empty filter raises.

Each query method returns a new Query. Chain them in any order, and call .results to run the search. A Query has no each, map or count. .results returns the page.

ActiveSearch.index(:articles)
  .search("ruby programming")
  .filter(account_id: 1)
  .highlight(title: true)
  .sort(published_at: :desc)
  .limit(20)
  .offset(40)
  .results

operator: :and requires every term. fields: limits the search to the named text fields:

Article.search("ruby rails", operator: :and)
Article.search("ruby", fields: [ :title ])

operator is not supported everywhere. Check supports_operator?. Without it, each engine uses its own default.

filter narrows the search by a field's value, a list of values or a range:

Article.search("rails").filter(account_id: 1)
Article.search("rails").filter(status: "published", account_id: 1)
ActiveSearch.index(:articles).filter(status: ["published", "draft"])
ActiveSearch.index(:articles).filter(views: 100..)            # >= 100
ActiveSearch.index(:articles).filter(views: ..100)            # <= 100
ActiveSearch.index(:articles).filter(views: ...100)           # < 100
ActiveSearch.index(:articles).filter(price: 10..50)           # between 10 and 50
ActiveSearch.index(:articles).filter(created_at: 1.week.ago..)

To filter on a missing value, pass nil. Check supports_missing_filters? first:

Article.search("rails").filter(status: nil)                # absent
Article.search("rails").filter(status: [ "draft", nil ])  # draft or absent
Article.search("rails").reject(status: nil)                # present

reject excludes matches:

ActiveSearch.index(:articles).reject(status: "draft")               # status != draft
ActiveSearch.index(:articles).reject(status: ["draft", "archived"]) # NOT IN
ActiveSearch.index(:articles).reject(views: 100..200)               # NOT BETWEEN

# Combined with regular filters
ActiveSearch.index(:articles)
  .filter(account_id: 1)
  .filter(views: 100..)
  .reject(status: "draft")
  .results

Several conditions in one reject are negated together, like where.not in Active Record:

reject(status: "draft", featured: true)   # not (draft and featured)
reject(status: "draft").reject(featured: true)   # neither draft nor featured

Several values for one field are one condition, so each of them is excluded.

filter_any takes an Array of alternatives and requires one of them. Each alternative is a Hash, and its conditions apply together.

# featured, or published this week
ActiveSearch.index(:articles).filter_any([
  { featured: true },
  { status: "published", published_at: 1.week.ago.. }
])

An empty list matches nothing. An alternative with no conditions raises. filter_any combines with the other filters through AND:

ActiveSearch.index(:articles)
  .filter(account_id: 1)
  .filter_any([ { featured: true }, { status: "published" } ])

results = Article.search("ruby").highlight(true).results

results.each do |result|
  result.hit.highlight(:title)   # => "<mark>Ruby</mark> Programming Guide"
  result.hit.highlight(:content) # => nil, because nothing in content matched
end

A field with no match has no highlight, and hit.highlight returns nil for it on every adapter. Fall back to the field value when you want one:

result.hit.highlight(:content) || result.content

# Per-field with snippets and custom markers
Article.search("ruby").highlight(
  title: { markers: ["<em>", "</em>"] },
  content: { snippet: { words: 15 } }       # or { characters: 150 }
)

highlight takes format, markers and snippet. Any other key raises. Set backend options through native.

Different options on different fields need supports_highlight_per_field_markers? and supports_highlight_per_field_snippets?. A field with no snippet counts as a different snippet setting:

# Needs supports_highlight_per_field_markers?
Article.search("ruby").highlight(title: { markers: ["<em>", "</em>"] },
                                 content: { markers: ["<b>", "</b>"] })

# Both need supports_highlight_per_field_snippets?
Article.search("ruby").highlight(title: { snippet: { words: 5 } },
                                 content: { snippet: { words: 15 } })

Article.search("ruby").highlight(title: true,
                                 content: { snippet: { words: 15 } })

results = Article.search("rails").highlight.results

results.total          # documents that matched, across all pages
results.total_exact?   # false when the total is an estimate or a lower bound
results.partial?       # true when the store stopped early
results.dropped        # hits on this page with no record to load
results.size           # records on this page
results.empty?
results.next_page?     # true when the store has hits beyond this page

results.each do |article|
  article.hit.score
  article.hit.highlight(:title)  # nil unless .highlight was called
  article.hit.fields[:status]
end

Each result is the record. hit.fields holds every declared field, and hit_fields limits it:

results = Article.search("rails").hit_fields(:status).results

total counts index hits. size counts the records loaded for this page. Some stores report an estimate or a lower bound, and then total_exact? is false.

Use next_page? to page. false means the store returned no further hits. On a partial? page there can still be matches the store did not reach.

partial? is true when the store stopped early, for example on a timeout. Then total may be low and next_page? may be false with matches unseen.

results = Article.search("rails").results
render_warning if results.partial?

A query without limit uses config.active_search.default_limit, which is 25.

# config/initializers/active_search.rb
Rails.application.config.active_search.default_limit = 50
Rails.application.config.active_search.default_limit = nil  # no limit, each store applies its own

# No limit for one query:
query.limit(nil).to_native_query

A search returns document ids, and the source loads the records with Model.where(id: ids). Set scope: on the model or on the query to load them through another relation:

class Recording < ApplicationRecord
  has_search scope: -> { not_removed }
end

Recording.search("q")                                             # the model's default
Recording.search("q", scope: Recording.preload(:creator))         # this query only
Topic.search("q", scope: identity.accessible_topics)              # a relation built per request

The query's relation is merged into the model's default with ActiveRecord::Relation#merge.

A scope does not filter the search. The store chooses the page of hits first, and the scope only loads them. Hits the scope excludes are counted in results.dropped.

On Elasticsearch and OpenSearch, route_by stores each document on a shard chosen by a field, and a search that filters on that field reads only that shard:

# config/search.rb
ActiveSearch.define_index(:articles, route_by: :account_id) do
  text :title
  integer :account_id
end

Article.search("q").filter(account_id: 1).results          # one shard
Article.search("q").filter(account_id: [ 1, 2 ]).results   # two shards
Article.search("q").filter(account_id: 1..5).results       # QueryError

A negated filter, or a filter_any with an alternative that does not name the field, reads every shard. Query#routing returns the route, or nil.

Do not change a record's route_by value. The old document stays on the old shard, and a later destroy does not reach it.

Test routing on an index with more than one shard. With one shard, a wrong route still returns everything.

to_native_query returns the request as built. native takes a block to change it before it is sent.

Treat a native block as trusted code. Its return value is sent to the backend without validation. It can remove the model's type filter. Never pass untrusted parameters into it.

Search engine adapters return a hash:

ActiveSearch.index(:articles).search("ruby").to_native_query
# => { query: { bool: { must: { simple_query_string: {...} } } } }

# Modify the query before execution
ActiveSearch.index(:articles).search("ruby").native { |query|
  query[:timeout] = "5s"
  query
}.results

Database adapters (SQLite, PostgreSQL, MySQL) return an Active Record relation on the document model. To merge it into a query on your model, join the document table first. Document ids are stored as strings, so the join casts. On MySQL cast to CHAR:

search = ActiveSearch.index(:articles).search("ruby").to_native_query
doc_table = search.model.table_name

Article
  .select("articles.*")
  .joins("INNER JOIN #{doc_table} ON #{doc_table}.article_id = CAST(articles.id AS TEXT)")
  .merge(search)

Every adapter supports text search, exact, negative, IN and range filters, sorting, pagination and hit fields. The rest depends on the adapter. Check the index's capabilities. An unsupported operation raises UnsupportedOperationError.

capabilities = ActiveSearch.index(:articles).capabilities
capabilities.supports_highlighting?
capabilities.supports_operator?
capabilities.supports_missing_filters?
capabilities.supports_search_subfields?
capabilities.supports_search_string_fields?
capabilities.supports_collection_ranges?  # a range over a multiple: field
capabilities.supports_snippet_unit?(:words)     # or :characters

Each of these is a property of the backend, and each is asserted in the test suite.

• Redis Search needs 2.10 or later, query dialect 2, and INDEXMISSING on every filterable field. Dialect 2 provides ismissing(), which is used for missing-value filters and for an empty filter list.
• Typesense approximates operator: :or. It maps to drop_tokens_threshold, so terms are dropped only when the full match finds fewer than 100 documents.
• Solr does not distinguish "" from an absent field. It indexes no term for an empty string, so an existence filter does not find it. false and zero are unaffected. An empty collection also reads back as absent. Every other adapter keeps the two apart.
• SQLite sorts relevance ascending. FTS5 rank is negative, and the best match is the most negative.
• Redis Search filters integers as doubles, so values above 53 bits are not distinct in a filter. They are stored and read back intact. Declare such a field as string if you filter on it.
• Redis Search indexes every collection as TAG. The values are joined with U+001F, so the FT.CREATE schema must declare SEPARATOR "\x1f" on the field, and a value containing U+001F is not supported. [ "" ] and [] both read back as [].
• Manticore has no NULL. An absent text field reads back as "". Every other adapter returns nil.
• Manticore stores an integer collection as a multi-value attribute, which is a set. (3, 1, 3) reads back as 1, 3. Other collection types go in a json column, which keeps order and duplicates.

Every claim in this README is tested against the versions in docker-compose.yml, pinned by digest.

Each Elasticsearch major is tested with its own client, because a client only works with its own major.

For a version not in the table, run the shared adapter tests in test/store_adapters/adapter_test.rb against it.

To write a source or an adapter of your own, see docs/custom_adapters.md.

rails active_search:install   # Write config/search.yml and config/search.rb (ADAPTER=name)
rails active_search:status    # What exists, and what to do next
rails active_search:health    # Ping each store; database adapters answer without a connection
rails active_search:verify    # Does each index provide what its declaration needs

install writes sqlite for development and test. Pass ADAPTER for another:

rails active_search:install ADAPTER=elasticsearch

Each adapter defaults to localhost on its own port. Typesense has no default key, so set TYPESENSE_API_KEY. See Configuration for the other settings.

To declare an index and connect its model:

rails generate active_search:index Article title body:text status:string views:integer

This writes the define_index block into config/search.rb and adds has_search to the model. On a database adapter it also writes the document model and its migration. A field with no type is text, and a third segment is a modifier:

rails generate active_search:index Article commenters:string:multiple

status and verify take INDEX=name for one index. status takes COUNT=1 to count the documents in each index.

verify exits non-zero when anything is wrong, so it can run in CI or before a deploy:

A run over several indexes exits with the worst code. Codes 3, 4 and 5 mean the check did not run.

To create an index on a search engine:

rails active_search:index:create INDEX=articles

This creates the fields in the declaration, and is safe to run again. Set analyzers through the engine's own API. Solr needs its core created with Solr's tools first.

On a database adapter, generate the document model and its migration, then migrate:

rails generate active_search:document articles
rails db:migrate

Five ActiveSupport::Notifications events:

store_name is the key from config/search.yml, or :default. total_relation is :equal, :lower_bound or :estimate. Duration comes from the event.

The search payload carries query_length, not the query text. ActiveSearch.filter_attributes lists payload values to filter. It defaults to config.filter_parameters.

• Query syntax. There is no common syntax. Each adapter sends the query string to its engine, with the escaping that engine needs.
• Index management. index:create builds a search engine index. Solr cores and database tables are created outside it. See Commands.
• Analyzers. Stemming, synonyms and other text analysis are set in the engine. On SQLite, see the FTS5 tokenizer under Configuration.
• Advanced features. No aggregations, nested documents, grouping or autocomplete.
• Redis Search. A document whose every field is nil is not stored, so a missing-value filter does not find it. The next write with any field present stores it.

MIT License