Search

Search is a word-prefix index, not a substring or full-text index. Knowing that is the difference between "the search is broken" and "the search is doing exactly what the index can do".

What the index contains

EventIngestor.stage_event normalises the line before tokenising it:

eventString.downcase!
eventString.gsub!(/([^a-zA-Z0-9 \-_:@\*\/.])/," " )
split_text = eventString.split(/\s+/)

So an indexed word contains only lowercase letters, digits, and - _ : @ * / . — anything else became a word separator. events.hunks still holds the original, unmodified line, which is what Displays and the event views render. The index and the display are built from different strings; that is the single most common source of "the search missed something I can see on screen".

How a query is executed

Event.search(search_string, time_range, offset, limit):

1. Split search_string on whitespace and downcase it. Each token is a term.
2. For each term, find every words.text that starts with the term (text like 'term%' escape '\'). % and _ in your term are escaped to literals, so foo_bar cannot match fooXbar.
3. Relationally divide: keep only events whose events_words rows cover every term. Counting distinct matching word_ids instead of distinct terms would break whenever one term is a prefix of another word on the same event — "test" also matches "testservice" — which silently hides events.
4. Page the result with offset/limit, ordered oldest-first.

An empty or whitespace-only query returns []. A non-Range time argument raises ArgumentError rather than binding as the string '0' and matching everything, which is what it used to do.

What this means in practice

You search for	You get	You do not get
sshd	any word starting with sshd — sshd:, sshd2	a line containing xsshd
failure	failure, failures	fail — a prefix match runs term → word, not word → term
connection refused	events containing both words, in any order	an exact phrase, or that the two are adjacent
user=bob	nothing, ever	— = is not in any indexed word; it was replaced by a space, so the words are user and bob

Terms are ANDed. There is no OR, no NOT, and no proximity. To narrow what the dashboard shows, use Exclusions; to change how a line renders, use Displays.

Because the query is downcased but not punctuation-normalised, a term holding any character outside - _ : @ * / . silently matches nothing. Strip it or split the term.

Word text is truncated at 255 characters on ingest (the column is a fixed-width string), so very long tokens collide on the truncated form and are searched as one word.

Performance notes

• The time window is not optional. A search over all history is not something this index answers cheaply; events_words is partitioned by generated on Postgres, so a bounded window also prunes partitions.
• Each term's word set is resolved exactly once. It used to be resolved twice — once to restrict events_words and again inside HAVING — which made the planner scan words 2n times. Measured on 280k words / 6M events_words rows with a term matching 60k words: 534ms → 394ms for one term, 1386ms → 1067ms for three.
• The term → word_id lookup stays a subquery rather than joining words into the scan: joining drives events_words off the time window and then re-filters, a full scan of every join row in that window. Restricting on word_id lets Postgres use index_events_words_on_word_id_and_generated_and_event_id as a covering index, which is why that index is composite and why it is the only one events_words carries. A standalone index on generated existed until 2026 and was dropped: EXPLAIN never chose it, and on Postgres the composite index recorded idx_scan=2 against its idx_scan=0. Search timings are identical with and without it (5.94 ms vs 5.95 ms).
• Prefix search was silently returning nothing on Postgres under an en_US.UTF-8 collation. The narrowing trick it relied on — a [term, term + U+10FFFF) range — is sound only under a byte-wise collation. Under glibc en_US.UTF-8, strcoll sorts U+10FFFF before ASCII letters, so the bound falls below every longer word and the range matches nothing: measured on 280k words, the term sshd matched 60 000 words by LIKE and 0 by that range. The range is gone; the LIKE 'term%' alone is used, served by index_words_on_text_pattern (varchar_pattern_ops, byte-wise, and therefore collation-blind). It cannot be repaired rather than removed either: that collation gives trailing punctuation no primary weight, so the strings above 'foo-' are unbounded — checked exhaustively over the 94 printable ASCII characters the tokenizer can emit, and incrementing the last character is an unsound bound for 35 of them. Do not reintroduce the range. This is exactly the class of bug a single-engine test suite cannot see; the suite now runs only on the engine production uses. See Development.

The search form

The bar, timeframe, and result limit sit above the live feed; results replace the feed in place. Screenshot from a seeded demonstration dataset; see Interface.

Unticked, Timeframe Search searches the relative window from the Timeframe dropdown. Ticked, it searches the absolute start/end range. When the date fields cannot be read the result line says so, rather than returning 0 events match for a window the operator never asked for.

This toggle was broken until recently (RECOMMENDATIONS §2.3). The checkbox is a Rails check_box, which always submits timesearch_enabled[enabled]=0 when unticked, and EventsController#search branched on the presence of that key — a Hash, so always truthy. Every search submitted from the form took the absolute-date branch with the datetime_select defaults, a now→now window, and reported 0 events match. Fixed in 7e0f435; the screenshot above predates the fix and was taken by ticking the box.

Stored searches

A Search row is a saved search string — a bookmark. There is no time window stored: searches carries string, user_id, description and short_description and nothing else, so the timeframe is chosen again when the search is re-run. The query is still executed by Event.search; nothing is cached.