Search Query System

Search Query System

Part of FreeREG-Core-Records.

Models: app/models/search_query.rb, search_result.rb, search_record.rb (index_hint), my_saved_search.rb, search_statistic.rb · Controllers: search_queries_controller.rb, search_records_controller.rb

root is search_queries#new: the search form is the FreeREG home page. Every search a researcher runs is persisted as a SearchQuery document along with a copy of its results. The results page, re-sorting, "next/previous record" navigation and the statistics reports all read from that stored copy, not from a fresh query.

Request flow

sequenceDiagram
    participant B as Browser
    participant C as SearchQueriesController
    participant Q as SearchQuery
    participant M as MongoDB search_records

    B->>C: POST /search_queries (form)
    C->>Q: new(params) + adjust (strip names, expand YKS/CHI)
    Q->>Q: validate
    C->>Q: search
    Q->>Q: search_params → index_hint
    par primary
        Q->>M: find(params).hint(idx).limit(500)
    and secondary date
        Q->>M: find(params with secondary_search_date).hint(idx2)
    end
    Q->>Q: persist_results → search_result.records (full hashes)
    Q->>Q: search_ucf (if place search & ucf_support)
    C-->>B: redirect /search_queries/:id
    B->>C: GET show
    C->>Q: get_and_sort_results_for_display (filter, sort in Ruby)

Loading

• Searching doesn't require login.
• Composite county picks are expanded in the controller: YKS → ERY, NRY, WRY; CHI (Channel Islands) → ALD, GSY, JSY, SRK.

Validation rules (what the user sees as form errors)

• A surname is required, unless the search has a forename and a county and a specific place (name_not_blank).
• No county selected → record type and a date range are both required (county_is_valid). At most 3 counties, except the four Channel Islands together.
• Start and end year must both be given, be valid int32s, and satisfy start ≤ end.
• Wildcards (*, ?) are only allowed when specific places are selected, need at least two literal letters before the first wildcard, and can't be combined with soundex ("fuzzy").
• "Search nearby places" needs a place to start from.

Building the MongoDB query (search_params)

1. Names (name_search_params): one $elemMatch on search_names so the first and last name must match the same person:
   • exact: {first_name, last_name, type: {$in: allowed_name_types}}, all lower-cased. "No surname" searches match last_name: nil.
   • fuzzy: the same on search_soundex with Text::Soundex codes.
   • wildcard: names converted to anchored regexes.
   • allowed_name_types is ['p'], plus 'f' if "include family members" (inclusive), plus 'w' if "include witnesses". Filtering by type in the query (not after it) means the 500 cap applies to already role-correct candidates (issue #1403).
2. Place: place_id: {$in: [...]} when places are selected (plus nearby places, below), otherwise chapman_code: {$in: chapman_codes}.
3. Record type: the selected one, or $in all three.
4. Dates: search_date: {$gte: DateParser.start_search_date(start_year), $lt: DateParser.end_search_date(end_year)}.
5. Embargo: $nor: [{embargoed: true, release_year: {$gt: <this year>}}].

Index hinting

SearchRecord.index_hint(params) picks an index by name from the REG_PLACE_INDEXES / REG_CHAPMAN_INDEXES / REG_BASIC_INDEXES constants (chosen by whether the query has place_id, chapman_code, or neither). It scores each candidate by how many of its leading fields appear in the query, stopping at the first missing one, and takes the best. Regexes that don't start with ^ don't count as indexable. The chosen name and the planner's actual winning plan (winning_plan_index_name, via explain) are both saved on the query (search_index, search_index_winning_plan), which is useful for spotting a bad hint.

The secondary-date search

Because a record can match a date range on search_date or secondary_search_date (Search-Record-Indexing), fetch_records_with_secondary_date runs two queries in parallel threads when there's a date range: the normal one, and a copy with search_date renamed to secondary_search_date (with its own hint). Secondary hits are de-duplicated against the primary ones, capped so the total stays ≤ MAXIMUM_NUMBER_OF_RESULTS_FREEREG (500), and stored through persist_additional_results. If the secondary query fails, the error is logged and the primary results are still returned.

Every query uses max_time_ms(config.max_search_time). The controller's rescue_from Mongo::Error::OperationFailure / Timeout::Error → search_taking_too_long sends the user back to the form with their criteria. That also catches any other Mongo operation failure, e.g. a bad index hint, so the user sees "taking too long" for errors that have nothing to do with time.

Persisting results

persist_results stores every matching record's full hash in the embedded SearchResult (search_result.records: a Hash of id → record hash), along with result_count, runtime and day. Along the way:

• For FreeREG, each hit's entry → batch → register → church → place chain is checked in batches (valid_freereg_entry_ids_for_result_hashes). A search record with a broken chain is deleted from search_records on the spot and left out of the results. This is how orphans left by batch removal (see Batch-Records-Freereg1CsvFile-and-Entry) eventually disappear, and it also means a search can have side effects.
• Hits with a blank search_date get it backfilled (add_search_date_when_absent).

search_queries is therefore a large, fast-growing collection. It's pruned by rake build:delete_search_queries[limit] (cron: lib/tasks/scripts/delete_old_search_queries.sh), which destroys queries older than config.days_to_retain_search_queries, one document at a time, oldest first. Don't be misled by rake freereg:clean_search_queries: despite its name it only recalculates SearchStatistic/SiteStatistic, and its delete_all line is commented out.

Displaying results

show → get_and_sort_results_for_display, all in Ruby on the stored hashes:

1. filter_name_types: re-checks each record has a name of an allowed type matching the query (standard / fuzzy / wildcard rules). This is a second pass over what the $elemMatch already did.
2. filter_embargoed: drops records still under embargo. Also already done in the query, but this catches records whose state changed after the search was stored.
3. sort_results: by order_field (search_date default, record_type, chapman_code, location, or name) and direction. reorder just flips these on the stored query and re-renders. It doesn't search again.
4. UCF results (ucf_results) are shown as a separate list, de-duplicated against the main list.

Other actions on a stored query: show_print_version, show_query, about, remember (→ MySavedSearch for logged-in users), and narrow/broaden, which create a new query with radius_factor halved/doubled. can_be_broadened? is hard-coded to false, so only narrowing is offered.

Nearby places (radius search)

radius_factor (default 101) isn't a distance. Place#places_near runs a $geoNear on places.location, filters data_present: true, and $limits to radius_factor places. So "nearby" means "the N nearest places with data, however far away". The query's own place is shifted off the front of the list.

UCF (wildcard-transcription) results

When config.ucf_support is true and the search is limited to specific places and it isn't a nearby search, search_ucf loads the ids in each selected place's ucf_list (Search-Record-Indexing), fetches those records, and keeps the ones whose UCF names regex-match the query (filter_ucf_records). The ids are saved in search_result.ucf_records. This is how Sm_th in a register can match a search for Smith.

Viewing a record (SearchRecordsController)

• Routes: search_records/:id (id must be a 24-hex ObjectId), search_records/:id/:friendly (SEO slug from SearchRecord#friendly_url: names + date), show_print_version, show_citation.
• The page renders the entry (Freereg1CsvEntry#display_fields, ordered by record type and, for DEF batches, the batch's column order) plus location and transcriber credit. Previous/next links come from SearchQuery#next_and_previous_records using the stored, sorted result set in session[:query].
• A record reached directly (without a stored query) still renders. Navigation links just don't appear.

Search statistics

• SearchQuery itself records runtime, runtime_additional, runtime_ucf, result_count, search_index, day for every search.
• SearchStatistic (via rake freereg:calculate_search_queries) aggregates those per day.