Recipe： Text search

Recipe: Text search

The SQL-equivalent question this answers: SELECT * FROM planets WHERE LOWER(name) LIKE '%tatoo%' OR LOWER(description) LIKE '%tatoo%'. A free-form "find-as-you-type" search across a handful of text fields. In SQL you reach for LOWER() + LIKE '%term%' or a full-text index; on DynamoDB neither is available — contains() is the closest primitive, and it's case-sensitive, so you end up storing a pre-folded mirror of each searchable field and filtering on the mirror.

Pattern: declare searchable fields. The built-in prepare step writes a lowercased mirror (<searchablePrefix><field>) on every write. The REST ?search=term query parameter (or options.search programmatically) compiles a contains(<mirror>, :term) OR contains(<mirror2>, :term) … FilterExpression. The term is lowercased before comparison. Results are filtered post-Scan / post-Query — this is scan-bound, not an index.

Why DynamoDB makes you compose

DynamoDB has no native text search. Three workarounds, each with a cost tail:

• contains(attr, :s) in a FilterExpression. Works on strings / sets / lists. Case-sensitive. Runs after the Scan / Query reads the item, so it doesn't reduce RCU — it only reduces the rows returned to the client.
• Pre-fold fields on write. Store a lowercased (or normalised) mirror alongside the canonical field. contains() on the mirror with a lowercased search term is case-insensitive in effect.
• Reach for real search. OpenSearch / Elasticsearch / Algolia for any meaningful text-search workload. DynamoDB contains() is a bridge — fine for small tables, admin search, prefix-match typeahead on a narrow dataset; wrong for anything larger or with multi-term / ranking semantics.

The toolkit picks #2 for the common small-table case: declare searchable, and the built-in prepare step maintains the mirror invisibly.

The pattern

import {Adapter} from 'dynamodb-toolkit';

const planets = new Adapter({
  client: docClient,
  table: 'planets',

  keyFields: ['name'],
  technicalPrefix: '-',

  searchable: {
    name: 1,
    description: 1,
    climate: 1
  }
  // searchablePrefix defaults to '-search-'
});

Three searchable fields → three mirror columns on every item: -search-name, -search-description, -search-climate. Each is the lowercased String(value) of its source.

A written item looks like this on disk:

{
  name: 'Tatooine',
  description: 'Desert planet, twin suns.',
  climate: 'Arid',

  // Built-in prepare step writes these on every full write:
  '-search-name':        'tatooine',
  '-search-description': 'desert planet, twin suns.',
  '-search-climate':     'arid'
}

The built-in revive step strips every field starting with technicalPrefix — the mirrors are invisible to the client.

Wire surface

The bundled REST handler exposes ?search=term on the list route. The framework adapters (koa, express, fetch, lambda) carry the same wire contract.

GET /planets?search=tatoo
GET /planets?search=desert&fields=name,climate
GET /planets?search=arid&limit=20&offset=0

On the wire, ?search=tatoo compiles (against the declaration above) to:

FilterExpression:
  contains(#sr0, :flt0) OR contains(#sr1, :flt0) OR contains(#sr2, :flt0)
ExpressionAttributeNames:
  '#sr0': '-search-name', '#sr1': '-search-description', '#sr2': '-search-climate'
ExpressionAttributeValues:
  ':flt0': 'tatoo'

Three OR'd contains calls — one per declared searchable field. The search term is lowercased before the wire call; no normalization is applied to tokens (no stemming, no unicode folding beyond String.prototype.toLowerCase).

Programmatic use

options.search on any list op runs the same compiler:

await planets.getList({search: 'tatoo', limit: 20});
await planets.getListByParams(
  {TableName: planets.table},
  {search: 'desert', fields: ['name', 'climate']}
);

// Mass ops take it too — e.g. delete every planet matching 'prototype':
await planets.deleteListByParams(
  {TableName: planets.table},
  {search: 'prototype', maxItems: 5000}
);

The options.fields projection is honored — the returned items contain only the projected fields, stripped of mirrors. The mirror is never user-visible.

Restricting which mirrors are searched

By default, options.search fans out OR across every declared searchable field. When the caller restricts fields, the search narrows to the intersection of searchable ∩ fields:

// Searches -search-name only, even though description and climate are declared searchable:
await planets.getList({search: 'tat', fields: ['name']});

Practical use: a typeahead that only searches the name column can pass fields: ['name'] and skip the OR overhead on rows whose description happens to contain the term.

Pairing search with filter

?search= and ?<op>-<field>= compose in the obvious way — both contribute to the FilterExpression and are AND-joined:

GET /vehicles?eq-status=active&search=hybrid&btw-year=-2020-2024

Wire shape:

KeyConditionExpression: ...          (from btw-year, if year is an indexed sort key)
FilterExpression:
  (contains(#sr0, :flt0) OR contains(#sr1, :flt0)) AND (#ff0 = :ffv0)

The three builders (applyFilter, buildSearch, pre-built KeyConditionExpression) each own their own placeholder namespace (#ff/:ffv, #sr/:flt, #kc/:kcv), so they merge cleanly regardless of order.

Case sensitivity and normalization

• Default: case-insensitive. buildSearch lowercases the search term, and the prepare hook lowercased the mirror on write.
• Override: options.caseSensitive: true. Leaves the search term as-is; the mirror is still lowercased on write, so this is mainly useful when the mirror was pre-populated some other way or for tests.
• Mirror values are String(value).toLowerCase(). Non-string fields (numbers, booleans, dates) are stringified first. For complex transforms (stripping diacritics, unicode folding, tokenisation), override hooks.prepare and write the normalised value yourself.

Custom normalisation example:

const stripDiacritics = s => s.normalize('NFD').replace(/\p{Diacritic}/gu, '');

new Adapter({
  client, table: 'planets', keyFields: ['name'],
  technicalPrefix: '-',
  searchable: {name: 1, description: 1},

  hooks: {
    prepare: item => {
      // Built-in prepare ran first; our override runs on the already-prepared item.
      // Rewrite the mirrors with our diacritic-stripped lowercase form.
      const out = {...item};
      if (item.name !== undefined) out['-search-name'] = stripDiacritics(String(item.name).toLowerCase());
      if (item.description !== undefined) out['-search-description'] = stripDiacritics(String(item.description).toLowerCase());
      return out;
    }
  }
});

The built-in step always runs first, so you overwrite rather than compose. Downside: the search term at read time has to go through the same stripDiacritics — wrap the handler or pre-normalise before calling getList({search: …}).

Updating and patching mirrors

The built-in prepare step writes mirrors for any searchable field present in the item on every write, including patches:

• put(item) / post(item) — all mirrors for all present searchable fields are refreshed.
• patch(key, {name: 'New'}, …) — -search-name is refreshed from the patch. Other mirrors are untouched because the other fields aren't in the patch payload.
• patch(key, {description: null}, …) — the toolkit writes -search-description: 'null' (because String(null) === 'null'). If you mean "delete description," use {delete: ['description']} instead; the mirror is not auto-removed. Manage mirror cleanup in a user prepare hook or via the arrayOps / delete surface when you remove the canonical field.

The "mirror isn't auto-removed on field delete" is the one sharp edge. Practical rules:

• If your schema never deletes searchable fields (most don't), the default behaviour is correct.
• If you delete a canonical field with patch({}, {delete: ['description']}), also delete the mirror: patch({}, {delete: ['description', '-search-description']}).
• If you rename a canonical field (description → summary), also rewrite the mirror field set and run a one-time backfill via editListByParams.

Cost model

Every ?search= request is a Scan (or Query, if a KCE narrows the range first) followed by a FilterExpression evaluated per-item. DynamoDB bills per item scanned, not per item returned:

Table size	Scan RCU (eventually consistent)	Search responsiveness
< 1 MB (~ thousands of rows at ~500 B each)	~128 RCU one-off	Cheap. Fine for typeahead, admin search.
1–100 MB	128 – 12,800 RCU per Scan	Expensive on every keystroke. Throttle + cache.
> 100 MB	> 12,800 RCU per Scan	Stop. Move to OpenSearch / Algolia / Athena.

Write-side overhead of searchable declaration:

• One extra attribute per declared field on every write. Lowercased string — typically same byte-length as the source for ASCII, up to 2× for unicode.
• Item size grows by roughly sum(len(searchable[field].value)). For searchable: {name: 1, description: 1} where each is ~50 bytes, ~100 bytes per write. DDB bills writes in 1 KB increments — tiny items still within one WCU won't feel it; items near the 400 KB limit will blow past it.

Rule of thumb: every KB of searchable mirror is a KB that Scans read from disk. Keep the searchable set narrow (typically 1–3 fields).

Alternative: structured filter (<op>-<field>=)

For prefix match on a specific field, ?beg-name=Tat (from the filter URL grammar) is cheaper than ?search=Tat:

• ?beg-name=Tat generates begins_with(#f, :v) in the FilterExpression (one clause, not OR'd). If name is an indexed sort key, it auto-promotes to KeyConditionExpression — billed by matched bytes, not scanned bytes.
• ?search=tat generates contains across every declared searchable mirror, always FE.

Use ?search= when the caller genuinely doesn't know which field contains the term. Use ?beg- / ?ct- when they do.

Alternative: OpenSearch / Algolia / Athena

DynamoDB contains() stops scaling well around single-digit MB of searchable text. Beyond that:

• OpenSearch (or Elasticsearch). Stream DDB changes via DynamoDB Streams → Lambda → OpenSearch Ingestion. Tokenized, stemmed, multi-field, relevance-ranked. Operational overhead: a cluster.
• Algolia / Typesense / Meilisearch. Managed-service alternatives. Same write-path: Stream → Lambda → SaaS. Faster to stand up than OpenSearch, but vendor cost scales with write rate.
• Athena over S3. For admin / ad-hoc analytical search where latency isn't critical. Scheduled export (via DynamoDB S3 export) + Athena query. Cheap; latency is minutes to hours.

The toolkit's searchable is the right choice when the dataset is small, the search is secondary to the main access pattern, and adding another service is over-investment. Anywhere else, plug in the upstream service and skip searchable.

When this pattern fits

• Small tables (< 100 MB of searchable text, ideally much less).
• Searches are a secondary access pattern, not the hot path.
• A handful of text fields (1–5). More fields = wider OR, more RCU.
• Case-insensitive substring match is an acceptable match contract (no relevance ranking, no tokenisation, no fuzzy match).
• Admin / internal UI where occasional Scan cost is fine.

When it doesn't fit

• Customer-facing search on a large catalogue. Relevance, typo tolerance, synonyms — not feasible with contains. Use a real search engine.
• Hot-path search (thousands of queries per second). Every search is a Scan; throttling and partition throughput limits will bite.
• Multi-term, multi-field relevance. DDB has no notion of term weighting or ranking; results are whatever DDB returns in partition-key order, truncated by limit.
• Text longer than a few KB per field. Mirror doubles the attribute size; large text blows up WCU on writes and Scan RCU on reads.
• Exact-match filter on a known field. Use ?eq-field=value via the filter URL grammar instead; it's cheaper and more honest about the cost model.

Related

• Concepts → searchablePrefix and searchable mirror columns.
• Adapter: Constructor options — searchable, searchablePrefix, technicalPrefix.
• Expressions: Filter builder — buildSearch, buildFilterByExample.
• REST core → parsers — parseSearch (with maxLength bound for DoS protection).
• Recipe: Filter URL grammar — the structured-filter complement to ?search=.