Recipe： Filter URL grammar

Recipe: Filter URL grammar

The SQL-equivalent question this answers: expose typed WHERE clauses through a REST query string. In SQL this is SELECT * FROM vehicles WHERE status = 'active' AND year BETWEEN 2020 AND 2024 AND state IN ('TX','CA','NY') — one SQL statement, several ad-hoc predicates per request. On the wire you want the same: ?eq-status=active&btw-year=-2020-2024&in-state=,TX,CA,NY. The trick is validating the field + op allowlist, coercing values to real types, and routing the predicate to KeyConditionExpression when possible (cheap) and FilterExpression otherwise (bills for scanned bytes, not matched bytes).

Pattern: declare filterable with per-field op allowlists + optional type overrides; parseFilter extracts clauses from the query string; applyFilter validates + coerces + auto-promotes index-compatible clauses to the KCE; the rest land in the FE. The REST handler wires it automatically — the URL grammar is the public surface.

Why DynamoDB makes you compose

Three gaps in the SDK make a "safe, typed filter from URL" nontrivial:

• No built-in op allowlist. Raw SDK accepts anything valid in the expression dialect — if you naively forwarded client input you'd expose contains, attribute_exists, and every comparator on every attribute, including internal ones.
• No value coercion. Query strings are always string-typed. A year = 2024 filter that compares DynamoDB N to JS '2024' silently returns zero rows (types must match byte-for-byte in DynamoDB).
• KCE vs FE is a cost decision, not a syntax decision. DynamoDB bills by scanned bytes, not returned bytes. A sort-key begins_with in the KCE reads exactly the key-range you asked for; the same predicate in the FE reads the whole partition and post-filters. The DDB expression dialect lets you put it in either slot — getting the cheap one takes work.

The toolkit does all three: filterable is the allowlist, filterable[field].type (or keyFields / indices inference) is the coercion contract, and applyFilter auto-promotes the clauses DynamoDB will actually accept in the KCE.

The grammar

One query parameter per clause. Key shape: <op>-<field>=<value>. Ops:

op	Meaning	Arity	Notes
eq	=	scalar	Can auto-promote to partition-key KCE (equality only).
ne	<>	scalar	Always FE — DDB rejects <> on key attributes in KCE.
lt / le / gt / ge	< / <= / > / >=	scalar	Auto-promotes on the sort key; on a plain attribute, lands in FE.
in	IN (...)	array (first-char delimiter)	Always FE — DDB doesn't support IN in KCE.
btw	BETWEEN a AND b	2-array (first-char delimiter)	Auto-promotes on the sort key.
beg	begins_with	scalar	Auto-promotes on the sort key. Rejected by DDB on non-string attributes.
ct	contains	scalar	Always FE. Works on string / set / list attributes.
ex / nx	attribute_exists / _not_exists	none	Always FE. Value is ignored (?ex-email= is the canonical form).

First-character delimiter for multi-value ops (in, btw): if the first character of the value is alphanumeric, the delimiter is ,; otherwise the delimiter is the first character itself. This lets you carry values that contain commas (using | or ; or any non-alphanumeric separator) without URL-encoding.

?in-state=,TX,CA,NY         → ['TX', 'CA', 'NY']
?in-state=|TX,foo|CA,bar    → ['TX,foo', 'CA,bar']
?btw-year=-2020-2024        → ['2020', '2024']   ← '-' is the delimiter; then coerced to number
?btw-price=;10;20           → ['10', '20']

The first character is consumed as the delimiter and dropped from the first value. Pick a delimiter that doesn't appear inside any value.

The pattern

Declare filterable with per-field op allowlists. Two shapes:

import {Adapter} from 'dynamodb-toolkit';

const vehicles = new Adapter({
  client: docClient,
  table: 'vehicles',

  keyFields: ['state', {name: 'vin', type: 'string'}],
  structuralKey: '-sk',
  technicalPrefix: '-',

  indices: {
    'by-status-year': {
      type: 'gsi',
      pk: 'status',
      sk: {name: 'year', type: 'number', width: 4},
      projection: 'all'
    }
  },

  filterable: {
    // Shape 1: array of ops. Type inferred from keyFields / indices.
    // `state` is a keyField (`'string'`), so values are passed through.
    state:  ['eq', 'in'],

    // Shape 2: `{ops, type}`. Explicit type overrides inference. Required
    // for plain data fields that DDB stores as Number / Binary — otherwise
    // the default `'string'` coercion silently mismatches the stored type.
    year:   {ops: ['eq', 'ge', 'le', 'btw'], type: 'number'},

    // Fields that are the pk/sk of a declared index can skip `type:` —
    // resolved from `indices`. Here `status` is the GSI pk (string).
    status: ['eq'],

    // Plain data attribute. No type → 'string'. Only `contains` allowed.
    description: ['ct'],

    // No-value ops for presence checks.
    priceOverride: ['ex', 'nx']
  }
});

Type-resolution precedence: explicit {type: …} → keyFields entry → indices (pk then sk) → fallback 'string'. For numeric DDB attributes, always declare {type: 'number'} — ?eq-year=2024 against stored N: 2024 with default string coercion will return zero rows. (The bug is invisible — no error, just empty results.)

How it wires to the REST handler

The bundled createHandler (and each framework adapter) plugs the grammar in automatically:

import {createServer} from 'node:http';
import {createHandler} from 'dynamodb-toolkit/handler';

createServer(createHandler(vehicles)).listen(3000);

Any <op>-<field> keys in the query string become clauses. Routes that accept filters:

Route	Effect
GET /?<op>-<field>=…	Filters the list on read. applyFilter runs before the paginated read.
DELETE /?<op>-<field>=…	Filters the resumable delete. Maps to deleteListByParams.
PUT /-clone?<op>-<field>=… / PUT /-move?<op>-<field>=…	Filters the source selection for the resumable copy/rename.

Unlisted fields throw BadFilterField → mapped to 422 by policy.mapErrorStatus. Unallowed ops on a listed field throw BadFilterOp → 422.

The auto-promotion rules

applyFilter walks each clause in URL order and decides "KCE or FE" per rule:

1. Target index. If params.IndexName is set (via ?sort=<field> resolved against a declared index, or explicit Query-shape params), the target is that index's pk/sk. Otherwise: base table → keyFields[0].name is pk, structuralKey.name is sk.
2. PK match. A clause {field: <pk>, op: 'eq'} promotes to the KCE. First such clause wins; a second eq-pk=… falls through to FE (DDB only accepts one pk equality).
3. SK match. A clause {field: <sk>, op: <eq|beg|btw|lt|le|gt|ge>} promotes. First such clause wins; a second falls through to FE.
4. Everything else — non-eq ops on the pk, ne / ct / in / ex / nx on anything, comparators on non-key attributes — lands in FE.

The promotion is silent. You can't tell from the wire whether a filter hit the KCE fast path or the FE slow path — only the billed RCUs do. This is a known rough edge (surfacing "KCE slot occupied, fell through to FE" as a result hint is a queued follow-up); for now, if your access pattern cares about the cost, verify with a tool like returnConsumedCapacity.

Worked examples

Cheap listing — KCE-only

GET /vehicles?eq-status=active&btw-year=-2020-2024&sort=year

• ?sort=year resolves to index by-status-year via sortableIndices / findIndexForSort.
• eq-status=active matches the GSI partition key → KCE.
• btw-year=-2020-2024 matches the GSI sort key → KCE.
• Both clauses land in the KCE. DynamoDB plans a Query against by-status-year reading exactly the status=active partition narrowed to year BETWEEN 2020 AND 2024. RCU scales with matched bytes.

Emitted on the wire:

TableName: 'vehicles'
IndexName: 'by-status-year'
KeyConditionExpression: (#ff0 = :ffv0) AND (#ff1 BETWEEN :ffv1 AND :ffv2)
ExpressionAttributeNames: {'#ff0': 'status', '#ff1': 'year'}
ExpressionAttributeValues: {':ffv0': 'active', ':ffv1': 2020, ':ffv2': 2024}

(The #ff/:ffv prefixes are applyFilter's private namespace — they co-exist with other builders' placeholders and can't collide.)

Mixed KCE + FE

GET /vehicles?eq-status=active&btw-year=-2020-2024&in-state=,TX,CA&ex-priceOverride=&sort=year

• eq-status + btw-year → KCE (as above).
• in-state → FE (IN not supported in KCE).
• ex-priceOverride → FE (attribute_exists not a KCE op).

DynamoDB reads the KCE-narrowed range, applies FE post-fetch. RCU scales with the KCE-matched bytes (including rows that the FE later filters out).

FE-only — base table list, no indices hit

GET /vehicles?ct-description=sport&in-state=,TX,CA

Nothing matches a key attribute → full base-table scan + FE. Expensive. Fine for small tables or admin UIs; avoid on hot paths. Typically a sign you should add an index or reshape the access pattern.

Programmatic use

parseFilter and applyFilter are both importable — the URL grammar isn't the only consumer. Any structured filter surface (RPC, GraphQL, internal admin CLI) can plug in directly:

import {parseFilter} from 'dynamodb-toolkit/rest-core';

// In a custom handler — clauses from any source can feed applyFilter:
const clauses = parseFilter(query);          // from the URL
// or:
const clauses = [
  {field: 'status', op: 'eq', value: 'active'},
  {field: 'year', op: 'btw', value: ['2020', '2024']}   // strings — applyFilter coerces
];

const params = {TableName: vehicles.table, IndexName: 'by-status-year'};
vehicles.applyFilter(params, clauses);
const result = await vehicles.getListByParams(params);

applyFilter mutates params in place (consistent with every other builder in dynamodb-toolkit/expressions) and returns it for chainability. Values carry as strings from the URL layer; coercion happens inside applyFilter per the resolved type.

Composition with options.filter

getListByParams / mass-ops accept options.filter: FilterClause[] that get threaded through applyFilter on every pagination call:

// Mass edit — same filter surface, applied per page:
await vehicles.editListByParams(
  {TableName: vehicles.table, IndexName: 'by-status-year'},
  vehicle => ({...vehicle, certified: true}),
  {
    filter: [
      {field: 'status', op: 'eq', value: 'active'},
      {field: 'year', op: 'ge', value: '2020'}
    ],
    maxItems: 5000
  }
);

Clauses compose cleanly with options.search (free-form substring), options.asOf (scope-freeze), and any pre-built KeyConditionExpression in params. Every builder (applyFilter, buildSearch, buildKeyCondition) AND-merges into existing FilterExpression / KeyConditionExpression strings without disturbing them.

Cost model

Per request:

Shape	KCE bytes	FE evaluated	RCU scale	Typical case
All-KCE (eq on pk, range on sk)	matched only	0	bytes matched	Full index utilisation. ?eq-status=A&btw-year=-2020-2024 on a (status, year) GSI.
KCE + FE	KCE range	in-range rows	KCE bytes	Common case. The FE cuts some rows post-fetch but you paid for the KCE range.
FE-only (base-table list with filters)	—	whole partition	every RCU in the range	Avoid on hot paths. Add an index or reshape.

The practical rule: if your filter has an equality on the pk (or the first pk of a declared index), and a range/beg clause on the sk, you're on the cheap path. Any deviation drops you to "scans partition + post-filters" territory.

When this pattern fits

• REST / GraphQL / RPC surface where queries come from clients.
• Filter dimensions are known in advance and fit on a handful of attributes.
• You control the index design so the common access patterns auto-promote to KCE.
• 422 on bad field / op is an acceptable error contract (i.e., client mistakes, not runtime data issues).

When it doesn't fit

• Fully ad-hoc analytical queries. Users need to filter on any attribute with any op — DDB is the wrong store; move the analytics to Athena / OpenSearch / Snowflake.
• Complex boolean composition. applyFilter AND-joins every clause. A AND (B OR C) isn't expressible in the URL grammar. Compose KeyConditionExpression + FilterExpression by hand when you need it.
• Operators outside the allowlist. size(list) > N, arithmetic predicates — hand-write the FE and skip applyFilter.
• Cross-partition range queries without an index. ?btw-year=-2020-2024 with no (pk, year) index devolves to a FE-only Scan. Add a GSI or pay for it.

Debugging checklist

• Filter returns zero rows when you expect hits: check filterable[field].type. If you stored N: 2024 and filtered with default string coercion, the bytes don't match.
• BadFilterField in 422: field not in filterable. Add it or check for a typo.
• BadFilterOp in 422: op not allowlisted for that field. Extend filterable[field] or pick a different op.
• Filter seems slow: print returnConsumedCapacity: 'INDEXES' on the Query params. If scanned bytes >> matched bytes, you're on the FE slow path — verify the KCE auto-promotion assumption for your index.
• in- / btw- value parses weird: remember the first-character delimiter rule. ?in-tag=a,b,c parses as one value 'a,b,c' if you forget the leading delimiter.

Related

• Concepts → Filter URL grammar — the one-paragraph conceptual summary this recipe expands.
• Adapter: Constructor options — filterable, searchable, searchablePrefix, indices.
• Expressions: Filter builder — lower-level buildFilterByExample and related primitives.
• REST core → parsers — parseFilter, parseSearch, parseFields.
• Recipe: Text search — the ?search= complement to structured filters.
• Recipe: Keys-only GSI with runtime projection — the index-cost conversation this recipe assumes.