Skip to content

Concepts

Jump to bottom
Eugene Lazutkin edited this page Apr 23, 2026 · 4 revisions

Concepts

A one-page reference for the vocabulary the rest of the wiki uses. Two buckets:

  1. Toolkit concepts — things this library introduces.
  2. DynamoDB concepts — AWS terms the toolkit wraps but does not redefine.

Every other page links in here on first use rather than redefining terms locally.

How to read the wiki

The toolkit is a thin helper on top of the AWS JS SDK v3. Most pages open with a See also (AWS JS SDK v3) header pointing at the relevant SDK / DynamoDB docs. Rules of thumb:

  • If the toolkit wraps an SDK concept, read the SDK page first, then read the wiki page for what the wrapper adds.
  • If the toolkit does not wrap something (e.g. composing a KeyConditionExpression, picking a GSI name), the wiki tells you to use the SDK directly and links the reference.
  • All code examples are extracted from the project's test suite — they run against DynamoDB Local. They are not runnable standalone from the wiki; they assume the fixtures in the test harness.

Toolkit concepts

Adapter as the composition root

Adapter is the entry point. One Adapter instance owns a client, a table, a keyFields list, and a hooks bag; it delegates real work to orthogonal sub-exports:

  • dynamodb-toolkit/expressions — build UpdateExpression, ConditionExpression, FilterExpression, ProjectionExpression.
  • dynamodb-toolkit/batch — applyBatch, applyTransaction, getBatch, getTransaction.
  • dynamodb-toolkit/mass — paginateList, iterateItems, writeList, deleteList, copyList, moveList.
  • dynamodb-toolkit/paths — dotted-path get/set/patch utilities.

Use the Adapter for "one entity, one table" flows. Drop to a sub-export when you need a slice of the toolkit without the full composition root.

Additive params (the builder contract)

Every expression builder in this toolkit mutates a params object in place and returns the same object. The design is deliberately additive: a caller accumulates a full DynamoDB Command input by calling one builder after another against the same params.

import {buildUpdate} from 'dynamodb-toolkit/expressions';
import {buildCondition} from 'dynamodb-toolkit/expressions';
import {cleanParams} from 'dynamodb-toolkit/expressions';

let params = {TableName: 'planets', Key: {name: 'Hoth'}};  // mandatory SDK fields
params = buildUpdate({diameter: 13000}, {}, params);        // adds UpdateExpression + names/values
params = buildCondition([{path: 'climate', op: '=', value: 'frozen'}], params);  // adds ConditionExpression
params = cleanParams(params);                               // drops unused names/values

await docClient.send(new UpdateCommand(params));

What each builder adds:

Builder Populates Works with
buildUpdate UpdateExpression, ExpressionAttributeNames, ExpressionAttributeValues UpdateCommand, TransactWriteItems.Update
buildCondition ConditionExpression (AND-merges with existing), ExpressionAttributeNames/Values any Command that accepts ConditionExpression
buildSearch / buildFilterByExample FilterExpression, ExpressionAttributeNames/Values QueryCommand, ScanCommand
addProjection ProjectionExpression, ExpressionAttributeNames any Command that accepts ProjectionExpression

Simple case: pass {} as the starting params, the builder fills in what it needs, use the returned object directly.

Typical case: start with the mandatory SDK fields (TableName, Key, Item, IndexName, ConsistentRead…), pass through every builder you need, call cleanParams at the end, hand the result to the SDK.

params is not a fluent chain (.a().b()). It is a passed-through mutable bag: each builder reads the running name/value counters, appends its own entries, writes the expression field, and returns. Builders can be run in any order; merging is safe because every builder allocates fresh #/: aliases from the existing counters.

cleanParams

The builders allocate every name/value placeholder they might need, but conditional paths (e.g. buildUpdate with an empty patch, or buildCondition with no clauses) can leave entries in ExpressionAttributeNames / ExpressionAttributeValues that don't appear in any expression field. DynamoDB rejects unused entries at request time.

cleanParams(params) removes every unused name/value, leaving only the entries referenced by one of KeyConditionExpression, ConditionExpression, UpdateExpression, ProjectionExpression, FilterExpression. Called automatically by every Adapter method; call it yourself at the end of any builder pipeline you construct by hand.

See src/expressions/clean-params.js for the implementation.

Technical fields (author convention, not toolkit reservation)

Wiki examples frequently show fields like '-t', '-internal', or '-t-name-index'. These are author convention for fields you treat as internal / index-only / not part of the public item shape. The toolkit does not reserve or enforce the - prefix anywhere — you can use any naming convention you want for your own "technical" fields. The convention you see is the one the examples happened to pick; your own code can pick another.

The one exception is searchablePrefix, which the toolkit does reserve for its searchable-mirror-column feature — and it is configurable. See below.

That said, the convention is typically load-bearing inside your own code. Once your hooks key on the prefix, it stops being decorative:

  • A typical prepare(item) hook adds '-t': 1, '-search-<field>': …, and any other technical fields as a write goes out to DynamoDB.
  • A typical revive(rawItem) hook strips every top-level field whose name starts with - before returning to the caller. The wiki examples (Adapter: Hooks, Adapter: Constructor options) use exactly k.startsWith('-') as the filter.

So the prefix is a machine-readable marker that prepare and revive coordinate on. Write-side adds, read-side strips; the application layer never sees a --prefixed field. If you change the convention (to _t, $t, a separate nested meta: {} bag, …), both hooks have to change together.

The hook layer is also the only place the convention is enforced. Any code path that bypasses the hooks has to mirror the convention by hand:

  • raw(item) / Raw<T> writes skip prepare — you must add the technical fields in the caller (see Adapter: Raw marker).
  • writeList(client, table, items) without a mapFn skips prepare too — pass a mapFn that mirrors your Adapter's prepare, or add technical fields upstream.
  • Direct SDK writes (docClient.send(new PutCommand(...))) bypass everything — the caller is fully responsible.

The toolkit's role is to give you the extension points (prepare, revive, searchablePrefix); the convention itself lives in your code.

searchablePrefix and searchable mirror columns

The Adapter can maintain lowercase "mirror" columns for substring filtering. When you declare searchable: {name: 1, climate: 1} on the Adapter and set up a prepare hook that writes '-search-name' / '-search-climate' alongside the real fields, the list-read filter option builds a contains(...) OR contains(...) FilterExpression against the mirror columns.

  • Prefix: default '-search-'. Override via searchablePrefix on the Adapter constructor, or on a per-call basis via ListOptions.prefix and the buildSearch options bag.
  • Only this one prefix is defaulted by the toolkit. Everything else you see as --prefixed in examples is the author's own technical-field convention.

See Expressions: Filter builder and Adapter: Hooks for the full pattern.

Raw<T> — the bypass marker

raw(item) wraps a value in a Raw<T> sentinel that tells the Adapter to skip prepare / prepareKey / validateItem / revive on that one call:

import {raw} from 'dynamodb-toolkit';

// Write exactly this, no hooks applied:
await adapter.put(raw({name: 'X', '-internal': 'kept'}), {force: true});

// Read as-is, skip revive:
const item = await adapter.getByKey(key, undefined, {reviveItems: false});

Use when you already have the canonical DB shape (a re-import, a replication sink, etc.) and don't want hooks to transform it. See Adapter: Raw marker.

Projection and patches — performance levers for read and write

DynamoDB items are often wide: dozens or hundreds of attributes, some of them large (embedded blobs, serialized graphs, long text). Reading and writing the full item every time is wasteful both on the DynamoDB side (read/write capacity, scanned bytes) and on the network (egress bytes back to your client). The toolkit ships two symmetric tools to trim that cost:

  • Projection (read side) — request only the attributes your caller actually needs. DynamoDB reads and returns just the projected slice, not the full item. The ProjectionExpression names exactly which fields come back; everything else stays on the server. Saves read capacity, scan bytes, and network egress. The toolkit's addProjection builds the expression; the Adapter's getByKey / getByKeys / getList / getListByParams take a fields argument that feeds it straight in. Projection is the primary read-performance lever — reach for it any time the caller doesn't need every attribute.
  • Patch (write side) — modify only the fields that changed. The UpdateExpression contains exactly the attributes being set / removed / incremented; DynamoDB leaves everything else alone. Saves write traffic (you ship only the delta, not the whole item) and avoids the read-modify-write round-trip you would need with PutItem. The toolkit's buildUpdate builds the expression; the Adapter's patch(key, patch, options) takes a partial object and wires it through. Patches are the write-side counterpart of projection — same philosophy, opposite direction.

These two tools pair naturally: read a projection of an item, mutate the fields you care about on the client, patch them back. The other fields never leave the server.

checkConsistency — invariants enforced at the database, no round-trip

The toolkit ships a checkConsistency hook plus a transaction auto-upgrade path (Adapter: Transaction auto-upgrade) that together let you enforce invariants on every write without reading the current item back to the client first.

When a partial update touches a subset of fields, the natural worry is "did I just break an invariant this item was supposed to hold?" The naive approach is: read the full item, combine it with the patch in memory, check the invariant in JavaScript, write back. That's two round-trips and a race window. The toolkit approach is: the checkConsistency(batch) hook inspects the outgoing write descriptor and returns extra makeCheck descriptors — each a ConditionExpression that DynamoDB evaluates server-side. If all checks pass, the write commits atomically as part of the same TransactWriteItems; if any fail, the whole transaction is rejected with TransactionCanceledException and the database is unchanged.

Net effect: the invariant check happens at the database level, without transferring anything to the client and back. The caller sees either the write commit cleanly or a single rejection — never a torn update or a stale snapshot. Use it for referential integrity ("parent must exist"), optimistic locking (revision / version checks), quota enforcement, tenant scoping, or any rule where the answer is "it depends on other rows".

Hooks

Per-Adapter customization lives in the hooks bag. See Adapter: Hooks for full signatures — at a glance:

Hook When What it does
prepare(item, isPatch) Every write Transform the client-side shape into the DB-side shape (add mirror columns, technical fields, etc.).
prepareKey(key, index) Every keyed op Rewrite a caller-supplied key (e.g. coerce types).
prepareListInput(example, index) getList Build the Query / Scan input from an example + index name.
updateInput(input, op) Every write (post/put/patch/delete) Last-chance mutation of the built params before SDK dispatch.
revive(item, fields) Every read Transform the DB-side shape back to the client-side shape (strip mirror columns, parse Sets, etc.).
validateItem(item, isPatch) Every validated write Throw on invalid items before hitting DynamoDB.
checkConsistency(batch) Every single-op write Return extra makeCheck descriptors that DynamoDB evaluates atomically with the write — invariant enforcement at the DB level, no client round-trip. See Adapter: Transaction auto-upgrade.

Indirect indices

A sparse GSI (one that projects KEYS_ONLY or a small subset) plus an indirectIndices entry tells the Adapter to do a second-hop read. The Adapter queries the GSI to get the primary keys, then does a BatchGetItem against the base table to hydrate full items. Callers get a single-shot list read with the GSI's ordering and the base table's full projection.

See Adapter: Indirect indices for the full pattern, including ignoreIndirection to opt out on a per-call basis.

DynamoDB concepts

Terms that belong to DynamoDB itself. These short definitions are here so every wiki page can link in rather than repeat them.

Primary index — the base table itself

The table's primary key (partition key + optional sort key) is the primary index. Every DynamoDB table is a key-value store sorted by the primary key; there is no separate "primary index" object to create. GetItem / Query / Scan on the table name target this index.

Everything in the rest of this section — GSI, LSI, projection, KeyConditionExpression — is about adding secondary access paths on top of the primary key. If your read pattern fits the primary key directly (e.g. "get item by id"), you don't need any of it.

Why reach for a secondary index

The primary key gives you exactly one way to look items up — the partition key plus (optionally) the sort key. Real apps need more:

  • A different ordering. Primary key orders by partition then sort. Want chronological? Need a GSI with created_at as sort key. Want alphabetical by name? Need a GSI with the name as sort key. The base table can have only one ordering; each GSI/LSI adds another.
  • A different access predicate. Look items up by email, by tenant, by status, by tag — anything that isn't the partition key. Each of these is a GSI whose partition key is the field you want to query by.
  • Compound/structured sort keys that support BEGINS_WITH(...) and range queries. Concatenate multiple attribute values (often with a separator like #) into a single sort-key string; query for begins_with(sortKey, 'active#2026-04-') to get "all active items from April 2026". This is the workhorse single-table-design pattern — one GSI can serve many access patterns if the sort key is designed around the prefixes you want to query.
  • Sparse indexes — include/exclude items by criteria. A secondary index only contains items that have a value for the index's key fields. Items where those fields are missing (undefined / not-set) are silently omitted from the index. This gives you "only items in state X" as a cheap read: populate the index key field conditionally in your prepare hook; items that don't qualify never enter the index, so a full scan of the GSI returns exactly the qualifying subset. Pairs well with the toolkit's indirect-indices pattern when the index projection is KEYS_ONLY.

Worked examples of each pattern:

  • Chronological GSI — created_at as GSI partition key means you can Query the whole table by time buckets (usually with a coarse partition like date-or-month) and get sorted results. Without it, listing "latest N items" requires a Scan with in-memory sort.
  • Email lookup GSI — base table keyed by user_id; GSI with email as partition key. Login flow: getByKey({email: 'x@y'}, …, {params: {IndexName: 'email-index'}}) → one point read on the GSI returns the matching user.
  • Compound sort key GSI — store 'tenant#status#name' as the GSI sort key (populated by prepare). One GSI serves: "all items for tenant T" (begins_with('T#')), "active items for tenant T" (begins_with('T#active#')), "active items alphabetically" (range query on the active# prefix).
  • Sparse tombstone GSI — prepare sets '-t': 1 on every active item and omits it (or sets null) for archived ones; GSI uses '-t' as partition key. The GSI holds only live items. A full scan of the index is the "list all non-archived" query, no filter needed.

Start picking the projection — KEYS_ONLY / INCLUDE / ALL — based on what the read side needs. See Projection below.

GSI — Global Secondary Index

A secondary index with its own partition/sort key, replicated asynchronously, with independent read/write capacity. Supports any primary key schema (independent of the base table). Can be created and deleted at any time. See Global Secondary Indexes (AWS docs).

LSI — Local Secondary Index

A secondary index that shares the base table's partition key but chooses a different sort key. Created only at table-creation time — can't be added later. Strongly-consistent reads are available (unlike GSI, which is eventually consistent). Use when you need a second sort order within each partition. See Local Secondary Indexes (AWS docs).

Projection (index attribute copy)

What attributes the secondary index copies from the base table. Three options, picked at index-create time:

  • KEYS_ONLY — only the base-table key and the index key. Cheapest; best for indirect indices.
  • INCLUDE — the keys plus a declared list of extra attributes.
  • ALL — the full item on every index write. Most expensive, fastest for direct reads.

See Attribute projections in secondary indexes (AWS docs).

KeyConditionExpression

The partition-key-plus-optional-sort-key predicate on a Query. Must reference the partition key with =; the sort key can use =, range ops, or BEGINS_WITH(...). This is not the same as ConditionExpression — you cannot filter arbitrary attributes here.

The toolkit does not ship a builder for KeyConditionExpression. Compose it yourself following the AWS Query API reference, or feed the Adapter an example via Adapter: Hooks#prepareListInput.

ConditionExpression

A predicate that gates a Put / Update / Delete (fail-on-mismatch semantics). The toolkit ships buildCondition for composing these. See Condition expressions (AWS docs).

FilterExpression

A predicate applied after the server-side Query / Scan returns matching items. Does not reduce read capacity — the scanned bytes still count. The toolkit ships buildSearch / buildFilterByExample. See Filter expressions (AWS docs).

ProjectionExpression

The list of attributes a read returns. The toolkit ships addProjection. See Projection expressions (AWS docs).

UpdateExpression

The SET / REMOVE / ADD / DELETE mutation on an UpdateCommand. The toolkit ships buildUpdate. See Update expressions (AWS docs).

BatchWriteItem / BatchGetItem

Non-atomic bulk endpoints. Per-call limits are asymmetric: BatchWriteItem caps at 25 items, BatchGetItem at 100. BatchWriteItem is the outlier — the other three DynamoDB bulk APIs (BatchGetItem, TransactWriteItems, TransactGetItems) are all 100. TransactWriteItems was raised from 25 to 100 in September 2022; BatchWriteItem never got the same treatment. See Batch and transactions → Per-call item limits for the full table. The toolkit's applyBatch chunks transparently and retries the SDK's UnprocessedItems with exponential backoff. See BatchWriteItem (AWS docs) and BatchGetItem (AWS docs).

TransactWriteItems / TransactGetItems

Atomic bulk endpoints. Per-call limit: 100 actions. The toolkit does not chunk transactions — atomicity rules that out. See TransactWriteItems (AWS docs) and Batch and transactions.

DocumentClient and marshalling

@aws-sdk/lib-dynamodb's DynamoDBDocumentClient.from(...) wraps the low-level client to marshal plain JS values (numbers, strings, booleans, arrays, Sets, nested objects) to/from DynamoDB's wire format automatically. Construct with {marshallOptions: {removeUndefinedValues: true}} for v2-parity behavior. See the lib-dynamodb package.

The toolkit assumes you are using the DocumentClient throughout — every example and type does. If you wire a raw DynamoDBClient in by mistake, marshalling errors surface immediately.

Glossary: cross-references

🔍 Search the wiki

dynamodb-toolkit

Start here

Guides

Adapter

Expression builders

Batch / transactions / mass / paths

REST surface

Framework adapters

Recipes

History

Clone this wiki locally