Skip to content

Recipe： Cascade subtree operations

Jump to bottom
Eugene Lazutkin edited this page Apr 23, 2026 · 1 revision

Recipe: Cascade subtree operations

The SQL-equivalent question this answers: DELETE FROM rentals WHERE state = 'TX' AND facility = 'Dallas' with ON DELETE CASCADE — removes the Dallas facility row and every vehicle underneath in one statement. Or INSERT INTO rentals SELECT … to duplicate a facility subtree. Or the atomic move-subtree-to-a-new-parent operation. In SQL the database enforces referential integrity and cascades for you; in DynamoDB there are no foreign keys, no triggers, no CASCADE. The toolkit ships six primitives that compose the pattern — subject to "not atomic, resumable on crash" disclaimers you need to read before shipping.

Pattern: declare relationships: {structural: true} on an Adapter whose composite keyFields already defines a parent-child hierarchy (enforced by the structuralKey composition). Call deleteAllUnder / cloneAllUnder[By] / moveAllUnder[By] at the subtree root; the toolkit paginates the Query, emits per-item writes, and returns a MassOpResult the caller can resume.

Why DynamoDB makes you compose

DynamoDB has no referential integrity layer. A parent row and a child row have no relationship DynamoDB understands; deleting the parent leaves orphaned children, moving the parent doesn't move children. Three practical consequences:

  • No native cascade. DELETE FROM rentals WHERE state = 'TX' AND facility = 'Dallas' is six lines of SQL; in DynamoDB it's one Query to find the rows plus N delete calls. Getting the sequencing right (descendants first? or root first? how do you handle a crash mid-way?) is the hard part.
  • Subtrees don't transact. A TransactWriteItems can hold ≤100 actions. A subtree with more than 100 items can't be atomically rewritten in one call.
  • The structural key is the only hierarchy signal. The toolkit composes {state, facility, vehicle} into a single sort key (TX|Dallas|1HGC…) via structuralKey. "Everything under Dallas" is begins_with(_sk, 'TX|Dallas|') — a structural Query. No FK → no schema-level knowledge that vehicle.facility is the parent.

The relationships: {structural: true} declaration is the toolkit's way of saying "yes, I mean it — treat the composite sk as a parent-child hierarchy." Without the declaration, the cascade primitives throw CascadeNotDeclared — you have to opt in explicitly because composite keyFields don't always mean hierarchy (sometimes they're just access-pattern grouping).

The pattern

import {Adapter} from 'dynamodb-toolkit';

const rentals = new Adapter({
  client: docClient,
  table: 'rentals',

  // The hierarchy: state > facility > vehicle.
  keyFields: ['state', 'facility', {name: 'vin', type: 'string'}],
  structuralKey: '_sk',                                        // composes 'TX|Dallas|1HGC…'
  technicalPrefix: '_',

  typeLabels: ['state', 'facility', 'vehicle'],
  typeField: 'kind',

  // Opt-in. Cascade primitives throw CascadeNotDeclared without this.
  relationships: {structural: true}
});

Invariants the declaration enforces:

  • keyFields.length > 1. Single-field adapters can't express hierarchy.
  • structuralKey is declared. The composition turns a subtree query into a cheap begins_with Query against the sort key.
  • Construction throws if either is missing.

Everything else is optional. typeLabels / typeField help identify what each row represents (needed when cascade results span multiple tiers); technicalPrefix lets the built-in prepare step manage the composed _sk attribute.

Delete a subtree

"Close the Dallas facility — remove the facility row and every vehicle under it":

const result = await rentals.deleteAllUnder({state: 'TX', facility: 'Dallas'});
// result: {processed, skipped, failed, conflicts, cursor?}

Order of operations:

  1. Descendants first (leaf-first). The toolkit runs deleteListByParams(buildKey(srcKey)) — Query finds every row whose _sk begins with TX|Dallas|, deletes them in BatchWriteItem-chunked batches.
  2. Self last. Only after descendants pagination completes (no cursor returned), the self row at {state: 'TX', facility: 'Dallas'} (depth 2) is deleted. Self-delete uses attribute_exists as a guard; if the self row was absent (e.g., the user did a two-step "delete children, delete parent"), the self phase buckets as skipped: 1.

Why leaf-first? If you crash mid-way, the self row is still there as an anchor for the next retry — deleteAllUnder(srcKey) is idempotent under resume. Root-first would orphan the leaves after a crash.

Subtree-only, preserve-the-parent: skip the self phase by calling deleteListByParams(buildKey(srcKey), options) directly — deleteAllUnder is the cascade umbrella, but the underlying primitive is accessible.

Clone a subtree

Two flavours.

Prefix-swap — cloneAllUnder(srcKey, dstKey)

"Duplicate Dallas as Houston, with every vehicle":

const result = await rentals.cloneAllUnder(
  {state: 'TX', facility: 'Dallas'},
  {state: 'TX', facility: 'Houston'}
);

Behind the scenes the toolkit builds a mapFn via adapter.swapPrefix(srcKey, dstKey) — a per-item transform that rewrites the facility field from 'Dallas' to 'Houston', leaves other keyFields and data attributes intact, and re-composes the structural key. Then cloneListByParams walks the subtree Query and writes each transformed item.

Order: root-first (self, then descendants). Source untouched.

The prefix must be contiguous-from-start — both srcKey and dstKey must supply the same keyFields, starting from the partition key. swapPrefix throws at construction if they don't; it throws at apply time if an item's keyField value doesn't match srcKey (sign of a mis-scoped upstream query).

Compose additional transforms — options.mapFn

Use the prefix-swap AND add your own per-item transform. The toolkit composes via mergeMapFn(swapPrefix(srcKey, dstKey), options.mapFn) — your mapFn sees the post-swap item and can override anything:

await rentals.cloneAllUnder(
  {state: 'TX', facility: 'Dallas'},
  {state: 'TX', facility: 'Houston'},
  {
    mapFn: item => ({
      ...item,
      // Reset every vehicle's availability to "available" at the new facility.
      available: true,
      // Strip any Dallas-specific metadata.
      locationNotes: undefined
    })
  }
);

Returning a falsy value from mapFn drops that item silently (bucketed as skipped, not failed).

MapFn-driven — cloneAllUnderBy(srcKey, mapFn)

When the destination isn't a uniform subtree — you want to fan-out across multiple subtrees based on a per-item property — use cloneAllUnderBy:

// Redistribute Austin's vehicles across Houston (SUVs) and Dallas (everything else).
await rentals.cloneAllUnderBy(
  {state: 'TX', facility: 'Austin'},
  vehicle => ({
    ...vehicle,
    facility: vehicle.type === 'SUV' ? 'Houston' : 'Dallas'
  })
);

No dstKey — your mapFn wholly decides where each item lands. The self row transforms the same way (facility on the parent row becomes 'Houston' or 'Dallas' per the mapFn's logic applied to the facility-level record).

Move a subtree

Move = clone + delete source. The toolkit composes this as a two-phase idempotent loop per item — constructive before destructive — so a crash mid-way is resumable without losing data.

Prefix-swap — moveAllUnder(srcKey, dstKey) 

// Relocate the Austin facility + all its vehicles to Dallas.
const result = await rentals.moveAllUnder(
  {state: 'TX', facility: 'Austin'},
  {state: 'TX', facility: 'Dallas'}
);

Per-item flow:

  1. Put destination with attribute_not_exists condition. If dst already exists (manual split-brain, or a resumed op re-trying an item we already migrated), skipped.
  2. Delete source. If delete fails after a successful put, the item is bucketed as failed — the caller sees the duplicate and can clean up.

Order across items: leaf-first. Descendants migrate before the self row. Shares the _subtreeRename machinery with the rename macro — same constructive-before-destructive guarantee.

MapFn-driven — moveAllUnderBy(srcKey, mapFn)

Same as cloneAllUnderBy but with the delete-source phase. Use when the destinations vary per item.

Resumability

Long-running cascades against large subtrees don't fit in a single Lambda invocation. The primitives honour options.maxItems + options.resumeToken on the descendants phase; the self phase runs once per full sequence (first call for root-first clones, last call for leaf-first deletes / moves).

// Scheduled Lambda, 15 min budget, 5000-item chunks:
const runCascade = async (cursor) => {
  return rentals.deleteAllUnder(
    {state: 'TX', facility: 'Dallas'},
    {maxItems: 5000, resumeToken: cursor}
  );
};

// EventBridge cron invokes this; each invocation either finishes the cascade
// or hands the cursor to the next run.
const result = await runCascade(previousCursor);
if (result.cursor) {
  // Save cursor for the next invocation.
} else {
  // Done. result.processed / skipped / failed report the full run.
}

Page-boundary semantics — the current page finishes, then the cap kicks in. Re-invoking with the same resumeToken is safe:

  • Delete: DynamoDB delete is idempotent; re-deleting a missing row succeeds, the re-run buckets it as processed: 1 (it didn't know it was already gone — DDB doesn't distinguish).
  • Clone: put-on-already-present-row is a collision; without ifNotExists, the later call clobbers the earlier one (usually fine — both calls write the same thing); with options.ifNotExists: true, collisions bucket as skipped.
  • Move: the constructive-before-destructive order makes this safe — a resumed item either sees dst missing (writes + deletes src) or dst present from the previous run (puts returns skipped, the delete still runs to clean up src).

The self phase sits at one end of the sequence specifically to avoid re-running: root-first clone stamps self on first call (no resumeToken); leaf-first delete / move runs self only when descendants pagination is complete. Intermediate cursor returns mean "more to do on descendants — self hasn't started / finished yet."

Partial failure

The cascade primitives are not transactional. DynamoDB TransactWriteItems caps at 100 actions; real subtrees blow past that. The toolkit emits per-item writes; a mid-cascade crash leaves partial state.

What the MassOpResult surfaces:

  • processed — items the cascade successfully wrote.
  • skipped — items intentionally or harmlessly bypassed: mapFn returned falsy; dst-already-exists on a conditional put; self row was absent at delete time.
  • failed[] — items that threw. Each entry carries {key, reason, details?, sdkError?} where reason is one of 'ConditionalCheckFailed', 'ValidationException', 'ProvisionedThroughputExceeded', 'Unknown'. The sdkError preserves the original AWS error for logging.
  • conflicts[] — separate bucket for VersionConflict failures (when versionField is declared). Lets callers distinguish "someone else's optimistic-concurrency retry" from other CCF.

Typical handling:

const result = await rentals.deleteAllUnder({state: 'TX', facility: 'Dallas'});
if (result.failed.length) {
  // Log for ops; decide whether to retry specific items or back off.
  for (const f of result.failed) console.error('delete failed', f.key, f.reason);
}

A deleteAllUnder that fails mid-run leaves the self row if descendants didn't finish (since self runs last). A resume call restarts from the cursor, re-Queries the subtree (now smaller), and continues; the self row gets deleted when the second run exhausts descendants.

Alternative: per-row orchestration

If the cascade operations you need are very irregular ("delete Dallas facility but keep the flagged vehicles; move vehicles older than 2020 to the archive facility") the declarative primitives stop fitting. Fall back to deleteListByParams / cloneListByParams / moveListByParams with the Query you built yourself:

// "Move every Dallas vehicle older than 2020 to the Archive facility"
const params = rentals.buildKey({state: 'TX', facility: 'Dallas'});
params.FilterExpression = 'yearFirstRegistered < :y';
params.ExpressionAttributeValues = {':y': 2020};

await rentals.moveListByParams(
  params,
  rentals.swapPrefix({facility: 'Dallas'}, {facility: 'Archive'}),
  {maxItems: 5000}
);

Cascade = convenience wrapper over this pattern for the common case. Drop down when the common case doesn't fit.

When this pattern fits

  • Your composite keyFields model a real parent-child hierarchy. The toolkit requires this upfront; if the keyFields stack for access-pattern reasons ("cluster these items together" with no parent/child meaning), don't declare relationships.
  • Subtrees are medium-sized (hundreds to millions of items). Big enough that you need pagination + resumability, small enough that per-item writes are economical.
  • Partial failure is recoverable through retry — the op is ultimately idempotent.
  • You run the operation on a schedule or as a user-triggered action, not in a tight request path. Cascade's latency is proportional to subtree size.

When it doesn't fit

  • Composite keyFields without hierarchy semantics. Stacking tenantId + groupId + itemId doesn't mean items are descendants of groups of tenants; leave relationships undeclared and use deleteListByParams with explicit buildKey shapes.
  • Atomic cascade required. Any intermediate crash leaves partial state. If you need "all or nothing" and the subtree is ≤100 items, use applyTransaction with hand-built descriptors instead. Above 100 items there is no atomic option in DynamoDB.
  • Very small subtrees (fewer than ~20 items). The pagination overhead isn't worth it; loop over adapter.delete / adapter.move yourself.
  • Cross-partition cascade without structural key (e.g., children in a different table). The structural-key assumption is hard-wired; cross-table cascades need orchestration at the application layer.

Related

🔍 Search the wiki

dynamodb-toolkit

Start here

Guides

Adapter

Expression builders

Batch / transactions / mass / paths

REST surface

Framework adapters

Recipes

History

Clone this wiki locally