Mass operation semantics

Mass operation semantics

The contract behind deleteListByParams, cloneListByParams, moveListByParams, editListByParams, and the cascade macros: what is atomic (almost nothing), what happens on partial failure (buckets, not exceptions), what a crash mid-operation leaves behind (a resumable, never-corrupt state), and how to run one as a queue worker. The orchestration cookbook is Resumable mass operations; this page is the semantics it relies on.

The model: a paged walk of individual writes

A mass operation is not a transaction. It is a Query/Scan walked page by page, each page's items written through chunked BatchWriteItem (25 per call) or per-item Commands (strategy: 'sequential', or automatically when per-item conditions are in play). The SQL intuition to unlearn: DELETE FROM t WHERE … is atomic in SQL; here it is thousands of individually-succeeding deletes. There is no rollback — there is accounting.

Consequences, stated plainly:

• Atomicity boundary = one item. Even within a 25-item batch, items succeed and fail independently.
• Isolation = none. Concurrent readers see the operation in progress; concurrent writers interleave with it. Scope-freeze (asOf) bounds which items the walk touches (only those created at or before the timestamp — requires createdAtField), not what readers observe.
• Durability = per item. Once an item's write is accepted, it stays regardless of what happens to the rest of the operation.

When you actually need multi-item atomicity, that's applyTransaction — all-or-nothing, capped at 100 actions, no chunking (chunking a transaction would forfeit exactly the atomicity you asked for). The two compose: mass ops for bulk shape, transactions for the small invariant-critical clusters.

The result envelope — accounting, not exceptions

Every list-op variant returns a MassOpResult:

interface MassOpResult {
  processed: number;   // writes DynamoDB accepted
  skipped: number;     // intentionally not written (see below)
  failed: MassOpFailure[];     // {key, reason, details?, sdkError?}
  conflicts: MassOpConflict[]; // {key, reason: 'VersionConflict', sdkError?}
  cursor?: string;     // present iff the op stopped early (maxItems)
}

What lands where — the bucket is the semantic:

Bucket	Meaning	Typical causes
processed	The write happened.	—
skipped	The operation chose not to write — expected outcome, not an error.	ifNotExists hit an existing item (including re-encounters on resume); edit produced no change; a mapFn returned falsy (per-item veto); rename put-collision at the destination.
failed	The write was attempted and rejected.	Validation errors, item-collection limits, unclassified SDK errors. reason names the class; sdkError carries the original.
conflicts	Optimistic-concurrency loss (versionField declared): someone else wrote the item between read and write.	Version mismatch on editListByParams and friends. Retryable by re-running — the next pass reads the fresh version.

Per-item problems never throw — throwing would abort a half-done walk and lose the accounting. Only operation-level problems throw: a broken query, retry exhaustion on persistent throttling (see RetryOptions — fast-fail for request-path calls, patient for offline jobs), user callbacks throwing (the toolkit does not wrap your mapFn — a throw there is your signal, and it aborts the walk).

Handling pattern — treat buckets as work queues, not logs:

const r = await adapter.editListByParams(params, migrateItem, {maxItems: 1000});
if (r.conflicts.length) await retryLater(r.conflicts.map(c => c.key)); // OC losers: re-run
if (r.failed.length) await deadLetter(r.failed);                       // real failures: investigate

Idempotent phases — why resuming is safe

maxItems is a soft, page-boundary cap: the current page always finishes, then the result carries a cursor. Resuming with resumeToken restarts at the last page boundary — which means items after the boundary-but-before-the-stop may be re-processed. The design makes re-processing a no-op instead of trying to prevent it:

• Delete is idempotent. Re-deleting an absent item succeeds silently.
• Put is idempotent (same input → same item). With ifNotExists, a resume re-encounter lands in skipped — intent preserved, no clobber.
• Edit re-reads. The diff is computed against the current item; an already-migrated item diffs to nothing → skipped.
• Two-phase macros order for crash-safety. rename/moveAllUnder are constructive-before-destructive: put everything at the destination, then delete the source. A crash between phases leaves both copies — recoverable — never neither. cloneWithOverwrite is the explicit opposite (destructive-first, destination declared disposable); deleteAllUnder is leaf-first so a partial delete never orphans children below a removed parent.

The operational rule this buys: any mass op can be killed and re-run with the same params + last cursor, without a repair step. Cursors are opaque, versioned (v: 1), and page-aligned; they are resumption bookmarks, not read positions — don't confuse them with pagination cursors, which page reads.

The queue-worker shape

maxItems + resumeToken turn an unbounded sweep into uniform, budgetable work units — the shape every queue runtime wants (Lambda time budgets, Step Functions iterations, cron ticks, SQS redrives):

// One worker invocation ≈ one bounded unit of work:
export const handler = async event => {
  const r = await adapter.deleteListByParams(params, {
    maxItems: 2000,
    resumeToken: event.cursor,          // undefined on the first run
    retry: {maxAttempts: 4}             // fail the invocation before the runtime kills it
  });
  if (r.failed.length) await pushDeadLetter(r.failed);
  return r.cursor ? {done: false, cursor: r.cursor} : {done: true};
};

Feed cursor back through your loop mechanism of choice — Step Functions Choice state, SQS self-enqueue, EventBridge retry — until it comes back absent. Progress metrics fall out of the envelope (processed per tick); the buckets are your alerting surface.

The same loop works over HTTP. The mass routes accept ?max-items= and ?resume=, and return the full envelope (optional buckets appear only when non-empty):

DELETE /vehicles/?eq-status=retired&max-items=2000
→ 200 {"processed": 2000, "cursor": "eyJ2IjoxLCJMYXN0…"}

DELETE /vehicles/?eq-status=retired&max-items=2000&resume=eyJ2IjoxLCJMYXN0…
→ 200 {"processed": 743}

A malformed resume token is rejected at the boundary (400 BadCursor); an unscoped DELETE / (no filter, no search, no tenant scope) is 400 UnscopedMassDelete unless ?confirm=true — the delete-all footgun has a safety on it (HTTP handler).

Concurrency during a sweep

Writers keep writing while your sweep runs. The tools, in escalation order:

1. asOf scope-freeze — the sweep only touches items that existed when it started (createdAt <= asOf); new writes are structurally outside the sweep. Requires createdAtField.
2. versionField — items modified mid-sweep lose the OC check and land in conflicts instead of being clobbered; re-run the conflict keys.
3. ifExists / ifNotExists — per-item existence conditions when the sweep's assumption is presence/absence rather than version.

What you cannot get: a consistent point-in-time view of the whole table during the walk. If a sweep's correctness depends on that, it isn't a mass op — it's an export (backup / ExportTableToPointInTime) plus offline processing.

Related

• Resumable mass operations — orchestration recipes (Lambda, Step Functions, driver scripts).
• Cascade subtree operations — the hierarchy macros built on these semantics.
• Batch and transactions — the atomic alternative and RetryOptions.
• Adapter: Mass methods — per-method reference.
• Pagination — read-side cursors, and why they're a different animal.