feat(wallet-toolbox): plan noSend batches locally and commit atomically

[ty-everett]

Summary

Add an automatically negotiated Wallet Toolbox fast path for high-volume noSend transaction sequences:

1. reserve a bounded funding pool on the first action;
2. plan, sign, validate, and track dependent actions in memory;
3. atomically persist the complete workspace on the final sendWith;
4. broadcast only after the storage transaction commits.

The BRC-100 wallet interface, CreateActionArgs, SignActionArgs, noSend, noSendChange, sendWith, and result shapes are unchanged. Providers that do not advertise actionBatch: 1 retain the legacy path before a workspace begins.

Why

The existing behavior is correct but latency-bound for long transaction sequences. Every signed noSend action performs a storage createAction call followed by processAction; the final sendWith adds another call. A 250-action sequence therefore serializes 501 storage control calls.

That cost is modest against a local database but dominates against remote storage. At 100 ms RTT, the storage-control component alone is approximately 50.1 seconds before transaction planning, signing, validation, payload transfer, persistence, or broadcast.

This change moves the middle of the workflow to the wallet process, where the keys and staged transactions already live, while preserving storage authority and an atomic remote durability boundary.

Stack status

This PR now targets current main directly. Commit 021951635 merges main at f63482576 (including #285) and resolves the integration conflicts while preserving the independent shared-session/auth migrations from #283.

The post-merge review also hardens workspace concurrency, provider-scoped capability negotiation, expired-reservation reacquisition, failed-extension cleanup, staged noSend listing behavior, and logical blob assembly limits.

Design

Capability negotiation and compatibility

• Add optional storage capability actionBatch: 1.
• Add WalletArgs.actionBatchMode?: 'auto' | 'legacy', defaulting to auto.
• Negotiate once per active provider.
• Fall back to the existing flow before creating any workspace when the provider lacks support.
• Never mix legacy and batch persistence within one workspace.

In-memory planning

• Extract shared action-planning behavior so legacy and batch paths use the same fee, output, change, randomization, and validation rules.
• Maintain one wallet-local workspace containing funding reservations, staged transactions, staged outputs, signing references, consumed inputs, normalized metadata, and a shared BEEF dependency graph.
• Overlay staged state onto relevant list, balance, and abort operations.
• Preserve returnTXIDOnly, knownTxids, commissions, custom inputs, two-step signing, deterministic output order, and noSendChange.
• Compact completed actions into structural records and content-addressed scripts rather than retaining repeated transactions, source transactions, locking scripts, or ancestry.

Reservations and lifecycle

• Add durable action-batch, per-output reservation, and staged-blob records for Knex and IndexedDB providers.
• Reserve only the first action's canonical inputs plus limited headroom.
• Extend adaptively from an EWMA consumption estimate with bounded geometric runway.
• Use 15-minute renewable leases and a 60-minute hard lifetime.
• Recheck reservation ownership and output spendability under row locks.
• Release unused state on commit, abort, cleanup, wallet destruction, or expiry.
• Keep unreserved outputs available to unrelated wallet activity.

Atomic commit and validation

Before persistence, validate:

• manifest and blob digests;
• dependency order and BEEF ancestry;
• transaction structure, TXIDs, input/output mappings, and duplicate spends;
• source amounts and scripts from proven transactions;
• signatures, fees, commissions, and requested output metadata;
• reservations, leases, noSendChange, and storage-controlled outputs.

Persist transactions, outputs, labels, tags, mappings, and proof requests in one storage transaction. Consume reserved inputs and release unused reservations in that same transaction.

Broadcast remains outside the database transaction. Persistence is idempotent by user, batch ID, and semantic manifest digest; retries reconcile the committed result without duplicating wallet state.

Large payload transport

• Inline content-addressed blobs up to 4 MiB.
• For larger manifests, prepare the commit, obtain missing digests, upload authenticated binary blobs, and commit by manifest digest.
• Enforce an 8 MiB per-blob limit and four concurrent uploads by default.
• Accept only digests authorized by the prepared manifest.
• Validate size before hashing and reject digest mismatches, duplicate conflicting uploads, and unprepared blobs.
• Clean incomplete staged blobs with batch expiry.

Main changes

• New wallet-local planner and workspace orchestration.
• New optional storage interfaces for begin, extend, renew, prepare, upload, commit, and abort.
• Knex and IndexedDB schema/migration support.
• Row-locking and reservation-aware allocation for SQL providers.
• User-scoped IndexedDB batch uniqueness.
• Authenticated StorageClient/StorageServer binary upload routes.
• Atomic manifest validation and persistence modules.
• One-minute opportunistic cleanup monitor.
• Staged list/balance/abort coherence.
• Deterministic legacy/batch parity tests.
• Retained benchmark and rollout documentation.
• Benchmark sources are now included in the package lint gate.

Benchmarks

Command:

pnpm --filter @bsv/wallet-toolbox bench:action-batch

Environment: local SQLite provider, Node v25.9.0, pnpm 10.33.2. The measured representative workload is a 250-action dependent chain with 1 KiB scripts and deterministic randomness.

Metric	Legacy	Batch	Change
Storage RPCs	501	2	-99.6%
Database transactions	252	3	-98.8%
Action staging	10,305.7 ms	3,741.2 ms	-63.7%
Planning	6,159.4 ms	118.1 ms	-98.1%
Signing and validation	4,146.2 ms	3,623.0 ms	-12.6%
Final commit	24.7 ms	3,540.0 ms	one atomic batch commit
Stage plus commit	10,330.4 ms	7,281.2 ms	-29.5%
Uploaded/request bytes	1,200,559	791,249	-34.1%
CPU user time	9,665.9 ms	7,011.8 ms	-27.5%
CPU system time	936.9 ms	467.2 ms	-50.1%
Incremental peak heap	372,938,672 B	336,458,528 B	-9.8%

Peak heap is GC-sensitive; the structural assertions are the stronger memory regression gate. They verify that staged records do not retain duplicate source transactions or locking scripts.

The modeled storage-control component for the same 250-action sequence is:

Simulated RTT	Legacy	Batch
25 ms	12,525 ms	50 ms
100 ms	50,100 ms	200 ms
250 ms	125,250 ms	500 ms

The benchmark retains 192 modeled cases across dependent, independent, mixed explicit-input, and two-step workloads; 1, 10, 50, and 250 actions; 1 KiB, 64 KiB, 1 MiB, and 4 MiB scripts; and 25/100/250 ms RTT.

It also physically executes 64 KiB, 1 MiB, and 4 MiB scripts. The 4 MiB one-action batch exercises the prepare/upload/commit path successfully. It is intentionally not presented as a local one-action speedup: on the measured machine, batch stage plus commit was about 2.53 seconds versus 1.47 seconds for legacy because validation and blob staging outweigh avoided local latency. The target benefit is long and/or remote sequences.

Validation

Completed locally after merging current main:

• pnpm install --frozen-lockfile
• Full workspace build across 35 packages
• Wallet Toolbox build and lint
• Wallet Toolbox full suite: 131 suites passed; 1,207 tests passed; 27 skipped
• Focused action-batch regression suites after the review fixes: 44 passed
• Hosted-failure reproducer (batch + StorageClient suites) with hermetic broadcasters: 34 passed in 4.3 seconds
• SDK build, lint, UMD build, and full suite: 138 suites passed; 5,657 tests passed; 3 skipped
• Retained action-batch benchmark, including the physical 4 MiB upload path
• Documentation validation and build: 89 source files / 93 generated HTML files
• Script conformance validation: 6,650 vectors
• Cross-package version and SDK peer-dependency checks
• git diff --check

The prior hosted build/test failure was traced to two new suites reaching live chain-root validation and broadcast services. Their providers now use pinned hermetic service fixtures for both boundaries. The final hosted build/test job passed in 10m51s; CodeQL, Socket, SonarCloud, Codecov, docs, and both conformance checks also passed.

Not yet completed:

• Live MySQL provider E2E. Knex/MySQL code compiles; shared storage behavior is exercised through SQLite and IndexedDB.

Reviewer notes

Suggested review order:

1. Compatibility boundary — confirm capability negotiation and legacy fallback in Wallet.ts, WalletStorageManager.ts, and the storage interfaces leave public BRC-100 behavior unchanged.
2. Planner parity — compare shared planning, deterministic transaction parity, returnTXIDOnly, knownTxids, two-step signing, and noSendChange behavior.
3. Reservation concurrency — inspect SQL row locking, per-user uniqueness, lease expiry/reacquisition, cleanup rechecks, and exclusion from legacy allocation.
4. Commit validation — focus on proven source values, signature verification, fees/commissions, requested output metadata, graph ordering, and duplicate-spend rejection.
5. Atomicity and idempotency — verify transaction/output/metadata/proof-request persistence, reservation release, response-loss retries, and broadcaster-failure recovery.
6. Blob authorization — inspect prepared-digest allowlisting, payload caps, hashing order, content addressing, duplicate chunks, and expiry cleanup.
7. Retained memory — verify compact plans and staged outputs hydrate from the shared BEEF/script store instead of retaining full duplicate objects.
8. Provider parity — pay particular attention to MySQL locking semantics and IndexedDB's composite [userId, batchId] key.
9. Benchmark interpretation — distinguish measured local results from modeled network RTT and avoid treating the single-action large-payload path as a throughput win.

Rollout considerations

A server-first rollout is safe:

1. deploy schema and provider capability support;
2. deploy clients with the default auto mode;
3. monitor reservation conflicts, extensions, expiries, commit retries, blob deduplication, commit time, and broadcaster outcomes;
4. retain actionBatchMode: 'legacy' for controlled comparison or rollback.

Intermediate workspaces are intentionally session-scoped. The final commit is the remote durability boundary.

[tonesnotes]

So much good stuff here :-)
This greatly enhances the value of remote storage (which will always have resilience, survivability, and security advantages over local).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[BraydenLangley]

Live E2E + unit test results: perf win confirmed, but batch funding fails deterministically on fragmented wallets at realistic fee rates

Tested PR head d75dbb4 deployed live: dedicated storage server running this branch's toolbox + SDK (single-instance verified), real HTTPS StorageClient boundary, mainnet.

Perf — real

A/B on the same wallet/server, un-chained noSend sequences committed via sendWith:

N	batch	legacy	round-trips
8	3,495 ms	8,289 ms	4 vs 17

2.4× overall; staging alone is 942 ms vs 6,454 ms. Round-trips go 2N+1 → constant, so the advantage widens with N. 14 txids confirmed on mainnet across N=1/5/8 runs.

Bug — batch staging dies with bogus WERR_INSUFFICIENT_FUNDS

Wallet: 63k sats, 188 outputs (150 × ~40-sat change dust + healthy 23,800/15,290/7,474 outputs).

• Batch N=16: fails 4/4, deterministic. Extend trail shows targetSatoshis pinned at 8 while requestedOutputs doubles 8→16→32→64; only dust ever reserved; the healthy outputs are never selected. Aborts cleanly, 0 sats lost.
• Legacy N=16: succeeds — same wallet, same server, same process. Only actionBatchMode differs.

Root cause

1. selectCanonicalChange (actionPlanning.ts) picks the smallest output ≥ target. The retry target is changeFirstSatoshis = minimumDesiredUTXOValue / 4 (8 sats here) — so it always selects another dust output, never a healthy one.
2. At real fee rates dust can't also fund a change output, so generateChangeSdk hits changeOutputs.length === 0 && feeExcessNow > 0 (generateChange.ts:390–392) and throws asking for changeFirstSatoshis more.
3. ActionBatchPlanner.planFunding retries with doubled requestedOutputs but an unchanged target → identical deterministic selection → identical throw. Non-convergent by construction; fails outright once the dust pool is exhausted.

Why the suite is green: test fixtures run feeModel { sat/kb: 1 }, where dust inputs are never fee-problematic. At the StorageProvider default (100) it reproduces immediately.

Unit repro (in-process, deterministic, no network)

Pushed to this branch as 8a94cee. test/Wallet/action/actionBatchFragmentedWallet.test.ts fragments the standard fixture wallet organically (basket set to 144 desired × 40 sat minimum, then 20 ordinary committed actions let generateChange mint the dust itself), sets feeModel to 100 sat/kb, then stages 16 un-chained noSend actions:

• batch ('auto'): fails on the first action — WERR_INSUFFICIENT_FUNDS: 10 more satoshis needed, for a total of 4 on a wallet holding 5.73M sats including a 2M-sat output. Extends: target 10 pinned, requested 8→64.
• legacy control: passes — 16 staged + committed on the identical wallet shape.

Side note that reconciles earlier testing: clean few-output wallets at small N stay within a single extend and work fine (and get the full speedup) — the loop only breaks when funding needs multiple extends against a dusty pool at real fees.

test/Wallet/action/actionBatchFragmentedWallet.test.ts — committed on this branch as 8a94cee; run with npx jest --runInBand --runTestsByPath test/Wallet/action/actionBatchFragmentedWallet.test.ts

import { _tu, TestWalletNoSetup } from '../../utils/TestUtilsWalletStorage'
import { WERR_INSUFFICIENT_FUNDS } from '../../../src/sdk/WERR_errors'
import { verifyOne } from '../../../src/utility/utilityHelpers'

/**
 * Reproduces the PR#289 batch-funding failure observed end-to-end against a
 * live storage server (see manifests: targetSatoshis pinned while
 * requestedOutputs doubles 8 -> 64, dust reserved every round, large outputs
 * never selected, WERR_INSUFFICIENT_FUNDS despite ample spendable funds).
 *
 * The wallet is fragmented ORGANICALLY: the default basket is configured to
 * desire many small change outputs (144 x 40 sats, mirroring the production
 * wallet that failed) and the wallet's own generateChange then mints the dust
 * through ordinary committed actions. No rows are hand-crafted.
 *
 * Expected (correct) behavior: an un-chained noSend sequence of 16 actions
 * stages successfully in batch mode, exactly as it does in legacy mode on the
 * identical wallet shape (the control test below passes).
 */

const randomVals = [0.1, 0.2, 0.3, 0.7, 0.8, 0.9]

function noSendArgs (i: number) {
  return {
    description: `fragmented batch action ${i}`,
    outputs: [{
      satoshis: 1,
      lockingScript: '7551',
      outputDescription: 'workload output'
    }],
    options: { noSend: true, randomizeOutputs: false }
  }
}

async function mockChain (ctx: TestWalletNoSetup): Promise<void> {
  _tu.mockPostServicesAsSuccess([ctx])
  jest.spyOn(ctx.services, 'getChainTracker').mockResolvedValue({ isValidRootForHeight: async () => true } as any)
  jest.spyOn(ctx.activeStorage, 'getServices').mockReturnValue(ctx.services)
  // Production fee regime (the test fixture defaults to sat/kb 1, under which
  // dust inputs are never fee-negative and the funding loop is never stressed).
  ctx.activeStorage.feeModel = { model: 'sat/kb', value: 100 }
}

/**
 * Reshape the default basket to desire a large pool of small change outputs,
 * then let the wallet's own change generator fragment the pool via ordinary,
 * immediately-processed actions. Returns the resulting spendable change set.
 */
async function fragmentWallet (ctx: TestWalletNoSetup): Promise<number[]> {
  const basket = verifyOne(await ctx.activeStorage.findOutputBaskets({
    partial: { userId: 1, name: 'default' }
  }))
  await ctx.activeStorage.updateOutputBasket(basket.basketId, {
    numberOfDesiredUTXOs: 144,
    minimumDesiredUTXOValue: 40
  })

  for (let i = 0; i < 20; i++) {
    await ctx.wallet.createAction({
      description: `fragmentation churn ${i}`,
      outputs: [{ satoshis: 1, lockingScript: '7551', outputDescription: 'churn output' }],
      options: { randomizeOutputs: false, acceptDelayedBroadcast: false }
    })
  }

  const outputs = await ctx.activeStorage.findOutputs({
    partial: { userId: 1, basketId: basket.basketId, change: true, spendable: true }
  })
  return outputs.map(o => o.satoshis).sort((a, b) => b - a)
}

describe('action batch funding on a fragmented wallet', () => {
  jest.setTimeout(300000)

  test('batch mode: un-chained noSend sequence of 16 must not exhaust funding (currently fails)', async () => {
    const ctx = await _tu.createLegacyWalletSQLiteCopy('fragmentedBatchFunding', 'auto')
    try {
      await mockChain(ctx)
      ctx.wallet.randomVals = [...randomVals]

      const sats = await fragmentWallet(ctx)
      const dust = sats.filter(s => s < 100)
      const healthy = sats.filter(s => s >= 1000)
      const total = sats.reduce((s, v) => s + v, 0)
      // Sanity: the wallet is fragmented but amply funded, with healthy
      // outputs available - the same shape as the production wallet that
      // failed (63k sats, 150 dust outputs, 5 healthy outputs).
      expect(dust.length).toBeGreaterThanOrEqual(50)
      expect(healthy.length).toBeGreaterThanOrEqual(1)
      expect(total).toBeGreaterThan(100000)

      // Diagnostic capture of the extend loop (targetSatoshis/requestedOutputs)
      const extendCalls: Array<{ targetSatoshis: number, requestedOutputs: number }> = []
      const origExtend = ctx.storage.extendActionBatch.bind(ctx.storage)
      jest.spyOn(ctx.storage, 'extendActionBatch').mockImplementation(async (args: any) => {
        extendCalls.push({ targetSatoshis: args.targetSatoshis, requestedOutputs: args.requestedOutputs })
        return await origExtend(args)
      })

      const txids: string[] = []
      try {
        for (let i = 0; i < 16; i++) {
          const result = await ctx.wallet.createAction(noSendArgs(i))
          txids.push(result.txid!)
        }
      } finally {
        // Surface the diagnostic trail whether or not the run failed.
        // eslint-disable-next-line no-console
        console.log(`staged ${txids.length}/16 on wallet holding ${total} sats (${dust.length} dust, ${healthy.length} healthy outputs); extend sequence: ${JSON.stringify(extendCalls)}`)
      }

      expect(txids).toHaveLength(16)
    } finally {
      await ctx.wallet.destroy()
    }
  })

  test('legacy mode control: identical fragmented wallet, identical un-chained sequence, succeeds', async () => {
    const ctx = await _tu.createLegacyWalletSQLiteCopy('fragmentedLegacyControl', 'legacy')
    try {
      await mockChain(ctx)
      ctx.wallet.randomVals = [...randomVals]

      const sats = await fragmentWallet(ctx)
      expect(sats.filter(s => s < 100).length).toBeGreaterThanOrEqual(50)

      const txids: string[] = []
      for (let i = 0; i < 16; i++) {
        const result = await ctx.wallet.createAction(noSendArgs(i))
        txids.push(result.txid!)
      }
      expect(txids).toHaveLength(16)

      const commit = await ctx.wallet.createAction({
        description: 'commit legacy control sequence',
        options: { sendWith: txids, acceptDelayedBroadcast: false }
      })
      expect(commit.sendWithResults).toHaveLength(16)
    } finally {
      await ctx.wallet.destroy()
    }
  })
})

[ty-everett]

Addressed these review items in the latest commit, it should be good for another round of review now!

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​express-rate-limit@​8.6.0

View full report

[sonarqubecloud]

Quality Gate passed

Issues
19 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ty-everett]

All of the issues raised by @BraydenLangley have been addressed, so I am going to merge this and then focus on dependency management, versioning, security issues, and new releases stack-wide.