Skip to content

Optimize large action-batch transport and Wallet Wire handoff - #360

Merged
ty-everett merged 1 commit into
bsv-blockchain:mainfrom
ty-everett:codex/action-batch-large-data-streaming
Jul 26, 2026
Merged

Optimize large action-batch transport and Wallet Wire handoff#360
ty-everett merged 1 commit into
bsv-blockchain:mainfrom
ty-everett:codex/action-batch-large-data-streaming

Conversation

@ty-everett

@ty-everett ty-everett commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a capability-negotiated format-2 action-batch manifest that derives repeated source/output scripts from signed transactions, retains only the external proof frontier, and commits prepared manifests by digest
  • add authenticated ABP1 multi-blob uploads with bounded gzip/Brotli/identity negotiation, bulk storage operations, decompression limits, and per-item content verification
  • authorize packed uploads through a prepared manifest: the standard provider rejects packs sent before preparation and rejects every digest not requested by that manifest, preventing unbounded eager upload/storage amplification
  • retain typed byte views and pre-sized frames through large Wallet Wire create/sign/internalize/list paths
  • preserve manifest format 1, one-blob uploads, legacy Wallet Wire number arrays, third-party storage implementations, and every BRC-100 argument/result shape

Physical pack sizes are transport boundaries only. Logical transactions and scripts are chunked across packs without introducing a consensus or script-size limit.

This head is rebased onto main at 78563db73edf078a7fa2f3ddd8f6b1e0a489d3e7.

Representative measurements

Latest retained benchmark on an Apple Silicon host with Node.js v24.18.0:

  • 250 dependent actions with 1 KiB generic scripts: 30,383 ms legacy versus 11,294 ms batch planning/signing (62.8% lower); storage calls 501 to 2 and database transactions 252 to 3
  • one generic 4 MiB zero-filled script: 1,085 ms legacy versus 150 ms batch planning/signing (86.1% lower); instrumented request bytes 13,982,465 to 8,067; one 4,195,234-byte logical pack to 5,236 compressed bytes
  • the modeled 250-action/4 MiB case preserves two persistence workflows while chunking physical pack transport

Compression falls back to identity whenever it would expand a pack.

Security properties

  • authenticated action-batch lifetime and per-pack item/byte/decompression bounds remain enforced
  • standard storage advertises eager: false; uploads require a prepared manifest digest
  • every packed blob digest must be present in that manifest's uploadDigests
  • content digests are verified after bounded decompression before persistence
  • duplicate authorized uploads are idempotent and large writes use bounded SQL chunks
  • request bodies are runtime-narrowed and normalized to a zero-copy byte view before binary decoding; authenticated JSON/object bodies are rejected
  • public-service CORS defaults remain unchanged and configurable

Validation

  • SDK: build/typecheck/lint, 145 suites / 5,739 passing tests; 92.79% line coverage
  • Wallet Toolbox: build/typecheck/lint, 144 suites / 1,301 passing tests; 71.33% line and 59.44% branch coverage
  • affected action-batch and transport suites: 80 passing tests
  • browser client: typecheck, build, lint/format, 17 tests, browser/Vite/esbuild contract, package artifact checks
  • mobile client: typecheck, build, lint/format, 18 tests, Metro/Hermes contract, package artifact checks
  • standalone WAB under Node 24: build, deny-warning lint, 10 suites / 86 tests
  • Overlay Express: build/lint, 12 suites / 276 passing tests (3 skipped)
  • synchronized edge-policy generation across seven contexts and rate-limit generation across four contexts
  • retained action-batch benchmark, including packed 4 MiB transport
  • SDK exact-tarball browser and package artifact checks; Wallet Toolbox package artifact checks
  • repository health checks, Open BSV License v6 manifest check, version-reference check, and security audit with no known vulnerabilities
  • PR CI run 30224170639: every build, test, platform, coverage, infra, policy, and merge-gate job passed
  • SonarCloud: quality gate OK, zero open/confirmed issues, zero hotspots
  • exact-head security-extended CodeQL run 30224176517: Actions and JavaScript/TypeScript passed; critical alerts fix(infra): chaintracks param typing + message-box-server lockfile resync #213chore: sync published workspace versions #218 were fixed by the request-body boundary remediation with no dismissal or suppression
  • clean diff checks and zero unresolved review threads

The tests and measurements use generic scripts and byte patterns only. This PR does not change package versions or publish packages.

@ty-everett
ty-everett force-pushed the codex/action-batch-large-data-streaming branch from ab0677b to 98a41f2 Compare July 26, 2026 22:42
@ty-everett
ty-everett force-pushed the codex/action-batch-large-data-streaming branch from 98a41f2 to 48f4e6c Compare July 26, 2026 22:56
@sonarqubecloud

Copy link
Copy Markdown

@ty-everett ty-everett left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer exact-head approval in substance (ty-everett): I reviewed commit 48f4e6cec22fd63b2141acdcc489a6d4de671a7a, rebased directly on main@78563db73edf078a7fa2f3ddd8f6b1e0a489d3e7.

Review findings were remediated before merge:

  • standard storage now rejects unprepared and unrequested packed digests, preventing eager upload/storage amplification;
  • authenticated HTTP pack bodies are explicitly runtime-narrowed and zero-copy normalized before binary decoding;
  • exact-head CodeQL run 30224176517 passed Actions and JavaScript/TypeScript with security-extended queries, and critical alerts #213#218 are fixed with no dismissal or suppression;
  • Sonar reports an OK quality gate, zero open/confirmed issues, and zero hotspots;
  • PR CI run 30224170639 passed every build, test, coverage, platform, infrastructure, policy, and final merge-gate job;
  • local full suites passed: SDK 5,739 tests at 92.79% line coverage and Wallet Toolbox 1,301 tests at 71.33% line / 59.44% branch coverage;
  • dependency review, Socket, audit, repository health, artifact/browser/mobile contracts, WAB, Overlay Express, synchronized edge/rate policies, and retained performance benchmarks passed;
  • there are no review threads or change requests, and the working diff is one clean commit over unchanged main.

GitHub does not permit an author to submit an APPROVED review on their own pull request, so this COMMENT review records the same exact-head maintainer decision and evidence. I approve this SHA for the authorized admin squash merge; admin authority is used only to satisfy that impossible self-approval requirement.

@ty-everett
ty-everett merged commit 96ad2b2 into bsv-blockchain:main Jul 26, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant