Skip to content

Govern stack-wide property fuzzing and test quality - #365

Merged
ty-everett merged 9 commits into
mainfrom
codex/test-governance-wave
Jul 27, 2026
Merged

Govern stack-wide property fuzzing and test quality#365
ty-everett merged 9 commits into
mainfrom
codex/test-governance-wave

Conversation

@ty-everett

@ty-everett ty-everett commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • establishes a machine-readable test-quality contract for required, manual/operator, scheduled-live, resource-intensive, and conformance-gap suites
  • reduces anonymous required-suite skips from 60 to 6, classifies 40 non-PR suites, removes empty/obsolete/stale manual cases, and prevents assertion-free or unregistered skips from entering CI
  • prevents TypeScript conformance dispatchers from passing vacuously after reaching not implemented; 211 intended gaps across 19 vector files now have governed ownership, reasons, exact counts, review dates, and removal criteria
  • adds fast-check property fuzzing across 25 packages and 88 independent properties covering codecs, binary and script framing, signed-body canonicalization, public protocol parsers, network destinations, GASP and reorg events, integrity hashes, native batch memory, wallet pairing, issuer allowlists, operator/scaffolder inputs, currency math, HTTP payment challenges, authenticated Express framing, replay admission, BTMS authorization, and untrusted metadata
  • inventories all 33 package manifests: 25 have meaningful registered property suites and 8 adapter/composition/example/platform surfaces have dated, owned exclusions, preventing both silent gaps and cosmetic tests
  • runs at least 300 generated cases per property in ordinary package tests and adds a weekly/manual 5,000-case campaign with explicit seed/path replay and failure-log retention
  • hardens boundaries exposed by the properties: canonical base64/base64url, strict Teranode envelopes and UTF-8, safe uint32 asset IDs and Bitcoin varints, cryptographically valid relay keys, canonical pairing origins/expiry, reserved Message Box destinations, safe GASP timestamps/limits and response shapes, reorg heights/hashes and arbitrary structured JSON, complete JS8 measurements, canonical safe BTMS amounts, isolated BTMS authorization state, and typed derivation instructions
  • documents the commands, selection rules, invariants, replay procedure, exclusion criteria, and responsibility for extending the registry when new trust boundaries are added

Why

The stack previously mixed deterministic tests with live, credentialed, resource-heavy, empty, and intentionally unsupported cases. That made coverage claims ambiguous and allowed non-required conformance dispatch paths to pass without assertions. High-risk parsers and serializers also relied mostly on hand-picked examples. This wave makes what runs auditable and adds broad generative checks that have already found concrete malformed-input, overflow, canonicalization, and authorization-boundary defects.

Validation

  • FAST_CHECK_NUM_RUNS=1000 pnpm test:property (full workspace build plus at least 88,000 generated cases from 88 independent properties across 25 packages)
  • failing Message Box seed reproduced and then passed for 5,000 runs after constraining generated public hosts to the runtime URL grammar
  • pnpm test:governance; governance unit tests: 6/6; all 33 package manifests classified
  • affected ordinary suites: 592 tests green across the five newly covered packages
  • affected coverage suites green; wallet script encoder 100% lines/branches, GASP 95.95% lines, BTMS topic manager 92.98% lines, BTMS permission module 89.76% lines, and create-bsv-app 97.65% lines
  • TS conformance: 6,439 passed / 211 governed skips / 6,650 total; structural runner: 6,650 vectors clean
  • affected typechecks green; the previously identified Fund Wallet dual-SDK nominal-type conflict remains isolated for the next corrective wave
  • full workspace build, lint, and format checks green
  • all package pack:check exact-tarball consumers green
  • SDK, Paymail, Message Box, Wallet Relay, VeriFast, and wallet browser/mobile consumer gates green
  • docs frontmatter, links, tests, typecheck, and production build green
  • 30 public package license tarballs green
  • pnpm audit:security: no known vulnerabilities
  • reproducible OpenAPI code generation: 9 generated files current
  • SonarCloud new-code quality gate with 0 issues and 0 hotspots, CodeQL, Dependency Review, Socket, and Codecov patch checks green on exact head a59cdc604b87a367ef170c5c6c4793ffa2f49e30

No package versions are changed and nothing is published by this PR.

Program tracker: #324

@socket-security

socket-security Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​fast-check@​4.9.010010010089100

View full report

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.38554% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ckages/overlays/overlay-express/src/ReorgStream.ts 83.33% 0 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@sonarqubecloud

Copy link
Copy Markdown

@ty-everett ty-everett left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer/admin exact-head attestation for a59cdc6: reviewed scope and evidence; the branch includes current main, all exact-head required checks are green, Sonar reports 0 new issues and 0 security hotspots, CodeQL/Socket/Dependency Review/Codecov gates pass, and the final local campaign exercised 88 independent properties across 25 packages with at least 88,000 generated cases while machine-classifying all 33 package manifests. There are no unresolved review threads. Admin merge is authorized for this maintenance wave.

@ty-everett
ty-everett merged commit 27986ed into main Jul 27, 2026
39 checks passed
@ty-everett
ty-everett deleted the codex/test-governance-wave branch July 27, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant