Skip to content

Validate stack fuzzing with mutation testing - #369

Merged
ty-everett merged 2 commits into
mainfrom
codex/mutation-coverage-wave
Jul 27, 2026
Merged

Validate stack fuzzing with mutation testing#369
ty-everett merged 2 commits into
mainfrom
codex/mutation-coverage-wave

Conversation

@ty-everett

Copy link
Copy Markdown
Collaborator

Advances #324.

End state

  • pairs every one of the 25 governed property/fuzz suites with a focused Stryker mutation target
  • enforces per-boundary score ratchets, zero uncovered mutants, and zero compile/runtime-error mutants
  • runs affected targets in parallel in required PR CI and all targets in a weekly/manual matrix
  • reuses shared build outputs and retains JSON reports for 30 days
  • strengthens generators and deterministic regressions where the mutation baseline found weak assertions, missing negative cases, or uncorrelated inputs
  • removes equivalent/redundant branches exposed by mutation analysis and hardens SDK Base58 alphabet validation
  • documents local replay, survivor triage, governance, and contribution expectations

Measured mutation evidence

  • 25/25 targets passed their reviewed ratchets with fixed seed 3242026 and 300 property cases
  • 3,073 valid mutants; 2,662 detected; 411 survived
  • aggregate mutation score: 86.63%
  • uncovered mutants: 0
  • invalid compile/runtime mutants: 0
  • target range: 80.30%–100.00%

Validation

  • pnpm build
  • pnpm test (complete 37-project workspace; all required suites passed)
  • pnpm typecheck
  • pnpm lint
  • pnpm format:check
  • pnpm health:check
  • pnpm test:governance
  • pnpm audit:security (high/critical gate passes; one existing moderate advisory remains visible)
  • pnpm test:mutation --all
  • focused final reruns: reorg-stream, btms-topic, btms-permission

No package versions are changed or published by this PR. SonarCloud configuration is intentionally unchanged.

@socket-security

socket-security Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​@​stryker-mutator/​vitest-runner@​9.6.11001007087100
Addednpm/​@​stryker-mutator/​jest-runner@​9.6.1991007588100
Addednpm/​@​stryker-mutator/​core@​9.6.1991009488100

View full report

@socket-security

socket-security Bot commented Jul 27, 2026

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: npm execa is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: pnpm-lock.yamlnpm/@stryker-mutator/core@9.6.1npm/execa@9.6.1

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/execa@9.6.1. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: npm underscore is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: pnpm-lock.yamlnpm/@stryker-mutator/core@9.6.1npm/underscore@1.13.8

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/underscore@1.13.8. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@ty-everett ty-everett left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Admin self-review completed by ty-everett. GitHub does not permit an author to submit an APPROVED review on their own pull request; repository policy requires zero approvals, and admin bypass will be used only after all required automated checks pass.

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ty-everett

Copy link
Copy Markdown
Collaborator Author

Follow-up quality and supply-chain review:

  • Addressed both Sonar findings in b1810c6c0: the mutation runner now uses top-level await, and mutation-governance validation is decomposed into focused validators to remain below the cognitive-complexity limit. Focused governance tests, the full governance inventory, repository health, script lint, formatting, and whitespace checks all pass locally.
  • Reviewed the Socket heuristic warnings rather than suppressing them. execa@9.6.1 is present only through the pinned @stryker-mutator/core@9.6.1 development toolchain; underscore@1.13.8 is present only through typed-rest-client in that same Stryker chain. Both resolve with the lockfile integrity values published by the npm registry, have established upstream repositories and MIT licenses, and introduce no install-script, audit, advisory, or blocking Socket finding. Socket scores remain 99/100 for supply chain and 100/100 for vulnerabilities. These transitive development-only dependencies are therefore retained as an explicitly reviewed Stryker requirement; no blanket ignore or policy weakening was added.

The new commit intentionally retriggers the complete required CI and 25-target mutation matrix so the quality fixes are independently validated before merge.

@sonarqubecloud

Copy link
Copy Markdown

@ty-everett

Copy link
Copy Markdown
Collaborator Author

Final dependency-review annotation assessment:

The seven non-blocking OpenSSF scorecard annotations (tunnel@0.0.6, progress@2.0.3, minimalistic-assert@1.0.1, json-rpc-2.0@1.7.1, js-md4@0.3.2, human-signals@8.0.1, and des.js@1.1.0) were traced with the frozen lockfile. Every one is reachable exclusively through the pinned @stryker-mutator/core@9.6.1 development dependency; none enters a published package or runtime dependency graph. Dependency Review, Socket vulnerability analysis, CodeQL, the repository high/critical audit gate, and the complete CI gate all pass. The warnings are retained visibly as supply-chain signals, with no suppression or threshold reduction; replacing or forking the maintained Stryker toolchain solely to remove low upstream project-health scores would add more risk than it removes.

@ty-everett
ty-everett merged commit 8edb096 into main Jul 27, 2026
65 checks passed
@ty-everett
ty-everett deleted the codex/mutation-coverage-wave branch July 27, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant