Skip to content

test: strengthen weak assertions in safe_outputs analyzer tests#891

Merged
jamesadevine merged 1 commit into
mainfrom
test-reducer/safe-outputs-strengthen-assertions-d12a872f0c7eb108
Jun 7, 2026
Merged

test: strengthen weak assertions in safe_outputs analyzer tests#891
jamesadevine merged 1 commit into
mainfrom
test-reducer/safe-outputs-strengthen-assertions-d12a872f0c7eb108

Conversation

@github-actions

@github-actions github-actions Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Test Suite Reduction: src/audit/analyzers/safe_outputs.rs

What was wrong

  • aggregate_detection_gate_rejects_all_proposals: The test never checked the summary struct at all — meaning not_processed_count (the primary metric tracking how many proposals were rejected by the detection gate) was completely unasserted. Additionally, only findings[0].severity was checked; category and title were not verified, so a regression in the finding content would have passed silently.

  • mixed_execution_outcomes_are_rolled_up: The rollup struct was asserted to exist and have specific by_reason entries, but rollup.total_rejected (the headline count) was never pinned. A bug in total_rejected = rejected_by_execution_count + not_processed_count could go undetected.

Changes

Test Action Reason
aggregate_detection_gate_rejects_all_proposals Rewritten (assertions added) Missing summary block; finding content unverified
mixed_execution_outcomes_are_rolled_up Rewritten (assertion added) rollup.total_rejected was unasserted

aggregate_detection_gate_rejects_all_proposals now asserts:

  • summary.proposed_count == 2
  • summary.not_processed_count == 2 (key gate-rejection metric)
  • summary.executed_count == 0
  • summary.rejected_by_execution_count == 0
  • findings[0].category == "safe_outputs"
  • findings[0].title == "Detection rejected 2 safe output(s)"

mixed_execution_outcomes_are_rolled_up now asserts:

  • rollup.total_rejected == 2

Verification

  • cargo test: all 1766 tests pass ✅
  • cargo clippy --all-targets --all-features: no warnings or errors ✅

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • spsprodeus21.vssps.visualstudio.com
  • spsprodweu4.vssps.visualstudio.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "spsprodeus21.vssps.visualstudio.com"
    - "spsprodweu4.vssps.visualstudio.com"

See Network Configuration for more information.

Generated by Test Reducer · sonnet46 2.8M ·

In aggregate_detection_gate_rejects_all_proposals:
- Add summary assertions (proposed_count, not_processed_count,
  executed_count, rejected_by_execution_count) — the primary metric
  for detection-gate rejection (not_processed_count) was completely
  unchecked.
- Add finding.category and finding.title assertions — only severity
  was checked before, meaning a regression in the finding content
  would have passed silently.

In mixed_execution_outcomes_are_rolled_up:
- Add rollup.total_rejected assertion — the field was computed but
  never asserted, leaving its correctness untested.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jamesadevine jamesadevine marked this pull request as ready for review June 7, 2026 16:54
@jamesadevine jamesadevine merged commit 6e36a86 into main Jun 7, 2026
@jamesadevine jamesadevine deleted the test-reducer/safe-outputs-strengthen-assertions-d12a872f0c7eb108 branch June 7, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant