Skip to content

feat: adversarial synthetic generator for ImpactGuard reliability testing#23

Merged
daedalus merged 2 commits intomasterfrom
copilot/add-synthetic-generator-test
May 6, 2026
Merged

feat: adversarial synthetic generator for ImpactGuard reliability testing#23
daedalus merged 2 commits intomasterfrom
copilot/add-synthetic-generator-test

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 6, 2026

Adds a synthetic generator that produces signature pairs where a breaking API change is camouflaged as a non-breaking one, stress-testing whether ImpactGuard's detection is reliable against deceptive diffs.

New module: adversarial_generator.py

10 camouflage strategies, each producing an AdversarialPair with old/new signature snapshots, expected breaking patterns, and human-readable deception notes:

Strategy Hidden breaking change Camouflage
required_param_hidden_by_type_annotation Required positional added str → str | None widening on sibling param
positional_reorder_hidden_by_optional_add Param order swapped Optional third param added
type_narrowing_disguised_as_cleanup str | None → str narrowing Looks like type-safety tightening
kwonly_removal_with_optional_addition Required kwarg removed Optional kwarg added
vararg_removal_with_kwarg_addition *args removed **kwargs added
return_type_narrowing_disguised_as_guarantee str | None → str return "Now always returns a value"
decorator_removal_hidden_by_optional_kwarg @cache removed Optional kwarg suggests cache control
required_kwonly_added_with_misleading_name Required kwarg added Named optional_context with no default
function_removal_hidden_by_rename_addition Old function removed _v2 variant added alongside
kwargs_removal_hidden_by_explicit_params **kwargs removed Explicit optional params added
from impactguard import generate_adversarial, generate_all_adversarial, list_adversarial_strategies

pair = generate_adversarial("required_kwonly_added_with_misleading_name")
# pair.old_signatures / pair.new_signatures → pass to compare()
# pair.expected_breaking_patterns → what ImpactGuard must catch
# pair.camouflage_notes → explains the deception

ok, failures = pair.verify(compare(old_path, new_path))

Tests (test_adversarial_generator.py)

54 tests verifying ImpactGuard catches every hidden breaking change — including parametrized bulk detection across all 10 strategies, camouflage noise presence checks, and AdversarialPair.verify() unit tests.

Exports

AdversarialPair, generate_adversarial, generate_all_adversarial, list_adversarial_strategies added to __init__.py / __all__.

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Copilot, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@daedalus daedalus marked this pull request as ready for review May 6, 2026 20:56
@daedalus daedalus merged commit 3107519 into master May 6, 2026
1 check was pending
@daedalus daedalus deleted the copilot/add-synthetic-generator-test branch May 6, 2026 20:56
@codacy-production
Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 24 high · 21 minor

Alerts:
⚠ 45 issues (≤ 0 issues of at least minor severity)

Results:
45 new issues

Category Results
UnusedCode 1 minor
Documentation 20 minor
Security 24 high

View in Codacy

🟢 Metrics 89 complexity · 2 duplication

Metric Results
Complexity 89
Duplication 2

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants