Skip to content

Canary v2.0.0 — Repository-Aware Semantic Regression Analysis

Latest
Latest

Choose a tag to compare

View all tags
@cybr-wisp cybr-wisp released this 24 Aug 22:27
· 10 commits to main since this release
v2.0.0
0daf75b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Canary v2.0.0 🐤

Canary v2 introduces repository-aware semantic regression analysis for Python pull requests.

Instead of stopping at “this API changed,” Canary can now trace that change across the repository, find affected call sites, and determine when an existing call is no longer compatible with the new interface.

API change
    ↓
Python AST analysis
    ↓
semantic compatibility detection
    ↓
repository-wide call-site discovery
    ↓
impact analysis
    ↓
argument-aware validation
    ↓
confirmed breakages

What's new

Semantic API compatibility analysis

Canary now parses BASE and HEAD versions of changed Python files using the Python AST and detects:

  • removed public APIs
  • newly added required parameters
  • removed parameters
  • reordered parameters
  • removed parameter defaults
  • return annotation changes
  • sync ↔ async behavior changes

Example:

-def authenticate(token):
+def authenticate(token, strict):
     ...

Canary reports:

REQUIRED_PARAMETER_ADDED

and records the old and new callable signatures as technical evidence.

Repository-wide call-site analysis

Canary now builds a lightweight repository symbol and call index.

When an API changes, Canary can locate callers across other Python files rather than analyzing only the changed diff.

For example:

from auth import authenticate

def login():
    return authenticate("demo-token")

If authenticate() gains a new required argument, Canary connects the API change to this existing caller.

Argument-aware compatibility validation

Discovered call sites are validated against the changed function signature.

Canary accounts for:

  • positional arguments
  • keyword arguments
  • required and optional parameters
  • positional-only parameters
  • keyword-only parameters
  • excess positional arguments
  • unexpected keywords
  • duplicate argument binding
  • *args
  • **kwargs
  • awaited and non-awaited calls

Each call site is classified as:

Status Meaning
BREAKS The call can be statically confirmed incompatible
UNAFFECTED The call remains compatible
UNKNOWN Static analysis cannot safely determine compatibility

Repository impact and blast-radius analysis

Semantic findings now carry repository impact information.

Canary reports:

  • call sites analyzed
  • confirmed breaking calls
  • compatible calls
  • calls requiring manual review
  • exact affected file and line locations

This separates a potentially risky API change from a confirmed repository breakage.

Richer GitHub Checks

GitHub Checks now include repository-aware impact information alongside the original risk analysis.

A breaking change can produce output such as:

🔴 HIGH RISK

REQUIRED_PARAMETER_ADDED

Required parameter `strict` was added to `authenticate`.

Repository impact
1 call site analyzed
1 confirmed breaking

Confirmed breakages
caller.py:5

Canary also provides:

  • HIGH / MEDIUM / LOW risk summaries
  • inline annotations
  • technical old → new signature evidence
  • repository-impact tables
  • merge-review recommendations

High-risk findings fail the Canary Check so regressions are visible before merge.

Richer CLI analysis

The terminal now exposes the same semantic impact information as the GitHub App:

canary inspect https://github.com/owner/repository/pull/123

GitHub Checks and the CLI share the same analysis engine.

Architecture

Canary v2 expands the original diff-based pipeline into:

GitHub PR
    ↓
BASE / HEAD source snapshots
    ↓
Python AST
    ↓
Compatibility Engine
    ↓
Repository Symbol / Call Index
    ↓
Impact Analysis
    ↓
Argument-Aware Call Validation
    ↓
AnalysisResult
    ↓
GitHub Check / CLI

The analysis remains deterministic and explainable and does not rely on an LLM.

Live validation

v2 was smoke-tested end-to-end against a real GitHub pull request containing an intentional breaking API change.

Canary successfully:

  • detected the newly required parameter
  • classified the finding as HIGH risk
  • found the affected cross-file call site
  • confirmed the call was incompatible
  • reported the exact caller.py:5 location
  • produced an inline GitHub annotation
  • failed the GitHub Check as expected

Test suite

Canary v2.0.0 ships with:

88 passing tests

covering:

  • AST extraction
  • semantic compatibility rules
  • repository symbol analysis
  • cross-file call-site discovery
  • impact analysis
  • argument-aware validation
  • pull-request orchestration
  • GitHub integration
  • GitHub Check presentation
  • CLI behavior
  • terminal presentation
  • webhook behavior
Assets 2
Loading