Skip to content

fix(core): a tolerated skip erases an earlier real failure — verdicts depend on resource order #293

Description

@refeed

core/core.py:95 assigns has_evaluation_passed = None unconditionally in the tolerated-skip
branch. When one resource violates the condition and a later resource of the same type is
tolerated away, the evaluator's verdict becomes None, and the None id is then deleted from the
eval_expression AST — the violation disappears. Reachable at every error_tolerance: a
destroyed resource has change.after: null → severity 0, and core.py:87 compares with a strict
>, so any terraform_plan policy is exposed on any plan that destroys a resource of the type it
checks.

Reproduction. Two resources of the same type, one violating; evaluate the plan twice with only
their order swapped. Violation-first returns null, destroy-first returns false — verified at
tolerances 0, 1 and 2. Under a negated expression (!id) the flip is ERROR → SUCCESS: a policy
that used to block a pipeline passes it.

Fix. Guard so None never overwrites False. Note the naive guard
(if has_skipped and not has_valid_evaluation) is fail-open — the hard-failure branches at
core.py:78-81 and :87-91 set False and continue without marking a valid evaluation. The
correct condition is if has_skipped and has_evaluation_passed is True and not has_valid_evaluation.
Decide the [PASS, skip, PASS] → None rollup in the same PR (today it prints "Passed: 0 Failed: 0
Skipped: 1" and exits 1).

Acceptance. Both orderings as regression tests; CHANGELOG labels this a verdict change, not a
compatibility-preserving cleanup.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions