Skip to content

Conversation

Piskoo
Copy link
Collaborator

@Piskoo Piskoo commented Aug 11, 2025

This PR adds error when no evaluations on provided material were performed.

Example

apiVersion: workflowcontract.chainloop.dev/v1
kind: Policy
metadata:
  name: policy
  description: Chainloop validation policy
spec:
  policies:
    - embedded: |
        package main

        import rego.v1

        ################################
        # Common section do NOT change #
        ################################

        result := {
          "skipped": skipped,
          "violations": violations,
          "skip_reason": skip_reason,
          "ignore": ignore,
        }

        default skip_reason := ""

        skip_reason := m if {
          not valid_input
          m := "invalid input"
        }

        default skipped := true

        skipped := false if valid_input

        default ignore := false

        ########################################
        # EO Common section, custom code below #
        ########################################
        # Validates if the input is valid and can be understood by this policy
        valid_input := true

        # insert code here

        # If the input is valid, check for any policy violation here
        # default violations := []

        violations contains msg if {
          valid_input
          msg := "test"
        }
      kind: EVIDENCE
    - embedded: |
        package main

        import rego.v1

        ################################
        # Common section do NOT change #
        ################################

        result := {
          "skipped": skipped,
          "violations": violations,
          "skip_reason": skip_reason,
          "ignore": ignore,
        }

        default skip_reason := ""

        skip_reason := m if {
          not valid_input
          m := "invalid input"
        }

        default skipped := true

        skipped := false if valid_input

        default ignore := false

        ########################################
        # EO Common section, custom code below #
        ########################################
        # Validates if the input is valid and can be understood by this policy
        valid_input := true

        # insert code here

        # If the input is valid, check for any policy violation here
        # default violations := []

        violations contains msg if {
          valid_input
          msg := "test"
        }
      kind: STRING

For policy with no defined path for kind SBOM_CYCLONEDX_JSON when validated against valid cyclonedx json file it outputs:

ERR evaluating policy: no execution branch matched for kind SBOM_CYCLONEDX_JSON
exit status 1

Closes #2334

Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>
@Piskoo Piskoo marked this pull request as ready for review August 11, 2025 11:53
@Piskoo Piskoo requested review from jiparis and migmartri August 11, 2025 11:53

// Check if any result was ignored
for _, res := range result {
if res.Ignored {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but does this also include if it was ignored on purpose? I mean, would ignore mean only that no script was evaluated?

cc/ @jiparis

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also do you think we could be more user friendly by including the kind, and the available kinds in the error output for example?

I'd like to make this tool super user friendly that would mean in general\

Copy link
Member

@jiparis jiparis Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignored means: The script was evaluated, but it must be ignored because it doesn't apply, so it will count as it if wasn't evaluated at all.
Note that this is different to skipped, where the evaluation still counts.
Ignored is for some cases where an input file has multiple purposes (CycloneDX for SBOMs and also for vulnerabilities report, for example)

So in this case, what we want to check whether result is empty.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to return error only when no eval results were returned. Included material kind in the error msg. Available kinds require some more work I can create another issue to address that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please write the description of the PR with some examples of outputs so we can understand better what it solves? Thanks!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the description

Piskoo added 2 commits August 11, 2025 15:52
Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>
Signed-off-by: Sylwester Piskozub <sylwesterpiskozub@gmail.com>
Copy link
Member

@migmartri migmartri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@migmartri migmartri merged commit 6744d32 into chainloop-dev:main Aug 11, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

policy eval should fail if no execution path is met

3 participants