policy develop eval: expose structured violation data in output

[migmartri]

Context

With the introduction of structured policy violation output (finding_type + chainloop.vulnerability() built-in), policies can now return rich violation objects containing fields like external_id, package_purl, severity, cvss_v3_score, and cwes.

However, chainloop policy develop eval only outputs the message field from these structured violations — the additional structured fields are invisible in the output. They can only be seen via --debug mode buried in raw_results.

Problem

There is no way to write integration tests that verify the structured fields are correct. The golden file comparison only sees the message strings, so a policy could return wrong package_purl or severity values and tests would still pass.

Proposal

Add an option to chainloop policy develop eval (e.g., --structured or --output-format=structured) that outputs the full violation objects instead of just the message strings:

{
  "result": {
    "violations": [
      {
        "message": "Vulnerability CVE-2023-45288...",
        "external_id": "CVE-2023-45288-stdlib",
        "package_purl": "pkg:golang/stdlib@1.21.5",
        "severity": "HIGH",
        "cvss_v3_score": 7.5
      }
    ]
  }
}

This would allow the integration test suite in compliance-manifests to verify structured output correctness.

Related

• PR chainloop-dev/compliance-manifests#257 (migrates vulnerabilities policy to structured output)
• Spec 015: Structured Policy Violation Output