b2_03_output_reporting_

Chapter 3: Output Reporting

In Chapter 1: Policy Definition & Evaluation, we learned how to set the rules (Policies), and in Chapter 2: Application Snapshot, we saw how to tell ec-cli what software components (like container images) to check.

Now, imagine the judge has checked the cake against the rulebook. How do they tell you the results? Do they just say "good" or "bad"? Or do they give you a detailed scorecard? And what if you need that scorecard in a specific format, maybe one for the competition organizers and a simpler one for your own notes?

This is where Output Reporting comes in. It's how ec-cli presents the validation results in a clear and structured way.

What Problem Does Output Reporting Solve?

After ec-cli evaluates your software components against the policies, it generates a set of findings: things that passed, things that failed (violations), and things that raised warnings.

But different consumers need these results in different ways:

• A developer running ec-cli locally might want a quick, easy-to-read summary in their terminal. Did it pass or fail? Why?
• A CI/CD pipeline (like Tekton or Jenkins) needs structured data, often in formats like JSON or JUnit XML, to automatically determine if a build step succeeded or failed, and potentially display results in a dashboard.
• An auditing tool might need a detailed JSON or YAML report containing all the findings for record-keeping.

Output Reporting handles this by taking the raw evaluation results and formatting them into various standard formats suitable for different needs.

Our Goal: To understand how ec-cli reports the outcome of policy validations and how we can choose different output formats.

Key Concepts

1. Evaluation Results (The Scorecard Details):

   • When ec-cli runs policies, the outcome for each rule check falls into a few categories:
     • Violations (Failures): A rule explicitly failed. This usually means the software component does not meet a required standard (e.g., contains a critical vulnerability, uses a forbidden base image). Violations typically cause the overall check to fail.
     • Warnings: A rule triggered a warning. This indicates a potential issue or something non-compliant with a best practice, but it might not be severe enough to block a release (e.g., a high-severity vulnerability, missing optional metadata). The overall check might still pass depending on configuration.
     • Successes: A rule passed successfully. The component met the requirement defined by the rule. By default, successes aren't always shown in the output to keep it concise, but you can request them.

2. Output Formats (How the Scorecard Looks):

   • ec-cli can present the results in various formats. Think of these as different templates for the scorecard:
     • text: Human-readable summary printed directly to your terminal. Good for quick checks. (This is often the default).
     • json: Detailed results structured in JSON format. Ideal for machines, scripts, or feeding into other tools.
     • yaml: Similar to JSON but uses YAML structure. Often preferred by humans for readability over JSON for complex data.
     • junit: Standard XML format used by many testing frameworks and CI/CD systems to report test results. Tools like Jenkins can parse this format to show test reports.
     • summary: A condensed JSON format focusing on counts of violations, warnings, and successes per component.
     • appstudio / hacbs: A specific JSON format tailored for integration with Red Hat AppStudio / HACBS environments, reporting results in their expected TEST_OUTPUT structure.
     • vsa: Generates a Verifiable Supply Chain Attestation (VSA) in JSON format, which is a type of attestation confirming the results of the verification.

How to Use Different Output Formats

We use the same ec validate image command from previous chapters, but now we add the --output flag to specify the desired format(s).

Let's say we want to validate the components in our snapshot file my-app-v1.2.0-snapshot.yaml against our policy my-org/standard-policy.

1. Default Output (Human-Readable Text):

If you don't specify --output, you usually get the text format.

ec validate image --images my-app-v1.2.0-snapshot.yaml \
                  --policy my-org/standard-policy

The output might look something like this (simplified):

Success: false
Result: FAILURE
Violations: 1, Warnings: 1, Successes: 15

Components:
  - Name: frontend-component
    Image: quay.io/my-org/frontend:v1.2.0
    Success: false
  - Name: backend-component
    Image: quay.io/my-org/backend:v1.2.0
    Success: true

Results:
  Violations:
    - Component: frontend-component
      Image: quay.io/my-org/frontend:v1.2.0
      Message: Found critical vulnerability CVE-2023-1234
      Code: builtin.policy.critical_vulnerability

  Warnings:
    - Component: frontend-component
      Image: quay.io/my-org/frontend:v1.2.0
      Message: Found high severity vulnerability CVE-2023-5678
      Code: builtin.policy.high_vulnerability

This tells us the overall check failed (Success: false), lists the components, and details the specific violation and warning found in the frontend-component.

2. JSON Output:

For machine processing, use --output json.

ec validate image --images my-app-v1.2.0-snapshot.yaml \
                  --policy my-org/standard-policy \
                  --output json

The output will be a detailed JSON object (showing only a small part):

{
  "success": false,
  "snapshot": "my-app-v1.2.0-snapshot.yaml",
  "components": [
    {
      "name": "frontend-component",
      "containerImage": "quay.io/my-org/frontend:v1.2.0",
      "violations": [
        {
          "msg": "Found critical vulnerability CVE-2023-1234",
          "metadata": { "code": "builtin.policy.critical_vulnerability", /* ... */ }
        }
      ],
      "warnings": [
        {
          "msg": "Found high severity vulnerability CVE-2023-5678",
          "metadata": { "code": "builtin.policy.high_vulnerability", /* ... */ }
        }
      ],
      "successes": [ /* ... many success results ... */ ],
      "success": false
    },
    {
      "name": "backend-component",
      "containerImage": "quay.io/my-org/backend:v1.2.0",
      "violations": [],
      "warnings": [],
      "successes": [ /* ... */ ],
      "success": true
    }
  ],
  "policy": { /* ... policy details ... */ },
  "ec-version": "x.y.z",
  "effective-time": "..."
}

This contains all the information in a structured way, perfect for scripts or other tools.

3. JUnit Output (for CI Systems):

To integrate with CI dashboards that understand JUnit XML.

ec validate image --images my-app-v1.2.0-snapshot.yaml \
                  --policy my-org/standard-policy \
                  --output junit=report.xml

This command will generate a file named report.xml. Notice the =report.xml part - this tells ec-cli to write the JUnit output to that specific file instead of printing it to the screen.

The report.xml file will look something like this (simplified):

<testsuites>
  <testsuite name="frontend-component (quay.io/my-org/frontend:v1.2.0)" timestamp="..." tests="3" failures="1" skipped="1">
    <properties>
      <property name="image" value="quay.io/my-org/frontend:v1.2.0"></property>
      <property name="success" value="false"></property>
      <!-- ... other properties ... -->
    </properties>
    <testcase name="builtin.policy.critical_vulnerability: Found critical vulnerability CVE-2023-1234 [...]" classname="...">
      <failure message="Found critical vulnerability CVE-2023-1234"></failure>
    </testcase>
    <testcase name="builtin.policy.high_vulnerability: Found high severity vulnerability CVE-2023-5678 [...]" classname="...">
      <skipped message="Found high severity vulnerability CVE-2023-5678"></skipped>
    </testcase>
    <testcase name="builtin.policy.allowed_base: Image base is allowed [...]" classname="...">
      <!-- Success: No failure or skipped element -->
    </testcase>
     <!-- ... other test cases ... -->
  </testsuite>
  <testsuite name="backend-component (quay.io/my-org/backend:v1.2.0)" ...>
    <!-- Test cases for the backend component -->
  </testsuite>
</testsuites>

JUnit represents violations as <failure> elements and warnings as <skipped> elements (this is a common convention). Successes are test cases without failure or skipped elements. CI systems can parse this file to display detailed test results.

4. Multiple Outputs:

You can even generate multiple reports at once by providing the --output flag multiple times.

ec validate image --images my-app-v1.2.0-snapshot.yaml \
                  --policy my-org/standard-policy \
                  --output text \
                  --output json=detailed_report.json \
                  --output junit=ci_report.xml

This will print the text report to the console, save a JSON report to detailed_report.json, and save a JUnit report to ci_report.xml.

Under the Hood: How Reporting Works

1. Gather Raw Results: After the policy evaluation engine (Chapter 1: Policy Definition & Evaluation) finishes checking each component defined in the Application Snapshot, ec-cli collects all the raw outcomes (violations, warnings, successes) for each component.
2. Build Report Structure: It organizes these raw results into an internal Report data structure. This structure holds information about the overall success, details about each component, the policy used, etc.
3. Parse Output Flags: ec-cli checks the --output flags provided by the user to determine which format(s) are needed and where to write them (console or file).
4. Format the Output: For each requested output:
   • It selects the appropriate formatting logic (e.g., JSON marshalling, YAML marshalling, JUnit generation function, text template rendering).
   • It converts the internal Report structure into the target format.
5. Write the Output: It writes the formatted data to the specified destination (standard output or a file).

Simplified Sequence Diagram:

sequenceDiagram
    participant User
    participant CLI as ec validate image
    participant Evaluator as Policy Engine
    participant Reporter as Output Formatter
    participant Destination as Console/File

    User->>CLI: Run with --output json=report.json --output text
    CLI->>Evaluator: Evaluate components against policy
    Evaluator-->>CLI: Raw Results (Violations, Warnings, Successes)
    CLI->>Reporter: Build internal Report struct from Raw Results
    CLI->>Reporter: Request 'json' format
    Reporter-->>CLI: Formatted JSON data
    CLI->>Destination: Write JSON data to report.json
    CLI->>Reporter: Request 'text' format
    Reporter-->>CLI: Formatted Text data
    CLI->>Destination: Write Text data to Console
    Destination-->>User: Show Text Output

Loading

Diving into the Code (A Glimpse)

1. The Report Structure (internal/applicationsnapshot/report.go)

This file defines the Report struct that holds all the information before formatting.

// Simplified concept from internal/applicationsnapshot/report.go
package applicationsnapshot

// ... imports ...

type Report struct {
	Success       bool `json:"success"`
	// ... other fields like Snapshot, Key, Policy, EcVersion ...
	Components    []Component                      `json:"components"`
	created       time.Time
	// ... internal fields like PolicyInput, ShowSuccesses ...
}

type Component struct {
	app.SnapshotComponent // Embeds Name, ContainerImage etc.
	Violations   []evaluator.Result          `json:"violations,omitempty"`
	Warnings     []evaluator.Result          `json:"warnings,omitempty"`
	Successes    []evaluator.Result          `json:"successes,omitempty"`
	Success      bool                        `json:"success"`
	// ... other component details like Signatures ...
}

// NewReport creates the main Report structure from evaluation results
func NewReport(snapshot string, components []Component, policy policy.Policy, /*...*/) (Report, error) {
    success := true
    for _, component := range components {
        if !component.Success {
            success = false
            break
        }
    }
    // ... get policy key, version info ...
	return Report{
		Snapshot:      snapshot,
		Success:       success,
		Components:    components,
		created:       time.Now().UTC(),
		// ... fill other fields ...
	}, nil
}

This shows how the Report struct organizes the overall status and a list of Component structs, each holding its specific violations, warnings, and successes. NewReport constructs this primary data structure.

2. Formatting Logic (internal/applicationsnapshot/report.go)

The toFormat method acts as a dispatcher, calling the right function based on the requested format.

// Simplified concept from internal/applicationsnapshot/report.go
func (r *Report) toFormat(format string) (data []byte, err error) {
	switch format {
	case JSON:
		data, err = json.Marshal(r) // Use standard JSON marshalling
	case YAML:
		data, err = yaml.Marshal(r) // Use YAML marshalling
	case Text:
		data, err = generateTextReport(r) // Call text template renderer
	case JUnit:
		data, err = xml.Marshal(r.toJUnit()) // Call JUnit generator and XML marshal
	case Summary:
		data, err = json.Marshal(r.toSummary()) // Call summary generator
	case AppStudio, HACBS:
		data, err = json.Marshal(r.toAppstudioReport()) // Call AppStudio format generator
	// ... other cases like VSA, PolicyInput ...
	default:
		return nil, fmt.Errorf("%q is not a valid report format", format)
	}
	return
}

This switch statement is the core of the formatting dispatch. It takes the Report struct (r) and converts it to bytes (data) based on the format string.

3. Generating JUnit (internal/applicationsnapshot/junit.go)

This file contains the logic specific to creating the JUnit XML structure.

// Simplified concept from internal/applicationsnapshot/junit.go
import (
	// ... imports ...
	"github.com/jstemmer/go-junit-report/v2/junit"
)

// toJUnit converts the Report struct to JUnit format
func (r *Report) toJUnit() junit.Testsuites {
	report := junit.Testsuites{} // Create the top-level JUnit structure

	for _, component := range r.Components {
		// Create a test suite for each component
		suite := junit.Testsuite{
			Name: fmt.Sprintf("%s (%s)", component.Name, component.ContainerImage),
			// ... add properties like image name, overall success ...
		}

		// Add test cases for successes, violations, and warnings
		mapResults(&suite, component.Successes, asTestCase)
		mapResults(&suite, component.Violations, func(res evaluator.Result) junit.Testcase {
			tc := asTestCase(res)
			tc.Failure = &junit.Result{Message: res.Message} // Mark as failure
			return tc
		})
		mapResults(&suite, component.Warnings, func(res evaluator.Result) junit.Testcase {
			tc := asTestCase(res)
			tc.Skipped = &junit.Result{Message: res.Message} // Mark as skipped (warning)
			return tc
		})

		report.AddSuite(suite) // Add this component's suite to the report
	}
	return report
}

// asTestCase converts an evaluator.Result into a basic junit.Testcase
func asTestCase(r evaluator.Result) junit.Testcase {
	// ... format name and description from result metadata ...
	return junit.Testcase{ Name: /* formatted name */, Classname: /*...*/ }
}

This shows how it iterates through components, creates a <testsuite> for each, and then adds <testcase> elements for each success, violation (marked with <failure>), and warning (marked with <skipped>).

4. Handling Output Destinations (internal/format/target.go)

This package handles parsing the --output flag value (like json=path/to/file.json or just text) and directs the output to the correct place (a file or standard output).

// Simplified concept from internal/format/target.go
package format

import "io"

// Target represents where to write output and in what format
type Target struct {
	Format  string
	Options Options // e.g., ShowSuccesses
	writer  io.Writer // Where to write (stdout, file, etc.)
}

// TargetParser figures out the format and writer from the --output flag string
type TargetParser struct {
	// ... default format, writer, filesystem handle ...
}

// Parse takes the string from --output flag (e.g., "junit=report.xml")
// and returns a Target object.
func (tm *TargetParser) Parse(given string) (*Target, error) {
	target := Target{writer: tm.defaultWriter} // Default to stdout
	// ... parse "format=path?options" string ...
	formatAndPath, opts, _ := strings.Cut(given, "?")
	target.Format, path, _ := strings.Cut(formatAndPath, "=")
	// ... handle options like ?show-successes=true ...

	if path != "" {
		// If a path was given (like junit=report.xml), set writer to a fileWriter
		target.writer = &fileWriter{path: path, fs: tm.fs}
	}
	// ... set target.Format, handle defaults ...
	return &target, nil
}

// Write sends the data to the configured writer (stdout or file)
func (t *Target) Write(data []byte) (int, error) {
	return t.writer.Write(data)
}

This code shows how Parse decodes the output string to figure out the format, destination file (if any), and options. The Write method then sends the already-formatted report data to the correct io.Writer.

Conclusion

Output Reporting is the final step in the validation process, where ec-cli communicates the results back to you or your automation systems. We learned that:

1. Results are categorized into Violations, Warnings, and Successes.
2. ec-cli can format these results into various Output Formats (text, json, yaml, junit, etc.) using the --output flag.
3. Different formats are suited for different needs (human reading vs. machine processing).
4. You can specify output files (e.g., --output junit=report.xml) and generate multiple reports simultaneously.

Understanding how to get and interpret the output is crucial for integrating ec-cli into your workflows.

Now that we've covered policies, snapshots, and reporting, you might wonder: where do the policies and the image information actually come from? Often, they reside in OCI registries. In the next chapter, we'll explore how ec-cli interacts with these registries.

Let's move on to Chapter 4: OCI Interaction.

---

Generated by AI Codebase Knowledge Builder