b1_07_output_reporting_

Chapter 7: Output Reporting

Welcome to the final core chapter of our ec-cli tutorial! In the previous chapter, we learned how ec-cli can track trusted versions of build components using Tekton Bundle Tracking.

So far, we've seen how ec-cli takes inputs (Application Snapshot), applies rules (Policy Definition & Sources), uses an engine to evaluate (Validation Engine (Evaluator)), verifies trust (Sigstore Verification), fetches artifacts (Image & OCI Artifact Handling), and tracks pipeline components (Tekton Bundle Tracking).

But after all that work, how do we actually see the results? Did our image pass the policy checks? What failed? And can we get this information in a format that other tools can understand?

This is where Output Reporting comes in.

What Problem Does it Solve? Presenting the Verdict

Imagine you've run ec validate image --images my-app-snapshot.yaml --policy my-company-policy.yaml. ec-cli has done its job, running all the checks. Now you need the final report.

• Use Case 1 (Human User): You just ran the command and want to quickly see on your screen if everything passed or what the specific failures were.
• Use Case 2 (CI/CD Pipeline): You're running ec-cli in an automated Jenkins or GitLab pipeline. You need the results in a standard format like JUnit XML so the pipeline can display test results clearly.
• Use Case 3 (Further Processing): Maybe you want to feed the detailed results into another script or dashboard. You need the output in a machine-readable format like JSON or YAML.
• Use Case 4 (Integration): You're using a platform like Red Hat AppStudio, which expects results in a specific JSON structure.

Output Reporting handles taking the raw results from the internal checks and formatting them into these different outputs, depending on what you need.

The Court Reporter Analogy

Think back to our judge analogy for the Validation Engine (Evaluator). The judge (engine) delivers a verdict based on evidence and rules.

The Output Reporting component is like the court reporter. Its job is to:

1. Listen to all the findings and the final verdict from the judge.
2. Carefully document everything.
3. Prepare the official record of the trial's outcome.
4. Produce this record in different formats as requested – a typed transcript for the lawyers (human-readable), a structured digital file for the court database (JSON/YAML), or a standardized report for legal journals (JUnit).

It takes the internal results and makes them presentable and usable for different audiences.

Supported Output Formats

ec-cli supports several output formats:

• text (Default): A human-readable summary printed directly to your console. Good for quick interactive checks.
• json: Detailed results structured in JSON format. Ideal for machine consumption, scripting, or detailed analysis.
• yaml: Detailed results structured in YAML format. Similar to JSON, often preferred for readability by humans looking at structured data.
• junit: Results formatted as JUnit XML, a standard format widely used in CI/CD systems to display test results.
• appstudio / hacbs: A specific JSON format tailored for integration with Red Hat AppStudio / Hybrid Application Cloud Build Service.
• summary: A condensed JSON overview, less verbose than the full json output.
• summary-markdown: A condensed overview formatted as Markdown, suitable for embedding in reports or comments.
• attestation: Outputs the results formatted as an in-toto attestation, which can be signed and stored to prove the validation occurred.
• vsa: Outputs a Verification Summary Attestation (VSA), another type of in-toto attestation specifically summarizing the verification results.
• policy-input: (Advanced) Outputs the combined JSON input data that was fed into the Validation Engine (Evaluator). Useful for debugging policies.

How to Choose Your Output Format

You control the output format using the --output flag when running commands like ec validate image.

Example 1: Default Human-Readable Output

If you don't specify --output, you get the text format.

ec validate image --images snapshot.yaml --policy policy.yaml

(Output will appear directly in your terminal, summarizing success/failure and listing violations/warnings)

Example 2: JSON Output

ec validate image --images snapshot.yaml --policy policy.yaml --output json

(Output will be a detailed JSON document printed to your terminal)

Example 3: Writing JUnit XML to a File

You can specify a filename after the format using =.

ec validate image --images snapshot.yaml --policy policy.yaml --output junit=test-results.xml

(No output to the terminal. A file named test-results.xml will be created with the JUnit report.)

Example 4: Multiple Outputs

You can specify --output multiple times to generate multiple reports simultaneously.

ec validate image --images snapshot.yaml --policy policy.yaml \
  --output text \
  --output json=report.json \
  --output junit=results.xml

(This will print the human-readable report to the terminal, create report.json with JSON data, AND create results.xml with the JUnit report.)

Example 5: Output Options

Some formats might have options. You can specify them using a ? after the format name, like URL query parameters. For example, to include successful checks in the JSON report (they are omitted by default):

ec validate image --images snapshot.yaml --policy policy.yaml --output json?show-successes=true

(The JSON output will now include details about checks that passed successfully.)

Under the Hood: From Raw Results to Formatted Report

Let's trace the journey from internal results to the final output file or console display.

1. Gathering Results: Commands like ec validate image run various checks:
   • Image Accessibility (Image & OCI Artifact Handling)
   • Signature Verification (Sigstore Verification)
   • Attestation Syntax & Signature Checks
   • Policy Evaluation (Validation Engine (Evaluator))
2. Building the Report Object: The results of all these checks (pass/fail status, messages, violations, warnings, successes from the policy engine, signatures found) are collected into internal Go structures, primarily output.Output which then gets converted into applicationsnapshot.Report and applicationsnapshot.Component. Think of this as the raw data collected by the court reporter.
3. Parsing Output Request: ec-cli parses the --output flag(s) to understand which formats are requested and where to write them (console or file).
4. Formatting: For each requested output target, the internal Report object is passed to a specific formatter function (e.g., toJUnit, toJSON, generateTextReport).
5. Writing: The formatter function generates the output as a byte stream (e.g., XML text, JSON text), which is then written to the specified destination (standard output or a file).

sequenceDiagram
    participant User
    participant EC_CLI as ec validate image
    participant Checker as Internal Checks (SigVerify, Evaluator, etc.)
    participant ReportBuilder as Report Builder
    participant Formatter as Output Formatter
    participant Destination as Console/File

    User->>EC_CLI: Runs `ec validate ... --output <format>[=file][?opts]`
    EC_CLI->>Checker: Perform validation checks
    Checker-->>EC_CLI: Raw Results (Pass/Fail, Violations, Warnings, etc.)
    EC_CLI->>ReportBuilder: Build internal Report object from Raw Results
    ReportBuilder-->>EC_CLI: Internal Report Object
    EC_CLI->>Formatter: Format Report object as `<format>` with options `<opts>`
    Formatter-->>EC_CLI: Formatted Bytes (JSON, XML, text, etc.)
    EC_CLI->>Destination: Write Formatted Bytes to Console or `<file>`
    Destination-->>User: Shows output / Saves file

Loading

Code Snippets

Let's look at some simplified code concepts related to output reporting.

1. The Report Structure (internal/applicationsnapshot/report.go) This structure holds the collected results for all components checked.

   // Simplified view of the main report structure
   type Report struct {
       Success       bool                             `json:"success"` // Overall success status
       Components    []Component                      `json:"components"` // Results for each component
       Policy        ecc.EnterpriseContractPolicySpec `json:"policy"` // Policy used
       EcVersion     string                           `json:"ec-version"` // Version of ec-cli
       // ... other fields like Key, EffectiveTime ...
   
       // Internal fields used during processing
       created       time.Time
       ShowSuccesses bool // Option parsed from --output flag
   }
   
   // Simplified view of the structure holding results for one component
   type Component struct {
       app.SnapshotComponent // Embeds info like name, image URL from snapshot
       Violations   []evaluator.Result          `json:"violations,omitempty"` // Policy failures
       Warnings     []evaluator.Result          `json:"warnings,omitempty"`   // Policy warnings
       Successes    []evaluator.Result          `json:"successes,omitempty"`  // Policy successes (if requested)
       Success      bool                        `json:"success"`              // Did this component pass all checks?
       Signatures   []signature.EntitySignature `json:"signatures,omitempty"` // Signatures found/verified
       // ... other fields like Attestations ...
   }

   Explanation: The Report acts as the main container for the validation outcome. It holds a list of Component structs, each detailing the results (Violations, Warnings, Successes, overall Success status) for a specific container image defined in the Application Snapshot.

2. Parsing the Output Target (internal/format/target.go) This code handles parsing the string provided to --output.

   // Simplified structure representing a single output destination and format
   type Target struct {
       Format  string    // e.g., "json", "junit", "text"
       Options Options   // e.g., ShowSuccesses=true
       writer  io.Writer // Where to write (console or file)
   }
   
   // Simplified structure for output options
   type Options struct {
       ShowSuccesses bool
   }
   
   // Simplified parsing logic
   func (tm *TargetParser) Parse(given string) (*Target, error) {
       target := Target{writer: tm.defaultWriter} // Start with default writer (console)
   
       // Split format/path from options (e.g., "json?show-successes=true")
       formatAndPath, optsStr, foundOpts := strings.Cut(given, "?")
   
       // Parse options if present
       target.Options = tm.defaultOptions // Start with default options
       if foundOpts {
           // simplified: parse optsStr like URL query params (show-successes=true)
           // and update target.Options
           if err := target.Options.mutate(optsStr); err != nil {
               return nil, err
           }
       }
   
       // Split format from path (e.g., "junit=results.xml")
       var path string
       target.Format, path, _ = strings.Cut(formatAndPath, "=")
   
       if target.Format == "" {
           target.Format = tm.defaultFormat // Use default format if none specified
       }
   
       // If a path was given, set up a file writer
       if path != "" {
           target.writer = &fileWriter{path: path, fs: tm.fs}
       }
   
       return &target, nil
   }

   Explanation: The Parse function takes the string from the --output flag (like junit=results.xml or json?show-successes=true). It splits the format name (junit, json), the optional file path (results.xml), and any options (show-successes=true). It configures a Target object with the correct format, options, and writer (either the console or a file).

3. Generating Output (internal/applicationsnapshot/report.go) The WriteAll method orchestrates writing the report in all requested formats.

   // Simplified WriteAll method
   func (r Report) WriteAll(targets []string, parser format.TargetParser) (allErrors error) {
       if len(targets) == 0 {
           targets = append(targets, Text) // Default to "text" if no --output given
       }
   
       // Loop through each requested target string (e.g., "text", "json=report.json")
       for _, targetName := range targets {
           // Parse the string into a Target object (format, options, writer)
           target, err := parser.Parse(targetName)
           if err != nil { /* handle error */ continue }
   
           // Apply options (like ShowSuccesses) to the report data before formatting
           r.applyOptions(target.Options)
   
           // Call the formatter for the requested format
           data, err := r.toFormat(target.Format)
           if err != nil { /* handle error */ continue }
   
           // Ensure output ends with a newline
           if !bytes.HasSuffix(data, []byte{'\n'}) {
               data = append(data, "\n"...)
           }
   
           // Write the formatted data using the target's writer (console or file)
           if _, err := target.Write(data); err != nil {
               allErrors = errors.Join(allErrors, err)
           }
       }
       return allErrors
   }
   
   // Simplified toFormat method (the dispatcher)
   func (r *Report) toFormat(format string) (data []byte, err error) {
       switch format {
       case JSON:
           data, err = json.Marshal(r) // Use Go's standard JSON library
       case YAML:
           data, err = yaml.Marshal(r) // Use a YAML library
       case Text:
           data, err = generateTextReport(r) // Use a text template
       case JUnit:
           data, err = xml.Marshal(r.toJUnit()) // Call specific JUnit formatter
       case AppStudio:
           data, err = json.Marshal(r.toAppstudioReport()) // Call specific AppStudio formatter
       // ... other cases for Summary, VSA, Attestation etc. ...
       default:
           return nil, fmt.Errorf("%q is not a valid report format", format)
       }
       return
   }

   Explanation: WriteAll loops through each string provided via --output. For each, it uses the TargetParser to figure out the format, options, and destination. It then calls toFormat, which acts as a dispatcher, calling the appropriate function (like json.Marshal, yaml.Marshal, toJUnit, generateTextReport) based on the requested format. Finally, it writes the resulting bytes to the target's writer.

4. Specific Formatter Example (internal/applicationsnapshot/junit.go) This function converts the internal Report into the JUnit XML structure.

   // Simplified JUnit formatter
   func (r *Report) toJUnit() junit.Testsuites {
       report := junit.Testsuites{} // The root JUnit element
   
       // Loop through each component in the report
       for _, component := range r.Components {
           // Create a Testsuite for each component
           suite := junit.Testsuite{
               Name: fmt.Sprintf("%s (%s)", component.Name, component.ContainerImage),
               // ... add properties like image URL, success status ...
           }
   
           // Add test cases for successes (if needed)
           mapResults(&suite, component.Successes, asTestCase)
   
           // Add test cases for violations, marking them as failures
           mapResults(&suite, component.Violations, func(res evaluator.Result) junit.Testcase {
               c := asTestCase(res) // Create basic test case info
               c.Failure = &junit.Result{ Message: res.Message } // Mark as failure
               return c
           })
   
           // Add test cases for warnings, marking them as skipped (common JUnit practice)
           mapResults(&suite, component.Warnings, func(res evaluator.Result) junit.Testcase {
               c := asTestCase(res) // Create basic test case info
               c.Skipped = &junit.Result{ Message: res.Message } // Mark as skipped
               return c
           })
   
           report.AddSuite(suite) // Add this component's suite to the main report
       }
       return report
   }
   
   // Helper to create a basic JUnit TestCase structure from an ec-cli Result
   func asTestCase(r evaluator.Result) junit.Testcase {
       // ... extract code and other metadata to build a descriptive name ...
       name := fmt.Sprintf("%s: %s [%s]", r.Metadata["code"], r.Message, /* other meta */)
       return junit.Testcase{ Name: name, Classname: name }
   }

   Explanation: The toJUnit function iterates through the Components in the Report. For each component, it creates a junit.Testsuite. Inside each suite, it converts ec-cli's Violations, Warnings, and Successes into junit.Testcase elements, marking violations as failures and warnings as skipped (a common convention in JUnit reporting). The final junit.Testsuites object is then marshalled into XML by the toFormat function shown earlier.

Conclusion

Congratulations! You've reached the end of the core ec-cli concepts. In this chapter, you learned about Output Reporting, the crucial final step where ec-cli presents the results of its validation checks. You saw how it acts like a court reporter, taking the raw findings and formatting them into various outputs like human-readable text, JSON, YAML, JUnit XML, and others using the --output flag. This flexibility allows results to be easily consumed by humans, CI/CD pipelines, and other tools.

You now have a solid understanding of the main components of ec-cli: defining what to check (Application Snapshot), specifying the rules (Policy Definition & Sources), running the checks (Validation Engine (Evaluator)), ensuring trust (Sigstore Verification), handling artifacts (Image & OCI Artifact Handling), tracking build components (Tekton Bundle Tracking), and presenting the results (Output Reporting).

With this knowledge, you're well-equipped to start using ec-cli to secure your software supply chain!

---

Generated by AI Codebase Knowledge Builder