Skip to content

b1_03_validation_engine__evaluator__

Jump to bottom
Simon Baird edited this page Apr 23, 2025 · 1 revision

Chapter 3: Validation Engine (Evaluator)

In the previous chapter, we learned about Policy Definition & Sources, which act as the "rulebook" for ec-cli. We know how to tell ec-cli which images to check using the Application Snapshot (the ingredients list) and what rules to use (Policy Definition & Sources - the recipe).

But who actually takes the recipe and checks if the ingredients meet the requirements? We need someone (or something!) to perform the actual inspection and deliver a verdict.

This is the job of the Validation Engine, also known as the Evaluator.

What Problem Does it Solve?

Imagine you have:

  1. The list of ingredients for your cake (the container images from the Application Snapshot).
  2. The recipe with strict instructions (the rules fetched via the Policy Definition & Sources), like "use organic eggs" or "no artificial sweeteners allowed".

You need a process to actually compare the ingredients against the recipe's rules and decide if the cake can be baked (or if the software can be released). The Validation Engine is that process. It's the core component that executes the rules against the data.

The Judge Analogy

Think of the Validation Engine like a judge in a courtroom:

  • The Rulebook: The judge receives the laws and regulations relevant to the case. This is like the Policy Rules (written in Rego) that ec-cli fetched based on your Policy Definition & Sources.
  • The Evidence: The judge examines the evidence presented. This is like the Input Data that ec-cli gathers about your container images (e.g., image details, security scan results found in attestations, configuration files). This data often comes from processing the Application Snapshot and related information.
  • The Verdict: The judge applies the rulebook to the evidence and delivers a verdict – guilty, not guilty, maybe with some warnings. This is like the Results the Validation Engine produces:
    • Successes: Rules that passed.
    • Failures/Violations: Rules that failed (the "guilty" verdict).
    • Warnings: Rules that passed but raised concerns.
    • Exceptions: Errors encountered during evaluation.

How it Works: Using Conftest (Usually!)

Under the hood, ec-cli primarily uses a powerful open-source tool called Conftest to act as its Validation Engine. Conftest specializes in testing configuration data using Rego policies. ec-cli might also be configured to use Open Policy Agent (OPA) directly, which is the foundation upon which Conftest is built.

You don't need to be an expert in Conftest or OPA to use ec-cli. The important part is understanding the role of the engine: it takes the Rego rules and the JSON data representing your images/attestations and runs the evaluation.

Inputs and Outputs

  • Input 1: Policy Rules: These are the .rego files fetched from the sources defined in your Policy Definition & Sources.
  • Input 2: Input Data: This is a structured data format (usually JSON) prepared by ec-cli. It contains details about the specific Application Snapshot component being checked, including its attestations, configuration, and other relevant metadata gathered during checks like Sigstore Verification.
  • Output: Results: A structured list of outcomes (successes, failures, warnings, exceptions), which are then used for Output Reporting.

How ec-cli Uses the Engine

You typically don't interact with the Validation Engine directly. Instead, commands like ec validate image orchestrate the process:

  1. Parse the Application Snapshot (--images).
  2. Parse the Policy Definition & Sources (--policy).
  3. Fetch the required Policy Rules (Rego) and Data files.
  4. For each image in the snapshot:
    • Gather relevant data (image config, attestations, etc.). Perform checks like Sigstore Verification.
    • Format this data into a JSON input document.
    • Invoke the Validation Engine, passing it the fetched rules and the input JSON.
    • Collect the results (Successes, Failures, Warnings) from the engine.
  5. Format and display the final report (Output Reporting).

The engine is the heart of step 4c.

Under the Hood: The Evaluation Flow

Let's visualize how the engine fits into the ec validate image command flow:

sequenceDiagram
    participant User
    participant ec_cli as ec validate image
    participant InputHandler as Input Data Handler
    participant PolicyHandler as Policy Rule Handler
    participant Engine as Validation Engine (e.g., Conftest)
    participant Reporter

    User->>ec_cli: Runs `ec validate image --images ... --policy ...`
    Note over ec_cli: Start processing...
    ec_cli->>InputHandler: Prepare input data for image (from Snapshot, Attestations etc.)
    InputHandler-->>ec_cli: Input data (JSON format)
    ec_cli->>PolicyHandler: Get policy rules (Rego files from sources)
    PolicyHandler-->>ec_cli: Policy rules (local paths)
    Note over ec_cli: Have Rules and Input Data ready
    ec_cli->>Engine: Evaluate(rules_paths, input_data)
    Engine->>Engine: Executes Rego rules against input JSON
    Engine-->>ec_cli: Returns structured results (Successes, Failures, Warnings)
    Note over ec_cli: Received evaluation verdict
    ec_cli->>Reporter: Format and display results
    Reporter-->>User: Show final report
Loading

The key interaction is ec_cli calling Engine.Evaluate with the rules and the data, and getting back the structured results.

Code Snippets

Let's look at some simplified code concepts:

  1. The Evaluator Interface: ec-cli defines a standard way to talk to any validation engine through an Evaluator interface. This makes it possible to potentially swap out Conftest for OPA or another engine in the future.

    File: internal/evaluator/evaluator.go

    package evaluator

// Represents the data needed for evaluation
type EvaluationTarget struct {
    Inputs []string // Paths to input JSON files
    Target string   // Identifier for what's being evaluated (e.g., image ref)
}

// The interface that all validation engines must implement
type Evaluator interface {
    // The core function: runs rules against inputs
    Evaluate(ctx context.Context, target EvaluationTarget) ([]Outcome, error)
    // Cleanup resources used by the evaluator
    Destroy()
    // ... other potential methods ...
}

    Explanation: This defines a contract. Any "engine" must provide an Evaluate function that takes the input data location (EvaluationTarget) and returns a list of Outcome objects.

  2. The Outcome Structure: The Evaluate function returns results structured like this:

    File: internal/evaluator/evaluator.go

    // Represents the results from evaluating one set of rules/inputs
type Outcome struct {
    FileName   string   `json:"filename"` // Which input file this relates to
    Namespace  string   `json:"namespace"` // Which Rego package generated results
    Successes  []Result `json:"successes,omitempty"` // Rules that passed
    Skipped    []Result `json:"skipped,omitempty"`   // Rules that were skipped
    Warnings   []Result `json:"warnings,omitempty"`  // Rules that generated warnings
    Failures   []Result `json:"failures,omitempty"`  // Rules that failed (Violations)
    Exceptions []Result `json:"exceptions,omitempty"`// Errors during evaluation
}

// Represents a single result (Success, Failure, etc.)
type Result struct {
    Message  string                 `json:"msg"` // Human-readable message
    Metadata map[string]interface{} `json:"metadata,omitempty"` // Extra info (rule code, etc.)
}

    Explanation: This structure neatly organizes the verdict from the engine, separating successes, failures, and warnings, each with a message and extra details.

  3. Calling the Evaluator: Inside the ec validate image logic, after preparing inputs and getting policies, ec-cli calls the engine's Evaluate method.

    File: internal/image/validate.go (Simplified logic)

    // Inside the ValidateImage function...
func ValidateImage(ctx context.Context, /* ... other args ... */, evaluators []evaluator.Evaluator) (*output.Output, error) {
    // ... prepare input data (attestations, image config) into inputPath ...
    inputPath, inputJSON, err := a.WriteInputFile(ctx)
    // ... handle error ...

    var allResults []evaluator.Outcome // To collect results from all evaluators

    // Loop through each configured evaluator (usually just Conftest)
    for _, e := range evaluators {
        // Define what the engine should evaluate
        target := evaluator.EvaluationTarget{Inputs: []string{inputPath}}
        // Potentially add the image reference as the target identifier
        target.Target = a.ImageReference(ctx) // Get the image ref string

        // *** THE CORE CALL TO THE ENGINE ***
        results, err := e.Evaluate(ctx, target)
        // ... handle evaluation error ...

        log.Debug("Policy check complete for one evaluator")
        allResults = append(allResults, results...)
    }

    // Store the collected results in the final output object
    out.SetPolicyCheck(allResults) // This prepares results for reporting

    return out, nil
}

    Explanation: This shows the main loop where ec-cli iterates through its configured evaluators (usually just one, like the Conftest evaluator). It creates the EvaluationTarget (telling the engine where the input JSON is) and calls e.Evaluate. The returned results (a slice of Outcome structs) are collected and eventually passed to the Output Reporting system.

Conclusion

You've learned about the Validation Engine (Evaluator), the crucial component in ec-cli that acts like a judge. It takes the "rulebook" (Policy Rules) and the "evidence" (Input Data prepared from the Application Snapshot and attestations) and delivers a "verdict" (Successes, Failures, Warnings). We saw that ec-cli typically uses Conftest for this task and how the ec validate image command orchestrates calling the engine.

The input data given to the engine often includes results from cryptographic verification, like checking digital signatures on attestations. How does ec-cli ensure this data is authentic and hasn't been tampered with? That's where our next topic comes in.

Let's move on to learn about ensuring trust in the evidence: Sigstore Verification.

Generated by AI Codebase Knowledge Builder

Clone this wiki locally