Skip to content

b2_01_policy_definition___evaluation_

Jump to bottom
Simon Baird edited this page Apr 23, 2025 · 2 revisions

Chapter 1: Policy Definition & Evaluation

Welcome to the ec-cli tutorial! This is the first chapter where we'll explore the very heart of ec-cli: Policy Definition & Evaluation.

Imagine you're baking a cake for a competition. There's a rulebook that says what ingredients are allowed, how long it should bake, and maybe even how it should be decorated. You (the baker) need to follow these rules, and a judge will check your cake against the rulebook.

In the world of software security and compliance, ec-cli acts like that judge, and the policies are the rulebook.

What Problem Do Policies Solve?

Modern software development involves many steps and tools. How can we ensure that the final software (like a container image) meets our organization's security standards and compliance requirements?

  • Is it built from an approved base image?
  • Does it contain known critical security vulnerabilities?
  • Was it built by a trusted pipeline?
  • Does it have the necessary documentation (like SBOMs)?

Checking all this manually for every piece of software is slow and error-prone. We need an automated way to define the rules and check our software against them. This is where Policy Definition & Evaluation comes in.

Our Goal: To understand how ec-cli uses predefined rules (Policies) to automatically check if our software artifacts (like container images) meet specific security and compliance standards.

Key Concepts

Let's break down the core ideas:

  1. Policy Definition (The Rulebook):

    • A "policy" is a set of rules that define what's acceptable or required for your software.
    • These rules are often written in a special language called Rego, which comes from the Open Policy Agent (OPA) project. Think of Rego as the specific language the rulebook is written in. You don't need to be a Rego expert to use ec-cli, but it's good to know it's under the hood.
    • Policies aren't just floating in the air; they need to be stored somewhere. ec-cli can fetch these policy rules from various Policy Sources. Common sources include:
      • Git repositories (like GitHub or GitLab)
      • OCI registries (like Docker Hub or Quay.io, where policies can be stored as OCI artifacts)

  2. Artifact (The Cake / The Player):

    • This is the "thing" we want to check against the rules. In ec-cli, this is often a container image, but it could also be other files or data. We'll learn more about how ec-cli gathers information about artifacts in the Application Snapshot chapter.

  3. Policy Evaluation (The Judging):

    • This is the process where ec-cli takes the artifact, fetches the relevant policies (rules), and checks if the artifact complies with those rules.
    • ec-cli acts as the evaluator or "referee", applying the rulebook to the player.

How ec-cli Uses Policies: A Simple Example

Let's revisit our simple use case: Verify if a container image is built using an approved base image and has no critical vulnerabilities.

  1. Define the Policy: Somewhere (e.g., in a Git repository), there would be policy rules written in Rego. Conceptually, they might look like this (this is not exact Rego, just for illustration):

    • FAIL if image_base_image is not "registry.example.com/approved-base:latest"
    • FAIL if vulnerability_severity is "critical"
    • WARN if vulnerability_severity is "high"

  2. Fetch the Policy: When you run ec-cli, you tell it where to find these rules. You might use a command like ec fetch policy (though validation commands often fetch implicitly) to download them.

    # Example: Fetching policy rules from a specific source
# (You usually don't need to run this separately for validation)
ec fetch policy --source github.com/your-org/security-policies//path/to/rules

    This command tells ec-cli to go to that Git repository, find the rules in the specified path, and download them locally (often to a temporary directory).

  3. Evaluate the Artifact: You then use a command like ec validate image to perform the check.

    # Example: Validate an image against a defined policy configuration
ec validate image --image registry.example.com/my-app:v1.0 \
                  --policy your-org/standard-policy
    • --image: Specifies the container image (the artifact) we want to check.
    • --policy: Tells ec-cli which set of policies to use (this configuration might point to the Git repo we mentioned earlier, or an OCI artifact, or a Kubernetes resource).

  4. Get the Results: ec-cli runs the evaluation. It checks the image's base image and vulnerability scan results (we'll see how it gets this info in Application Snapshot) against the Rego rules it fetched. The output will tell you if the image passed or failed, and why. We'll cover this in detail in the Output Reporting chapter.

    • Success: All rules passed!
    • Failure: One or more critical rules failed (e.g., wrong base image, critical vulnerability found).
    • Warning: Some non-critical rules triggered warnings (e.g., high severity vulnerability found).

Under the Hood: How Evaluation Works

What happens internally when you run ec validate image?

High-Level Steps:

  1. Parse Input: ec-cli reads the command-line arguments (image reference, policy reference, etc.).
  2. Load Policy Configuration: It figures out where the policy rules and associated configuration live based on the --policy flag (Git, OCI, Kubernetes).
  3. Fetch Policy Code: It downloads the actual Rego policy code and any associated data files from the specified sources (using logic similar to ec fetch policy).
  4. Gather Artifact Data: It collects information about the specified image(s), such as metadata, attestations (like vulnerability scans or SBOMs), and signatures. This collected data is often structured as an Application Snapshot.
  5. Prepare Input for Evaluator: It bundles the gathered artifact data into a format (usually JSON) that the policy engine can understand.
  6. Evaluate: It feeds the policy code and the artifact data (as input) into an evaluation engine (like OPA or Conftest). The engine runs the Rego rules against the input.
  7. Process Results: It takes the raw results from the engine (passes, failures, warnings).
  8. Generate Report: It formats these results into a user-friendly report based on the requested output format (Output Reporting).

Simplified Sequence Diagram:

sequenceDiagram
    participant User
    participant CLI as ec validate image
    participant PolicyFetcher as Policy Source Fetcher
    participant Evaluator as OPA/Conftest Engine
    participant Reporter as Output Formatter

    User->>CLI: Run command with image & policy refs
    CLI->>PolicyFetcher: Request policy code for ref
    PolicyFetcher-->>CLI: Return policy code location
    CLI->>Evaluator: Prepare artifact data (Input)
    CLI->>Evaluator: Evaluate Input against Policy Code
    Evaluator-->>CLI: Raw evaluation results
    CLI->>Reporter: Process results
    Reporter-->>User: Display formatted report (Pass/Fail)
Loading

Diving into the Code (A Glimpse)

Let's look at some simplified code concepts.

1. Handling Validation Commands (cmd/validate/image.go, cmd/validate/input.go)

These files define the ec validate subcommands. They parse flags like --image, --policy, --strict and orchestrate the validation process.

// Simplified concept from cmd/validate/image.go
func validateImageCmd(validate imageValidationFunc) *cobra.Command {
    // ... (flag definitions for --image, --policy, etc.) ...
    cmd := &cobra.Command{
        Use:   "image",
        Short: "Validate conformance of container images...",
        PreRunE: func(cmd *cobra.Command, args []string) error {
            // 1. Determine the input (image refs)
            data.spec = applicationsnapshot.DetermineInputSpec(...)
            // 2. Load and pre-process the policy configuration
            data.policy, _, err = policy.PreProcessPolicy(ctx, policyOptions)
            // ... handle errors ...
            return nil
        },
        RunE: func(cmd *cobra.Command, args []string) error {
            // 3. Initialize evaluator(s) based on policy
            evaluators := // ... create evaluators ...
            // 4. Loop through images/components and validate each one
            for comp := range jobs {
                out, err := validate(ctx, comp, data.spec, data.policy, evaluators, ...)
                // ... collect results ...
            }
            // 5. Generate the final report
            report, _ := applicationsnapshot.NewReport(...)
            report.WriteAll(...) // Calls Output Reporting logic
            // 6. Handle --strict flag (exit code)
            // ...
            return nil
        },
    }
    // ...
    return cmd
}

This shows the orchestration: parse flags (PreRunE), run the core validation logic (RunE), which involves getting policy, getting evaluators, running validation per component, and reporting results.

2. Representing and Loading Policies (internal/policy/policy.go)

This package defines the Policy interface and handles loading the policy configuration (spec) from various sources (Kubernetes, files, JSON strings).

// Simplified concept from internal/policy/policy.go
type Policy interface {
	// Spec returns the loaded policy configuration details
	Spec() ecc.EnterpriseContractPolicySpec
	// CheckOpts provides options for signature verification (covered later)
	CheckOpts() (*cosign.CheckOpts, error)
	// ... other methods ...
}

// loadPolicy handles getting the policy spec from different places
func (p *policy) loadPolicy(ctx context.Context, policyRef string) error {
	if strings.Contains(policyRef, ":") { // Is it inline JSON/YAML or maybe git/oci?
        // Try parsing as EnterpriseContractPolicySpec YAML/JSON
        err := yaml.Unmarshal([]byte(policyRef), &p.EnterpriseContractPolicySpec)
        // ... handle errors and different formats ...
	} else {
        // Assume it's a Kubernetes reference [namespace/]name
        k8s, _ := kubernetes.NewClient(ctx)
		ecp, err := k8s.FetchEnterpriseContractPolicy(ctx, policyRef)
        // ... handle errors ...
		p.EnterpriseContractPolicySpec = ecp.Spec
	}
	return nil
}

This code figures out if the --policy value is raw configuration or a Kubernetes reference and loads the EnterpriseContractPolicySpec accordingly.

3. Fetching Policy Sources (internal/policy/source/source.go)

This handles the actual downloading of policy code (Rego files, data files) from Git, OCI, etc.

// Simplified concept from internal/policy/source/source.go
type PolicySource interface {
	// GetPolicy downloads the policy code to a local path (dest)
	GetPolicy(ctx context.Context, dest string, showMsg bool) (string, error)
	PolicyUrl() string // The source URL (e.g., git://..., oci://...)
	Type() PolicyType  // Is it policy code, data, config?
}

// PolicyUrl implements PolicySource for remote URLs
type PolicyUrl struct {
	Url  string
	Kind PolicyType
}

func (p *PolicyUrl) GetPolicy(ctx context.Context, workDir string, showMsg bool) (string, error) {
    // Use a downloader (like go-getter) to fetch p.Url
    // Place it in a unique destination within workDir
    dest := uniqueDestination(workDir, string(p.Kind), p.Url)
    // Download logic here, often cached
    metadata, err := downloader.Download(ctx, dest, p.Url, showMsg)
    // Update p.Url with pinned version (e.g., git sha, image digest) if possible
    p.Url, _ = metadata.GetPinnedURL(p.Url)
    return dest, err
}

This shows how different sources are handled via an interface, and how a specific URL type fetches the content.

4. The Evaluator Interface (internal/evaluator/evaluator.go)

This defines a standard way to interact with different policy evaluation engines.

// Simplified concept from internal/evaluator/evaluator.go
type Evaluator interface {
	// Evaluate runs the policies against the given input data
	Evaluate(ctx context.Context, target EvaluationTarget) ([]Outcome, error)
	// Destroy cleans up any resources used by the evaluator
	Destroy()
}

// EvaluationTarget holds the input data (artifact info) for the policy engine
type EvaluationTarget struct {
	Inputs []string // Paths to JSON files containing artifact data
	Target string   // Usually "main" or the specific policy target
}

// Outcome represents the results from a single evaluation run
type Outcome struct {
	FileName   string   // Policy file where results originated
	Namespace  string   // Policy namespace (e.g., lib.release)
	Successes  []Result // Rules that passed
	Failures   []Result // Rules that failed (violations)
	Warnings   []Result // Rules that generated warnings
	// ... Skipped, Exceptions ...
}

This interface allows ec-cli to potentially use different backends (like Conftest or a native OPA integration) for the actual Rego evaluation, just by calling the Evaluate method.

Conclusion

We've learned that Policy Definition & Evaluation is the core engine of ec-cli. It allows us to:

  1. Define security and compliance rules (policies), often using Rego.
  2. Fetch these rules from various sources like Git or OCI registries.
  3. Evaluate software artifacts (like container images) against these rules automatically.
  4. Get clear results (Pass/Fail/Warn) indicating compliance.

Think of it as the automated "judge" ensuring your software "cake" follows the "rulebook".

But how does the judge get the specific details about the cake to check against the rules? In the next chapter, we'll explore the Application Snapshot, which represents the collected data about the artifact being evaluated.

Generated by AI Codebase Knowledge Builder

Clone this wiki locally