Skip to content

b2_08_acceptance_testing_framework_

Jump to bottom
Simon Baird edited this page Apr 23, 2025 · 2 revisions

Chapter 8: Acceptance Testing Framework

In the previous chapter, Chapter 7: Bundle Tracking, we learned how ec-cli helps manage the versions of reusable Tekton Task bundles. We've now covered many core features of ec-cli. But how do we ensure all these features work together correctly, not just individually, but as a whole system, especially before a new version is released?

This is where the Acceptance Testing Framework comes into play. Think of it as the final, comprehensive quality check for ec-cli.

What Problem Does This Solve?

Imagine ec-cli is a complex machine with many moving parts (Policy Evaluation, OCI Interaction, Sigstore Integration, etc.). We've tested some individual parts, but how do we know the entire machine works smoothly end-to-end? What if fixing a bug in one part accidentally breaks another?

We need a way to automatically run ec-cli through realistic scenarios that mimic how users would actually use it, ensuring all its features work together as expected. This prevents regressions (old bugs reappearing) and confirms that new features meet the requirements.

It's like designing a challenging obstacle course for ec-cli. Before we release a new version, we put it through this course. If it completes the course successfully, we have high confidence it's ready for users.

Our Goal: To understand how ec-cli uses its acceptance testing framework to ensure the tool functions correctly end-to-end in simulated real-world scenarios.

Key Concepts: Building the Obstacle Course

  1. Cucumber / Gherkin (The Course Map):

    • Gherkin is a simple, human-readable language used to describe the test scenarios (the obstacles). It uses keywords like Given, When, Then. Example: 
      Scenario: Validate a signed image
  Given a stub registry running
  And an image named "my-app"
  And a valid image signature of "my-app" image signed by the "prod-key" key
  When ec command is run with "validate image --image registry/my-app --policy my-policy --public-key prod-key.pub"
  Then the exit status should be 0
    • Cucumber is the tool that reads these Gherkin files and executes the corresponding test code.

  2. GoDog (The Course Builder - Go Version):

    • Since ec-cli is written in Go, we use GoDog, a Go implementation of Cucumber.
    • It connects the plain-English Gherkin steps (like Given an image named "my-app") to actual Go functions that perform the actions (like creating a test image and pushing it to a mock registry).

  3. Test Environment (The Controlled Course Area):

    • Running tests against real, live systems (like public registries or Git servers) can be slow, unreliable, and might have side effects.
    • The acceptance tests set up a temporary, isolated environment using tools like:
      • Testcontainers: Automatically starts and manages temporary Docker containers for services needed during the test (e.g., a fake container registry, a fake Git server, a fake Rekor instance).
      • WireMock: A tool used to create fake ("stubbed") HTTP APIs, often used to simulate Kubernetes API servers or other web services ec-cli might interact with.
    • These "mocks" or "stubs" behave like the real services but are completely controlled by the test, ensuring consistent and fast results.

  4. Running ec-cli (The Athlete):

    • Crucially, the acceptance tests compile the actual ec-cli binary from the current code and run it as a command-line process, just like a user would.
    • The tests then check its output (stdout, stderr) and exit code against the expectations defined in the Gherkin scenario.

How an Acceptance Test Works

Let's trace the simple Gherkin scenario from above:

Scenario: Validate a signed image
  Given a stub registry running
  And an image named "my-app"
  And a valid image signature of "my-app" image signed by the "prod-key" key
  When ec command is run with "validate image --image registry/my-app --policy my-policy --public-key prod-key.pub"
  Then the exit status should be 0
  1. Test Start: You run the acceptance tests (e.g., using make acceptance).
  2. GoDog Reads Feature: GoDog finds the .feature file containing this scenario.
  3. Setup Steps (Given):
    • Given a stub registry running: GoDog finds the Go function matching this step. That function uses Testcontainers to start a temporary container registry.
    • And an image named "my-app": The corresponding Go function creates a simple test container image.
    • And a valid image signature...: Another Go function signs this test image using a test key ("prod-key") and pushes both the image and its signature to the stub registry started earlier.
  4. Action Step (When):
    • When ec command is run with "...": The Go function for this step:
      • Constructs the full ec validate image ... command string.
      • Uses Go's os/exec package to run the compiled ec-cli binary as a separate process with that command line.
      • It captures the standard output, standard error, and exit code of the ec-cli process.
  5. Assertion Step (Then):
    • Then the exit status should be 0: The Go function for this step checks the captured exit code from the ec-cli process. If it's 0, the step passes. If not, the step fails, and the test scenario fails.
  6. Cleanup: After the scenario (or the whole test suite) finishes, GoDog (often via Testcontainers) automatically stops and removes the temporary containers (registry, etc.), unless specifically told to persist them for debugging.

Under the Hood: Orchestration

High-Level Steps:

  1. Test Runner Invocation: The Go test runner (go test) is invoked, typically via make acceptance.
  2. GoDog Initialization: The acceptance_test.go file sets up GoDog, telling it where to find .feature files and registering the Go functions that implement the Gherkin steps.
  3. Feature File Parsing: GoDog parses the Gherkin .feature files.
  4. Scenario Execution Loop: For each scenario:
    • Context Setup: A new Go context.Context is created to hold state for this specific scenario.
    • Step Execution: GoDog executes the Go function corresponding to each Gherkin step (Given, When, Then).
      • Setup steps often interact with testenv helpers to start containers (registry, Rekor, Git) or WireMock stubs.
      • The When ec command is run... step uses cli.ecCommandIsRunWith to execute the real ec-cli binary.
      • Assertion steps (Then) check the results stored in the context (exit code, stdout, stderr).
    • Cleanup: After the scenario, cleanup functions (registered via testenv.Testing(ctx).Cleanup(...)) are run to tear down resources, unless -persist flag is used.
  5. Reporting: GoDog reports the results (pass/fail) for each scenario and step.

Simplified Sequence Diagram:

sequenceDiagram
    participant User
    participant Make as make acceptance
    participant GoTest as go test ./acceptance
    participant GoDog as GoDog Runner
    participant StepImpl as Step Implementation (Go Funcs)
    participant TestEnv as Test Environment (Containers/Mocks)
    participant EcCLI as Compiled ec-cli binary

    User->>Make: Run tests
    Make->>GoTest: Compile ec-cli & Run tests
    GoTest->>GoDog: Initialize & Find features
    GoDog->>StepImpl: Execute 'Given a stub registry running'
    StepImpl->>TestEnv: Start registry container
    TestEnv-->>StepImpl: Registry running
    GoDog->>StepImpl: Execute 'When ec command is run...'
    StepImpl->>EcCLI: Run 'ec validate image...' command
    EcCLI->>TestEnv: Interact with mock registry/Rekor
    TestEnv-->>EcCLI: Mocked responses
    EcCLI-->>StepImpl: Exit Code, Stdout, Stderr
    GoDog->>StepImpl: Execute 'Then the exit status should be 0'
    StepImpl-->>GoDog: Pass/Fail based on exit code
    GoDog-->>GoTest: Report results
    GoTest-->>Make: Overall test status
    Make-->>User: Display results
Loading

Diving into the Code (A Glimpse)

Let's look at some key files involved.

1. Test Entry Point (acceptance/acceptance_test.go)

This file configures and runs the GoDog test suite using the standard Go testing framework.

// Simplified concept from acceptance/acceptance_test.go
package acceptance

import (
	"testing"
	"github.com/cucumber/godog"
	// ... other imports for step packages ...
	"github.com/enterprise-contract/ec-cli/acceptance/cli"
	"github.com/enterprise-contract/ec-cli/acceptance/testenv"
)

// TestMain sets up command-line flags for the tests (like -persist)
func TestMain(m *testing.M) {
	testenv.Setup() // Parses flags like -persist, -tags
	os.Exit(m.Run())
}

// InitializeScenario wires up the Gherkin steps to Go functions.
func InitializeScenario(sc *godog.ScenarioContext) {
	// Register steps from different packages
	cli.AddStepsTo(sc) // Steps for running 'ec' command
	registry.AddStepsTo(sc) // Steps for managing mock registry
	git.AddStepsTo(sc) // Steps for managing mock git server
	rekor.AddStepsTo(sc) // Steps for managing mock rekor
	kubernetes.AddStepsTo(sc) // Steps for managing mock k8s
	// ... other step packages ...

	// Register common setup/cleanup hooks
	testenv.AddStepsTo(sc)
}

// TestAcceptance is the actual test function Go discovers.
func TestAcceptance(t *testing.T) {
	suite := godog.TestSuite{
		Name:                "acceptance",
		ScenarioInitializer: InitializeScenario, // Connects steps
		Options: &godog.Options{
			Format:   "pretty", // Output format
			Paths:    []string{"../features"}, // Where to find .feature files
			Tags:     testenv.Tags(), // Filter scenarios by tags (e.g., @wip)
			Strict:   true,
			// ... other options like concurrency ...
		},
	}

	// Run the test suite
	status := suite.Run()
	if status != 0 {
		t.Fail()
	}
}

This code shows how:

  • TestMain handles test flags.
  • InitializeScenario links Gherkin steps defined in various packages (cli, registry, git, etc.) to their Go implementations.
  • TestAcceptance configures and runs the godog.TestSuite.

2. Gherkin Feature File (features/validate_image.feature)

This is where the human-readable scenarios are defined.

# Simplified from features/validate_image.feature
Feature: evaluate enterprise contract
  The ec command line should evaluate enterprise contract

  Background: # Runs before each scenario in this file
    Given a stub cluster running
    Given stub rekord running
    Given stub registry running
    Given stub git daemon running
    Given stub tuf running

  Scenario: happy day
    Given a key pair named "known"
    Given an image named "acceptance/ec-happy-day"
    Given a valid image signature of "acceptance/ec-happy-day" image signed by the "known" key
    # ... other Given steps ...
    When ec command is run with "validate image --image ${REGISTRY}/acceptance/ec-happy-day --policy ... --public-key ${known_PUBLIC_KEY} ..."
    Then the exit status should be 0
    And the output should match the snapshot # Compares output to a stored baseline

This shows the Gherkin syntax with Given, When, Then steps. The Background section sets up common prerequisites. Note the use of ${VAR} for dynamic values substituted by the test code.

3. CLI Step Implementation (acceptance/cli/cli.go)

This package contains the Go functions that implement steps related to running the ec command.

// Simplified concept from acceptance/cli/cli.go
package cli

import (
	"context"
	"os/exec"
	"path"
	"runtime"
	"strings"
	// ... other imports ...
	"github.com/cucumber/godog"
	"github.com/enterprise-contract/ec-cli/acceptance/testenv"
)

// Holds the result of running the 'ec' command
type status struct {
	*exec.Cmd
	vars   map[string]string // Variables substituted in the command
	err    error             // Error from cmd.Run()
	stdout *bytes.Buffer     // Captured standard output
	stderr *bytes.Buffer     // Captured standard error
}

const processStatusKey key = iota // Key to store 'status' in context

// ecCommandIsRunWith implements the "When ec command is run with..." step
func ecCommandIsRunWith(ctx context.Context, parameters string) (context.Context, error) {
	// Determine path to the compiled 'ec' binary
	ecBinaryPath := path.Join("..", "dist", fmt.Sprintf("ec_%s_%s", runtime.GOOS, runtime.GOARCH))
	// ... check if binary exists ...

	// Get environment variables and substitution vars for this scenario
	ctx, environment, vars, err := variables(ctx) // Helper to setup env (e.g., KUBECONFIG, registry URL)
	if err != nil { return ctx, err }

	// Substitute ${VAR} placeholders in the command parameters
	argsString := os.Expand(parameters, func(key string) string { return vars[key] })
	args := strings.Split(argsString, " ")

	// Prepare the command
	cmd := exec.CommandContext(ctx, ecBinaryPath, args...)
	cmd.Env = environment // Set environment variables

	// Capture output
	var stdout, stderr bytes.Buffer
	cmd.Stdout = &stdout
	cmd.Stderr = &stderr

	// Run the command
	runErr := cmd.Run() // We store the error, don't fail the step here

	// Store the results in the context for 'Then' steps to check
	sts := status{Cmd: cmd, vars: vars, err: runErr, stdout: &stdout, stderr: &stderr}
	ctx = context.WithValue(ctx, processStatusKey, &sts)

	return ctx, nil // Step succeeds even if ec command failed; 'Then' step asserts outcome
}

// theExitStatusIs implements the "Then the exit status should be..." step
func theExitStatusIs(ctx context.Context, expected int) error {
	status, ok := ctx.Value(processStatusKey).(*status)
	if !ok { return errors.New("ec command status not found in context") }

	// Check for errors *running* the command (not application errors)
	var exitErr *exec.ExitError
	if status.err != nil && !errors.As(status.err, &exitErr) {
		return fmt.Errorf("failed to invoke the ec command: %w", status.err)
	}

	// Get the actual exit code
	actualExitCode := 0
	if exitErr != nil {
		actualExitCode = exitErr.ExitCode() // Get code from ExitError if command failed
	} else if status.err == nil {
		actualExitCode = 0 // Success
	} else {
		// Should not happen if error handling above is correct
		return fmt.Errorf("unexpected error state: %v", status.err)
	}


	// Compare actual vs expected
	if actualExitCode != expected {
		// Log detailed output on failure (handled by After hook)
		return fmt.Errorf("ec exited with code %d, expected %d", actualExitCode, expected)
	}

	return nil // Success!
}

// AddStepsTo registers the step functions with GoDog
func AddStepsTo(sc *godog.ScenarioContext) {
	sc.Step(`^ec command is run with "(.+)"$`, ecCommandIsRunWith)
	sc.Step(`^the exit status should be (\d+)$`, theExitStatusIs)
	// ... other steps like "the standard output should contain" ...
	sc.After(func(ctx context.Context, sc *godog.Scenario, err error) (context.Context, error) {
		if err != nil {
			logExecution(ctx) // Log details if a step failed
		}
		return ctx, nil
	})
}

This shows:

  • The status struct holding the outcome of running ec.
  • The ecCommandIsRunWith function that prepares and runs the ec binary, captures its output, and stores the status in the context.
  • The theExitStatusIs function that retrieves the status from the context and compares the actual exit code to the expected one.
  • The AddStepsTo function registers these implementations.

4. Test Environment Setup (acceptance/testenv/testenv.go)

This package provides helpers for managing the test environment state and resources.

// Simplified concept from acceptance/testenv/testenv.go
package testenv

import (
	"context"
	"flag"
	"os"
	"sync"
	"testing"

	"github.com/cucumber/godog"
	"github.com/google/uuid"
	"github.com/testcontainers/testcontainers-go"
)

// Global test flags
var persist bool // Keep environment after test?
var tags string    // Filter scenarios by tags

// TestState holds scenario-specific data
type TestState struct {
	testing.TB            // Go test runner interface
	id          uuid.UUID // Unique ID for this scenario run
	tempDir     string
	state       map[any]any // Holds state for different parts (registry, git, etc.)
	stateMu     sync.RWMutex
	cleanupFuncs []func()    // Functions to run after scenario
}

// Setup parses command-line flags
func Setup() {
	flag.BoolVar(&persist, "persist", false, "Persist environment after tests")
	flag.StringVar(&tags, "tags", "", "Filter scenarios by tags")
	// ... parse other flags ...
}

// AddStepsTo registers Before/After scenario hooks
func AddStepsTo(sc *godog.ScenarioContext) {
	sc.Before(func(ctx context.Context, sc *godog.Scenario) (context.Context, error) {
		// Create a new TestState for this scenario
		t := &TestState{
			TB:    &testing.T{}, // Dummy test runner for cleanup
			id:    uuid.New(),
			state: make(map[any]any),
		}
		// Create a temp dir for this scenario run
		t.tempDir, _ = os.MkdirTemp("", "ec-acceptance-"+t.id.String()+"-*")
		// Store TestState in context
		return context.WithValue(ctx, testStateKey, t), nil
	})

	sc.After(func(ctx context.Context, sc *godog.Scenario, err error) (context.Context, error) {
		// Run cleanup functions unless -persist is set
		if !persist {
			t := fetchTestState(ctx)
			for _, cleanup := range t.cleanupFuncs {
				cleanup() // Stop containers, remove temp files etc.
			}
			if t.tempDir != "" {
				os.RemoveAll(t.tempDir)
			}
		}
		return ctx, nil
	})
}

// SetupState retrieves or initializes state for a specific component (like registry)
func SetupState[T State](ctx context.Context, state **T) (context.Context, error) {
	t := fetchTestState(ctx)
	t.stateMu.RLock()
	// Check if state already exists for this type
	var key any = (*T)(nil) // Use nil pointer of type T as key
	existing, found := t.state[key]
	t.stateMu.RUnlock()

	if found {
		*state = existing.(T) // Return existing state
		return ctx, nil
	}

	// State not found, initialize it
	t.stateMu.Lock()
	defer t.stateMu.Unlock()
	// Double check in case another goroutine initialized it while waiting for lock
	if existing, found = t.state[key]; found {
		*state = existing.(T)
		return ctx, nil
	}

	// Create new state and call its Initialize method if it exists
	newState := new(T)
	if initializer, ok := any(newState).(Initializer); ok {
		initializer.Initialize()
	}
	t.state[key] = newState // Store the new state
	*state = newState
	return ctx, nil
}

// FetchState retrieves the state for a specific component
func FetchState[T State](ctx context.Context) T {
	// ... similar logic to SetupState but returns existing state directly ...
}

// TestContainersRequest wraps the container request to honor the -persist flag
func TestContainersRequest(ctx context.Context, req testcontainers.ContainerRequest) testcontainers.ContainerRequest {
	if !persist {
		// Add auto-removal label if not persisting
		req.Labels = map[string]string{testcontainers.LabelAutoRemove: "true"}
		req.Labels[testcontainers.LabelSessionID] = fetchTestState(ctx).id.String()
	}
	// ... maybe add common network settings etc. ...
	return req
}

This shows simplified helpers for:

  • Managing test flags (persist, tags).
  • Setting up and cleaning up per-scenario state using Before and After hooks.
  • Storing and retrieving component-specific state (like registry URL or git repo path) in the context using generics (SetupState, FetchState).
  • Configuring Testcontainers to automatically clean up containers unless -persist is used.

Conclusion

The Acceptance Testing Framework is the guardian of ec-cli's quality and reliability. By using:

  1. Gherkin/Cucumber: To define tests in plain English.
  2. GoDog: To execute these tests using Go code.
  3. Testcontainers/WireMock: To create isolated, predictable test environments with mock services.
  4. Actual ec-cli Execution: To test the real binary end-to-end.

This framework provides a powerful way to simulate real-world usage, catch regressions, and ensure that all the components we've learned about (Policy Definition & Evaluation, Application Snapshot, OCI Interaction, Sigstore Integration, etc.) work together seamlessly.

Tutorial Conclusion

Congratulations! You've reached the end of the ec-cli tutorial. We've journeyed through the core concepts, from defining security policies and understanding application snapshots to interacting with registries, verifying signatures with Sigstore, integrating with Tekton, tracking bundles, and finally, seeing how the tool ensures its own quality through acceptance testing.

Hopefully, you now have a solid foundation for using ec-cli to enforce security and compliance standards for your software artifacts. Remember that the concepts build upon each other, forming a comprehensive system for supply chain security.

Feel free to revisit these chapters as needed, explore the ec-cli command-line help (ec --help), and dive into the official documentation for more advanced topics and configuration options. Happy validating!

Generated by AI Codebase Knowledge Builder

Clone this wiki locally