Skip to content

Fixup e2e - #118

Merged
JasonPowr merged 2 commits into
mainfrom
fixup-e2e
Dec 4, 2025
Merged

Fixup e2e#118
JasonPowr merged 2 commits into
mainfrom
fixup-e2e

Conversation

@JasonPowr

@JasonPowr JasonPowr commented Dec 4, 2025

Copy link
Copy Markdown
Member

PR Type

Tests, Enhancement


Description

  • Add Serial flag to e2e test suites for sequential execution

  • Implement proper wait function for policy controller resource cleanup

  • Add retry logic with polling for pod admission test cases

  • Extract image repository prefix for cluster image policy templates

  • Update policy controller image version in helm values


Diagram Walkthrough

flowchart LR
  A["E2E Test Files"] -->|Add Serial flag| B["Sequential Test Execution"]
  A -->|Add retry logic| C["Pod Admission Tests"]
  D["Utils"] -->|New ImageRepoPrefix function| E["Template Rendering"]
  D -->|New WaitForPolicyControllerResourcesDeleted| F["Resource Cleanup"]
  G["Helm Values"] -->|Update image version| H["Policy Controller Deployment"]
Loading

File Walkthrough

Relevant files
Tests
byok_install_test.go
Add serial execution and retry logic to BYOK tests             

test/e2e/byok_install_test.go

  • Add Serial flag to test suite for sequential execution
  • Add WaitForPolicyControllerResourcesDeleted call in cleanup
  • Add TEST_IMAGE_PREFIX parameter to template rendering
  • Wrap pod creation in Eventually with retry logic and polling
  • Import time and apierrors packages
+21/-11 
common_install_test.go
Add serial execution and retry logic to common tests         

test/e2e/common_install_test.go

  • Add Serial flag to test suite for sequential execution
  • Add WaitForPolicyControllerResourcesDeleted call in cleanup
  • Add TEST_IMAGE_PREFIX parameter to template rendering
  • Wrap pod creation in Eventually with retry logic and polling
  • Import time and apierrors packages
+21/-11 
serialized_tuf_root_install_test.go
Add serial execution and retry logic to TUF tests               

test/e2e/serialized_tuf_root_install_test.go

  • Add Serial flag to test suite for sequential execution
  • Add WaitForPolicyControllerResourcesDeleted call in cleanup
  • Add TEST_IMAGE_PREFIX parameter to template rendering
  • Wrap pod creation in Eventually with retry logic and polling
  • Import time and apierrors packages
+21/-11 
Enhancement
image.go
Add image repository prefix extraction utility                     

test/e2e/utils/image.go

  • Add ImageRepoPrefix function to extract image repository prefix
  • Function handles image names with tags or digests by finding first @
    or : character
  • Fix indentation formatting in imports section
  • Import strings package
+11/-3   
kubernetes.go
Add comprehensive resource cleanup wait function                 

test/e2e/utils/kubernetes.go

  • Add WaitForPolicyControllerResourcesDeleted function for comprehensive
    resource cleanup
  • Function waits for deletion of 12 policy controller resources with
    exponential backoff
  • Handles both cluster-scoped and namespace-scoped resources
  • Reorder imports for better organization
  • Add admissionregistrationv1 import
+57/-4   
Configuration changes
common_cluster_image_policy.yaml.tpl
Update cluster image policy glob pattern template               

test/e2e/custom_resources/cluster_image_policies/common_cluster_image_policy.yaml.tpl

  • Update glob pattern to use TEST_IMAGE_PREFIX template variable
  • Change from wildcard ** to {{ .TEST_IMAGE_PREFIX }}** for more
    specific image matching
+1/-1     
Dependencies
values.yaml
Update policy controller image version                                     

helm-charts/policy-controller-operator/values.yaml

  • Update policy controller webhook image version hash
  • Change from
    sha256:ea4ac2b005571bc28270f8420dff30a0f02381bcd78153e1014f9006d1a17609
    to
    sha256:8b9bd18603f4cdba8b1243c3285e716b31e00d238e0566ba101ffe5f5678739a
+1/-1     

@sourcery-ai

sourcery-ai Bot commented Dec 4, 2025

Copy link
Copy Markdown

Reviewer's Guide

This PR hardens Kubernetes e2e tests for the policy-controller-operator by adding robust cleanup of controller resources, making the installation suites serial, retrying pod admission, and tightening image policy configuration using a computed image repository prefix while also updating the helm chart webhook image digest.

Sequence diagram for hardened e2e installation and cleanup flow

sequenceDiagram
    participant E2E as E2ETestSuite
    participant K8s as KubernetesCluster
    participant Op as PolicyControllerOperator
    participant WH as PolicyControllerWebhook

    E2E->>K8s: Apply operator Helm chart
    K8s-->>Op: Create operator deployment
    Op-->>K8s: Create webhook deployment
    K8s-->>WH: Start webhook pods

    E2E->>K8s: Create test namespace and resources
    E2E->>K8s: Create test pod

    loop PodAdmissionRetry
        K8s->>WH: AdmissionReview for test pod
        WH-->>K8s: Validate pod image against ClusterImagePolicy
        alt AdmissionDeniedOrWebhookNotReady
            E2E->>K8s: Wait and retry getting pod status
        else AdmissionAllowed
            K8s-->>E2E: Pod becomes Running (loop ends)
        end
    end

    E2E->>K8s: Run verification assertions

    par RobustCleanup
        E2E->>K8s: Delete test pod and namespace
        E2E->>K8s: Delete test ClusterImagePolicy and other CRs
        E2E->>K8s: Delete operator resources
    and
        K8s-->>WH: Terminate webhook pods
        K8s-->>Op: Terminate operator deployment
    end

    K8s-->>E2E: All resources removed
Loading

File-Level Changes

Change Details Files
Add a reusable helper to wait for deletion of all policy controller Kubernetes resources used in e2e tests.
  • Introduce WaitForPolicyControllerResourcesDeleted that iterates over deployment, webhooks, services, secrets, and configmaps, deleting them and polling with exponential backoff until they are gone
  • Use DeepCopyObject to construct per-iteration client.Objects and handle NotFound vs other API errors properly in get/delete calls
  • Add required imports for admissionregistration, apps, core, and metav1 APIs in the Kubernetes e2e utils
test/e2e/utils/kubernetes.go
Ensure each installation e2e suite runs serially, fully cleans up controller resources, and tolerates pod creation races by retrying until admission is allowed.
  • Mark BYOK, common installation, and serialized TUF root Describe blocks as Serial to avoid parallel interference between suites
  • Extend AfterAll cleanup in each suite to call WaitForPolicyControllerResourcesDeleted after deleting custom resources and the PolicyController instance
  • Wrap CreateTestPod calls in Ginkgo Eventually with context and 5s polling, treating AlreadyExists as success and failing if pod admission never becomes allowed
  • Add time and apierrors imports needed for polling and error handling in tests
test/e2e/byok_install_test.go
test/e2e/common_install_test.go
test/e2e/serialized_tuf_root_install_test.go
Refine cluster image policy templates to match only the test image repository prefix and plumb that value from tests via a helper.
  • Add ImageRepoPrefix helper that strips any tag or digest from an image reference to derive the repository prefix
  • Pass TEST_IMAGE_PREFIX (computed via ImageRepoPrefix) into BYOK, common, and serialized TUF ClusterImagePolicy templates when rendering
  • Change common_cluster_image_policy.yaml.tpl to use TEST_IMAGE_PREFIX in the glob instead of the previous catch‑all "**" pattern
test/e2e/utils/image.go
test/e2e/byok_install_test.go
test/e2e/common_install_test.go
test/e2e/serialized_tuf_root_install_test.go
test/e2e/custom_resources/cluster_image_policies/common_cluster_image_policy.yaml.tpl
Update the policy-controller-operator webhook container image digest used by the helm chart.
  • Bump policy-controller webhook.image.version digest to a new sha256 value in values.yaml
helm-charts/policy-controller-operator/values.yaml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@qodo-code-review

Copy link
Copy Markdown

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢
No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
🎫 No ticket provided
  • Create ticket/issue
Codebase Duplication Compliance
Codebase context is not defined

Follow the guide to enable codebase context checks.

Custom Compliance
🟢
Generic: Meaningful Naming and Self-Documenting Code

Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting

Status: Passed

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Robust Error Handling and Edge Case Management

Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation

Status: Passed

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Secure Error Handling

Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.

Status: Passed

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Comprehensive Audit Trails

Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.

Status:
Missing Audits: The added cleanup function and test logic perform critical actions (creating/deleting
Kubernetes resources) without adding any audit logging, which may be acceptable for tests
but cannot be confirmed from the diff.

Referred Code
func WaitForPolicyControllerResourcesDeleted(ctx context.Context, k8sClient client.Client) error {
	resources := []struct {
		name      string
		namespace string
		obj       client.Object
	}{
		{DeploymentName, InstallNamespace, &appsv1.Deployment{}},
		{ValidatingWebhookName, "", &admissionregistrationv1.ValidatingWebhookConfiguration{}},
		{MutatingWebhookName, "", &admissionregistrationv1.MutatingWebhookConfiguration{}},
		{CipValidatingWebhookName, "", &admissionregistrationv1.ValidatingWebhookConfiguration{}},
		{CipMutatingWebhookName, "", &admissionregistrationv1.MutatingWebhookConfiguration{}},
		{WebhookSvc, InstallNamespace, &corev1.Service{}},
		{MetricsSvc, InstallNamespace, &corev1.Service{}},
		{SecretName, InstallNamespace, &corev1.Secret{}},
		{"config-policy-controller", InstallNamespace, &corev1.ConfigMap{}},
		{"config-image-policies", InstallNamespace, &corev1.ConfigMap{}},
		{"config-sigstore-keys", InstallNamespace, &corev1.ConfigMap{}},
		{"policycontroller-sample-policy-controller-webhook-logging", InstallNamespace, &corev1.ConfigMap{}},
	}

	backoff := wait.Backoff{


 ... (clipped 31 lines)

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Secure Logging Practices

Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.

Status:
No Log Context: The tests and helpers perform Kubernetes operations but do not add or modify any logging;
while typical for tests, we cannot verify from the diff whether logs elsewhere might
expose sensitive data or lack structure.

Referred Code
It("should reject pod creation in a watched namespace and attach an SBOM", func(ctx SpecContext) {
	Expect(e2e_utils.CreateTestPod(ctx, k8sClient, byokTestNS, byokImage)).
		To(MatchError(ContainSubstring(`admission webhook "policy.rhtas.com" denied the request`)))
	e2e_utils.AttachSBOM(ctx, byokImage)
})

It("should accept the pod", func(ctx SpecContext) {
	Eventually(func(ctx SpecContext) error {
		err := e2e_utils.CreateTestPod(ctx, k8sClient, byokTestNS, byokImage)
		if apierrors.IsAlreadyExists(err) {
			return nil
		}
		return err
	}).WithContext(ctx).WithPolling(5*time.Second).Should(Succeed(), "pod admission never became allowed")
})

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Security-First Input Validation and Data Handling

Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities

Status:
Input Handling: The new ImageRepoPrefix-based glob narrows matching which improves safety, but external
inputs (image name envs) flow into templates and Kubernetes objects without visible
validation in this diff.

Referred Code
metadata:
  name: {{ .CIP_NAME }}
spec:
  images:
    - glob: "{{ .TEST_IMAGE_PREFIX }}**"
  authorities:
    - keyless:
        url: {{ .FULCIO_URL }}
        trustRootRef: {{ .TRUST_ROOT_REF }}

Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend 🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes - here's some feedback:

  • The Eventually blocks that wrap CreateTestPod in the three e2e suites are identical aside from namespace/image; consider extracting this into a shared helper to avoid repetition and keep the behavior consistent if the polling logic needs to change again.
  • In WaitForPolicyControllerResourcesDeleted, the hard-coded list of policy-controller resources may drift from what the operator actually creates; if possible, consider deriving this list from labels or ownership (e.g., listing by label selector) to make the cleanup more resilient to future changes.
  • For the new Eventually usages, you configure polling but rely on the default timeout; consider setting an explicit .WithTimeout(...) to make the expected maximum wait time clear and avoid excessively long hangs if admission never becomes allowed.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `Eventually` blocks that wrap `CreateTestPod` in the three e2e suites are identical aside from namespace/image; consider extracting this into a shared helper to avoid repetition and keep the behavior consistent if the polling logic needs to change again.
- In `WaitForPolicyControllerResourcesDeleted`, the hard-coded list of policy-controller resources may drift from what the operator actually creates; if possible, consider deriving this list from labels or ownership (e.g., listing by label selector) to make the cleanup more resilient to future changes.
- For the new `Eventually` usages, you configure polling but rely on the default timeout; consider setting an explicit `.WithTimeout(...)` to make the expected maximum wait time clear and avoid excessively long hangs if admission never becomes allowed.

## Individual Comments

### Comment 1
<location> `test/e2e/byok_install_test.go:249-256` </location>
<code_context>
 	})

 	It("should accept the pod", func(ctx SpecContext) {
-		Expect(e2e_utils.CreateTestPod(ctx, k8sClient, byokTestNS, byokImage)).NotTo(HaveOccurred())
+		Eventually(func(ctx SpecContext) error {
+			err := e2e_utils.CreateTestPod(ctx, k8sClient, byokTestNS, byokImage)
+			if apierrors.IsAlreadyExists(err) {
+				return nil
+			}
+			return err
+		}).WithContext(ctx).WithPolling(5*time.Second).Should(Succeed(), "pod admission never became allowed")
 	})
 })
</code_context>

<issue_to_address>
**suggestion (testing):** Deduplicate the new Eventually pod-creation pattern across e2e suites

This Eventually-based pod admission check (including the AlreadyExists handling) is now duplicated across BYOK, common installation, and serialized TUF root suites. To simplify future changes to polling or error handling, extract it into a shared helper in `test/e2e/utils` (e.g., `EventuallyCreateTestPod(...)`) and reuse it in all three tests.

Suggested implementation:

```golang
	It("should accept the pod", func(ctx SpecContext) {
		e2e_utils.EventuallyCreateTestPod(ctx, k8sClient, byokTestNS, byokImage)
	})
})

```

To fully implement the deduplication you described, you will also need to:
1. Add a helper to `test/e2e/utils` (e.g., in `test/e2e/utils/pod.go` or similar):

   - `func EventuallyCreateTestPod(ctx SpecContext, k8sClient client.Client, ns, image string)` that:
     - Wraps `Eventually(func(ctx SpecContext) error { ... }).WithContext(ctx).WithPolling(5*time.Second).Should(Succeed(), "pod admission never became allowed")`
     - Calls `CreateTestPod(...)` and treats `apierrors.IsAlreadyExists(err)` as success.

2. Update the other suites that currently duplicate this pattern (common installation and serialized TUF root tests) to call `e2e_utils.EventuallyCreateTestPod(...)` instead of inlining the `Eventually` block.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread test/e2e/byok_install_test.go
@qodo-code-review

Copy link
Copy Markdown

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
High-level
Re-evaluate the need for serial execution

Consider removing the Serial flag from the test suites. The newly added cleanup
and retry logic might be sufficient to ensure test stability, allowing for a
return to faster parallel execution.

Examples:

test/e2e/byok_install_test.go [42]
var _ = Describe("policy-controller-operator byok", Ordered, Serial, func() {
test/e2e/common_install_test.go [42]
var _ = Describe("policy-controller-operator common installation", Ordered, Serial, func() {

Solution Walkthrough:

Before:

// In byok_install_test.go, common_install_test.go, etc.

var _ = Describe("test suite description", Ordered, Serial, func() {
  DeferCleanup(func(ctx SpecContext) {
    // ... other cleanup
    Expect(e2e_utils.WaitForPolicyControllerResourcesDeleted(ctx, k8sClient)).To(Succeed())
  })

  // ... other tests

  It("should accept the pod", func(ctx SpecContext) {
    Eventually(func(ctx SpecContext) error {
      // retry logic for pod creation
    }).Should(Succeed())
  })
})

After:

// In byok_install_test.go, common_install_test.go, etc.

var _ = Describe("test suite description", Ordered, func() { // Note: Serial flag removed
  DeferCleanup(func(ctx SpecContext) {
    // ... other cleanup
    Expect(e2e_utils.WaitForPolicyControllerResourcesDeleted(ctx, k8sClient)).To(Succeed())
  })

  // ... other tests

  It("should accept the pod", func(ctx SpecContext) {
    Eventually(func(ctx SpecContext) error {
      // retry logic for pod creation
    }).Should(Succeed())
  })
})
Suggestion importance[1-10]: 8

__

Why: This is a significant suggestion that correctly identifies a major performance trade-off, questioning if the addition of the Serial flag is necessary given the other stability improvements, which could restore faster parallel test execution.

Medium
General
Remove redundant deletion attempts

Refactor the WaitForPolicyControllerResourcesDeleted function to only check for
resource deletion instead of actively deleting them. This avoids redundant API
calls and potential race conditions with the controller's own cleanup process.

test/e2e/utils/kubernetes.go [153-174]

 for _, res := range resources {
 	if err := wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) {
 		obj := res.obj.DeepCopyObject().(client.Object)
 		obj.SetName(res.name)
 		obj.SetNamespace(res.namespace)
 
 		err := k8sClient.Get(ctx, client.ObjectKey{Namespace: res.namespace, Name: res.name}, obj)
-		switch {
-		case errors.IsNotFound(err):
+		if errors.IsNotFound(err) {
 			return true, nil
-		case err != nil:
-			return false, err
-		default:
-			if err := k8sClient.Delete(ctx, obj); err != nil && !errors.IsNotFound(err) {
-				return false, err
-			}
-			return false, nil
 		}
+		return false, err
 	}); err != nil {
 		return fmt.Errorf("resource %T %q still present: %w", res.obj, res.name, err)
 	}
 }
  • Apply / Chat
Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies that the WaitForPolicyControllerResourcesDeleted function should only wait for deletion, not perform it, as another part of the test already triggers the deletion. Removing the redundant Delete call simplifies the logic and avoids potential race conditions, improving the test's robustness.

Medium
  • More

@JasonPowr
JasonPowr merged commit 9e9d699 into main Dec 4, 2025
8 checks passed
@JasonPowr
JasonPowr deleted the fixup-e2e branch December 4, 2025 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants