Skip to content

CMP-3858: Enhanced TestScheduledSuiteTimeoutFail to also cover MaxRetryOnTimeout#1162

Open
taimurhafeez wants to merge 1 commit intoComplianceAsCode:masterfrom
taimurhafeez:CMP-3858
Open

CMP-3858: Enhanced TestScheduledSuiteTimeoutFail to also cover MaxRetryOnTimeout#1162
taimurhafeez wants to merge 1 commit intoComplianceAsCode:masterfrom
taimurhafeez:CMP-3858

Conversation

@taimurhafeez
Copy link
Copy Markdown
Collaborator

This PR adds another point to TestScheduledSuiteTimeoutFail test that validates retry attempts in combination with Timeout.

Thus, the test successfully validates that the compliance operator correctly handles both:

  • Immediate timeout failures (no retry)
  • Timeout with retry attempts (retry exhaustion)

Tested on OCP 4.21:

make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScheduledSuiteTimeoutFail"
cd config/manager && /home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize edit set image compliance-operator=ghcr.io/complianceascode/compliance-operator:latest
rm -rf tests/_setup
mkdir -p tests/_setup
/home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize build config/no-ns | sed -e 's%ghcr.io/complianceascode/compliance-operator:latest%ghcr.io/complianceascode/compliance-operator:latest%' -e 's%ghcr.io/complianceascode/k8scontent:latest%ghcr.io/complianceascode/k8scontent:latest%' -e 's%ghcr.io/complianceascode/openscap-ocp:latest%ghcr.io/complianceascode/openscap-ocp:latest%'  > tests/_setup/deploy_rbac.yaml
/home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize build config/crd > tests/_setup/crd.yaml
2026/04/07 12:03:11 switching to /home/thafeez/repos/CaC/compliance-operator directory to setup and execute tests
2026/04/07 12:03:11 creating namespace osdk-e2e-64799c8e-c506-45aa-8235-4de092296f58
2026/04/07 12:03:12 creating cluster resources in tests/_setup/crd.yaml
2026/04/07 12:03:12 creating CustomResourceDefinition compliancecheckresults.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition complianceremediations.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition compliancescans.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition compliancesuites.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition customrules.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition profilebundles.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition profiles.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition rules.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition scansettingbindings.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition scansettings.compliance.openshift.io
2026/04/07 12:03:15 creating CustomResourceDefinition tailoredprofiles.compliance.openshift.io
2026/04/07 12:03:15 creating CustomResourceDefinition variables.compliance.openshift.io
2026/04/07 12:03:22 updating manifest tests/_setup/deploy_rbac.yaml with namespace osdk-e2e-64799c8e-c506-45aa-8235-4de092296f58
2026/04/07 12:03:22 creating namespaced resources in tests/_setup/deploy_rbac.yaml
2026/04/07 12:03:22 creating ServiceAccount api-resource-collector
2026/04/07 12:03:22 creating ServiceAccount compliance-operator
2026/04/07 12:03:22 creating ServiceAccount compliance-operator-metrics
2026/04/07 12:03:23 creating ServiceAccount profileparser
2026/04/07 12:03:23 creating ServiceAccount remediation-aggregator
2026/04/07 12:03:23 creating ServiceAccount rerunner
2026/04/07 12:03:23 creating ServiceAccount resultscollector
2026/04/07 12:03:23 creating ServiceAccount resultserver
2026/04/07 12:03:24 creating Role api-resource-collector
2026/04/07 12:03:24 creating Role compliance-operator
2026/04/07 12:03:24 creating Role leader-election-role
2026/04/07 12:03:24 creating Role profileparser
2026/04/07 12:03:24 creating Role remediation-aggregator
2026/04/07 12:03:24 creating Role rerunner
2026/04/07 12:03:25 creating Role resultscollector
2026/04/07 12:03:25 creating Role resultserver
2026/04/07 12:03:25 creating ClusterRole api-resource-collector
2026/04/07 12:03:25 creating ClusterRole compliance-operator
2026/04/07 12:03:25 creating ClusterRole compliance-operator-metrics
2026/04/07 12:03:26 creating ClusterRole complianceremediation-editor-role
2026/04/07 12:03:26 creating ClusterRole complianceremediation-viewer-role
2026/04/07 12:03:26 creating ClusterRole compliancescan-editor-role
2026/04/07 12:03:26 creating ClusterRole compliancescan-viewer-role
2026/04/07 12:03:26 creating ClusterRole compliancesuite-editor-role
2026/04/07 12:03:27 creating ClusterRole compliancesuite-viewer-role
2026/04/07 12:03:27 creating ClusterRole profilebundle-editor-role
2026/04/07 12:03:27 creating ClusterRole profilebundle-viewer-role
2026/04/07 12:03:27 creating ClusterRole remediation-aggregator
2026/04/07 12:03:27 creating ClusterRole scansettingbinding-editor-role
2026/04/07 12:03:28 creating ClusterRole scansettingbinding-viewer-role
2026/04/07 12:03:28 creating ClusterRole tailoredprofile-editor-role
2026/04/07 12:03:28 creating ClusterRole tailoredprofile-viewer-role
2026/04/07 12:03:28 creating RoleBinding api-resource-collector
2026/04/07 12:03:28 creating RoleBinding compliance-operator
2026/04/07 12:03:29 creating RoleBinding compliance-operator-metrics
2026/04/07 12:03:29 creating RoleBinding leader-election-rolebinding
2026/04/07 12:03:29 creating RoleBinding profileparser
2026/04/07 12:03:29 creating RoleBinding remediation-aggregator
2026/04/07 12:03:30 creating RoleBinding rerunner
2026/04/07 12:03:30 creating RoleBinding resultscollector
2026/04/07 12:03:30 creating RoleBinding resultserver
2026/04/07 12:03:30 creating ClusterRoleBinding api-resource-collector
2026/04/07 12:03:30 creating ClusterRoleBinding compliance-operator
2026/04/07 12:03:31 creating ClusterRoleBinding remediation-aggregator
2026/04/07 12:03:31 creating Deployment compliance-operator
2026/04/07 12:03:37 Deployment available (1/1)
2026/04/07 12:03:42 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:47 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:52 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:57 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:04:02 ProfileBundle ready (VALID)
2026/04/07 12:04:07 ProfileBundle ready (VALID)
2026/04/07 12:04:09 adding label node-role.kubernetes.io/e2e to node ip-10-0-115-247.us-west-2.compute.internal
2026/04/07 12:04:10 successfully created Machine Config Pool e2e
2026/04/07 12:04:10 ValidatingAdmissionPolicy 'custom-machine-config-pool-selector' exists, adding minimal selectors to MachineConfigPool e2e-invalid
2026/04/07 12:04:10 Machine Config Pool e2e-invalid already exists
=== RUN   TestScheduledSuiteTimeoutFail
=== PAUSE TestScheduledSuiteTimeoutFail
=== CONT  TestScheduledSuiteTimeoutFail
2026/04/07 12:04:16 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: RUNNING
2026/04/07 12:04:21 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: RUNNING
2026/04/07 12:04:26 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: LAUNCHING
2026/04/07 12:04:31 All scans in ComplianceSuite have finished (test-scheduled-suite-timeout-fail)
--- PASS: TestScheduledSuiteTimeoutFail (21.50s)
PASS

Assisted by Claude.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

@taimurhafeez: This pull request references CMP-3858 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds another point to TestScheduledSuiteTimeoutFail test that validates retry attempts in combination with Timeout.

Thus, the test successfully validates that the compliance operator correctly handles both:

  • Immediate timeout failures (no retry)
  • Timeout with retry attempts (retry exhaustion)

Tested on OCP 4.21:

make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScheduledSuiteTimeoutFail"
cd config/manager && /home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize edit set image compliance-operator=ghcr.io/complianceascode/compliance-operator:latest
rm -rf tests/_setup
mkdir -p tests/_setup
/home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize build config/no-ns | sed -e 's%ghcr.io/complianceascode/compliance-operator:latest%ghcr.io/complianceascode/compliance-operator:latest%' -e 's%ghcr.io/complianceascode/k8scontent:latest%ghcr.io/complianceascode/k8scontent:latest%' -e 's%ghcr.io/complianceascode/openscap-ocp:latest%ghcr.io/complianceascode/openscap-ocp:latest%'  > tests/_setup/deploy_rbac.yaml
/home/thafeez/repos/CaC/compliance-operator/build/_tools/kustomize build config/crd > tests/_setup/crd.yaml
2026/04/07 12:03:11 switching to /home/thafeez/repos/CaC/compliance-operator directory to setup and execute tests
2026/04/07 12:03:11 creating namespace osdk-e2e-64799c8e-c506-45aa-8235-4de092296f58
2026/04/07 12:03:12 creating cluster resources in tests/_setup/crd.yaml
2026/04/07 12:03:12 creating CustomResourceDefinition compliancecheckresults.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition complianceremediations.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition compliancescans.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition compliancesuites.compliance.openshift.io
2026/04/07 12:03:13 creating CustomResourceDefinition customrules.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition profilebundles.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition profiles.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition rules.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition scansettingbindings.compliance.openshift.io
2026/04/07 12:03:14 creating CustomResourceDefinition scansettings.compliance.openshift.io
2026/04/07 12:03:15 creating CustomResourceDefinition tailoredprofiles.compliance.openshift.io
2026/04/07 12:03:15 creating CustomResourceDefinition variables.compliance.openshift.io
2026/04/07 12:03:22 updating manifest tests/_setup/deploy_rbac.yaml with namespace osdk-e2e-64799c8e-c506-45aa-8235-4de092296f58
2026/04/07 12:03:22 creating namespaced resources in tests/_setup/deploy_rbac.yaml
2026/04/07 12:03:22 creating ServiceAccount api-resource-collector
2026/04/07 12:03:22 creating ServiceAccount compliance-operator
2026/04/07 12:03:22 creating ServiceAccount compliance-operator-metrics
2026/04/07 12:03:23 creating ServiceAccount profileparser
2026/04/07 12:03:23 creating ServiceAccount remediation-aggregator
2026/04/07 12:03:23 creating ServiceAccount rerunner
2026/04/07 12:03:23 creating ServiceAccount resultscollector
2026/04/07 12:03:23 creating ServiceAccount resultserver
2026/04/07 12:03:24 creating Role api-resource-collector
2026/04/07 12:03:24 creating Role compliance-operator
2026/04/07 12:03:24 creating Role leader-election-role
2026/04/07 12:03:24 creating Role profileparser
2026/04/07 12:03:24 creating Role remediation-aggregator
2026/04/07 12:03:24 creating Role rerunner
2026/04/07 12:03:25 creating Role resultscollector
2026/04/07 12:03:25 creating Role resultserver
2026/04/07 12:03:25 creating ClusterRole api-resource-collector
2026/04/07 12:03:25 creating ClusterRole compliance-operator
2026/04/07 12:03:25 creating ClusterRole compliance-operator-metrics
2026/04/07 12:03:26 creating ClusterRole complianceremediation-editor-role
2026/04/07 12:03:26 creating ClusterRole complianceremediation-viewer-role
2026/04/07 12:03:26 creating ClusterRole compliancescan-editor-role
2026/04/07 12:03:26 creating ClusterRole compliancescan-viewer-role
2026/04/07 12:03:26 creating ClusterRole compliancesuite-editor-role
2026/04/07 12:03:27 creating ClusterRole compliancesuite-viewer-role
2026/04/07 12:03:27 creating ClusterRole profilebundle-editor-role
2026/04/07 12:03:27 creating ClusterRole profilebundle-viewer-role
2026/04/07 12:03:27 creating ClusterRole remediation-aggregator
2026/04/07 12:03:27 creating ClusterRole scansettingbinding-editor-role
2026/04/07 12:03:28 creating ClusterRole scansettingbinding-viewer-role
2026/04/07 12:03:28 creating ClusterRole tailoredprofile-editor-role
2026/04/07 12:03:28 creating ClusterRole tailoredprofile-viewer-role
2026/04/07 12:03:28 creating RoleBinding api-resource-collector
2026/04/07 12:03:28 creating RoleBinding compliance-operator
2026/04/07 12:03:29 creating RoleBinding compliance-operator-metrics
2026/04/07 12:03:29 creating RoleBinding leader-election-rolebinding
2026/04/07 12:03:29 creating RoleBinding profileparser
2026/04/07 12:03:29 creating RoleBinding remediation-aggregator
2026/04/07 12:03:30 creating RoleBinding rerunner
2026/04/07 12:03:30 creating RoleBinding resultscollector
2026/04/07 12:03:30 creating RoleBinding resultserver
2026/04/07 12:03:30 creating ClusterRoleBinding api-resource-collector
2026/04/07 12:03:30 creating ClusterRoleBinding compliance-operator
2026/04/07 12:03:31 creating ClusterRoleBinding remediation-aggregator
2026/04/07 12:03:31 creating Deployment compliance-operator
2026/04/07 12:03:37 Deployment available (1/1)
2026/04/07 12:03:42 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:47 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:52 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:03:57 waiting ProfileBundle rhcos4 to become VALID (PENDING)
2026/04/07 12:04:02 ProfileBundle ready (VALID)
2026/04/07 12:04:07 ProfileBundle ready (VALID)
2026/04/07 12:04:09 adding label node-role.kubernetes.io/e2e to node ip-10-0-115-247.us-west-2.compute.internal
2026/04/07 12:04:10 successfully created Machine Config Pool e2e
2026/04/07 12:04:10 ValidatingAdmissionPolicy 'custom-machine-config-pool-selector' exists, adding minimal selectors to MachineConfigPool e2e-invalid
2026/04/07 12:04:10 Machine Config Pool e2e-invalid already exists
=== RUN   TestScheduledSuiteTimeoutFail
=== PAUSE TestScheduledSuiteTimeoutFail
=== CONT  TestScheduledSuiteTimeoutFail
2026/04/07 12:04:16 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: RUNNING
2026/04/07 12:04:21 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: RUNNING
2026/04/07 12:04:26 waiting until suite test-scheduled-suite-timeout-fail reaches target status 'DONE'. Current status: LAUNCHING
2026/04/07 12:04:31 All scans in ComplianceSuite have finished (test-scheduled-suite-timeout-fail)
--- PASS: TestScheduledSuiteTimeoutFail (21.50s)
PASS

Assisted by Claude.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 7, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved label Apr 7, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 7, 2026

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1162-cfb1e5e658b75d946706b0b73bad1f85f4e1c63f

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 7, 2026

@taimurhafeez: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-rosa cfb1e5e link true /test e2e-rosa
ci/prow/e2e-aws-serial-arm cfb1e5e link true /test e2e-aws-serial-arm
ci/prow/e2e-aws-serial cfb1e5e link true /test e2e-aws-serial

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants