NO-JIRA: Switch integration tests to ordered to save time

[JoelSpeed]

On my machine, this reduces the runtime of the integration tests from about 28m to 6m.

The change here is that we install/uninstall the CRD file once per suite file, rather than once per test case.

We lose randomness of tests within each test suite file, but, the time saving is probably worth that.

Each suite file we have will still be randomised.

[openshift-ci]

Hello @JoelSpeed! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

[openshift-ci-robot]

@JoelSpeed: This pull request explicitly references no jira issue.

In response to this:

On my machine, this reduces the runtime of the integration tests from about 28m to 6m.

The change here is that we install/uninstall the CRD file once per suite file, rather than once per test case.

We lose randomness of tests within each test suite file, but, the time saving is probably worth that.

Each suite file we have will still be randomised.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[hexfusion]

In terms of risk do we have/need a periodic test that would still fully randomize to gate nightly? The time saving is a big deal though 👍

vs ~36m

[JoelSpeed]

To be honest, I think we don't need it no. The tests should all be isolated. After each test case we issue a DeleteAll to delete all resources of the kind we are working on. We are also using generateName. So in theory, every resource we create/attempt to create should have a clean slate and a unique name anyway, theoretically meaning every test is independent.

I think we are probably ok without the randomisation, I added it originally because it's a force of habit for normal testing that is more likely to have side effects from hangovers from other tests

[hexfusion]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hexfusion, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JoelSpeed]

/label acknowledge-critical-fixes-only

[openshift-ci]

@JoelSpeed: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-cluster-config-api-container-v4.16.0-202403221044.p0.g6187e2f.assembly.stream.el9 for distgit ose-cluster-config-api.
All builds following this will include this PR.