alpha/beta periodic jobs: re-enable running slow tests

[pohly]

What would you like to be added:

Revert #34607, i.e. re-apply #34584.

We had to revert because it made the job flaky. There are two issues which may have to be solved first:

• The "Pods should cap back-off at MaxContainerBackOff" test runs for ~27 minutes, by design. If it doesn't get started early enough, it runs into the 1 hour time limit for these jobs (example).
• Something, perhaps the same test, caused stability issues in other tests which were stable before. One symptom was failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unable to freeze, another an unexpected exit code of 2.

Why is this needed:

Not running slow tests missed a regression.

/cc @aojea @BenTheElder

[BenTheElder]

/sig testing
/triage accepted
/milestone v1.34

[k8s-ci-robot]

@BenTheElder: The provided milestone is not valid for this repository. Milestones in this repository: [1.33, someday]

Use /milestone clear to clear the milestone.

In response to this:

/sig testing
/triage accepted
/milestone v1.34

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

The "Pods should cap back-off at MaxContainerBackOff" test runs for ~27 minutes, by design. If it doesn't get started early enough, it runs into the 1 hour time limit for these jobs (example).

We might need another tier like [Slow][VerySlow] or something 🤔

[swastik959]

I would Like to work on it if you permit
@BenTheElder Should I add separate timeout for different scenarios for slow and very slow and then add an verslow label to kubernetes main repo ?
/assign

[BenTheElder]

cc @aojea

I think:

The "Pods should cap back-off at MaxContainerBackOff" test runs for ~27 minutes, by design. If it doesn't get started early enough, it runs into the 1 hour time limit for these jobs (example).

This test could almost be categorized as Disruptive.

I'm hesitant to add a label like VerySlow that most of our CI jobs will not be configured to skip (we need to not only consider these specific jobs) and that may encourage writing more of these ridiculously slow tests.

[pohly]

I was discussing with @onsi that it would be useful to prioritize slow tests so that they get started early and then overlap with most of the other tests, without running into the job timeout. It's not available yet in Ginkgo.

[aojea]

I was discussing with @onsi that it would be useful to prioritize slow tests so that they get started early and then overlap with most of the other tests, without running into the job timeout. It's not available yet in Ginkgo.

I do not know if that will be a good solution, as this seems a kind of knapsack problem, usually Slow test consume more resources, so it practically will serialize the first interval, when running a mix of Slow and Fast seems the most optimal at first sight.

Also, I think we do not need to generalize much since we seems to have only one problematic test

The "Pods should cap back-off at MaxContainerBackOff" test runs for ~27 minutes, by design.

that test is indeed Disruptive to me, a test that takes 27 minutes has to make a lot of assumptions, and tagging it as Disruptive seems to solve all the existing problems