Actions failing due to insufficient CPU

[psschwei]

Description

We have a Action that in the past week or so has started failing with due to an insufficient CPU error. It had previously been running without issue quite a while, so would like to figure out why its all of a sudden failing now.

The action in question is https://github.com/knative-sandbox/kn-plugin-quickstart/actions/workflows/kind-e2e.yaml

Last update was around six months ago, so don't think it's anything we changed: https://github.com/knative-sandbox/kn-plugin-quickstart/commits/main/.github/workflows/kind-e2e.yaml

Per Github support:

"""
I noticed that between the last successful run and the failing run, the runner image version was updated from 20230118.2 to 20230129.2

See below:

Last successful run:
https://github.com/knative-sandbox/kn-plugin-quickstart/actions/runs/3987835817/jobs/6838179199#step:1:9

Failing run:
https://github.com/knative-sandbox/kn-plugin-quickstart/actions/runs/4086545332/jobs/7045957983#step:1:9
Virtual environment updates can sometimes cause problems for previous setups.

For these types of issues, we recommend you open an issue in the repository below:

The maintainers are more suitable to answer your question.
"""

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Failing build: https://github.com/knative-sandbox/kn-plugin-quickstart/actions/runs/4086545332/jobs/7045957983#step:1:9

Runner image:
Image: ubuntu-22.04
Version: 20230129.2
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230129.2/images/linux/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20230129.2

Is it regression?

https://github.com/knative-sandbox/kn-plugin-quickstart/actions/runs/3987835817/jobs/6838179199#step:1:9

Expected behavior

We'd expect to be able to set up a Knative cluster on Kind, have all pods become ready, and the tests in the action to complete successfully.

Actual behavior

The 3scale-kourier-gateway-xxxxxxx pod in the kourier-system namespace is stuck in Pending status due to Warning FailedScheduling 4m42s (x2 over 10m) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Repro steps

Run the action

[mikhailkoliada]

@psschwei Hello! Due to all the difference in setup / code changes / project specifics we do not advise on how to fix issues like this if there is no direct connection between software on the runner we provide and the failure, please make sure that your runs do not consume more than hosted runners may provide (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources). If you have more questions feel free to reach us again.

[psschwei]

@mikhailkoliada our job ran fine for 9 months, it only started running out of CPU when the image runner was updated to v20230129.2, so the question we have is why something that didn't consume more than 2 cpus before now all of a sudden is when our job hasn't changed.