Prow pod for scale tests are stuck eternally in running state and not reported back to Prow the final state. #31769
Labels
kind/bug
Categorizes issue or PR as related to a bug.
lifecycle/stale
Denotes an issue or PR has remained open with no activity and has become stale.
sig/k8s-infra
Categorizes an issue or PR as relevant to SIG K8s Infra.
What happened:
The prow scale test pod here has been running for more than
16hrs
however ideally this particular scale tests runs roughly for around 4hrs.In this particular case, the prow scale test pod OOM'd out, I'm not sure if that is the reason why it kept reporting as
running
forever (19hrs) until I kicked off another test.What you expected to happen:
I would expect the prow to report back failure or success instead of getting stuck eternally or until next test is kicked off to prow cluster.
How to reproduce it (as minimally and precisely as possible):
This happens inconsistently on scale tests, may be try to OOM out the prow test pod to see if it could get into this state.
For more context - kubernetes/k8s.io#6303 (comment)
Please provide links to example occurrences, if any:
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kops/16296/presubmit-kops-aws-scale-amazonvpc-using-cl2/1752113814377074688
Anything else we need to know?:
The text was updated successfully, but these errors were encountered: