-
Notifications
You must be signed in to change notification settings - Fork 229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
E2E tests startup is flaky #1372
Comments
/kind flake |
Looks like flaked again: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/1378/pull-kueue-test-e2e-main-1-26/1730154566130864128
|
Uhm.... I wonder if we need to increase the timeout or cpu/memory of the job. |
According to the Grafana dashboard, resource requests seem to be sufficient. The max usage for CPU is 8.27 Core, and memory is 1.69 GiB. |
I'm wondering if there are noisy neighbors 🤔 |
Should we just increase the timeout for waiting for the webhook? |
I think so since it seems that jobs have enough resource requests. |
It looks like jobset is still having issues. We should start by increasing the visibility into what is pending. Perhaps obtaining the status of the kueue-controller Pod is enough. |
I'll take this for now |
Comparing the kueue logs from a successful run with an unsuccessful run, something that catches my eye is the absence of the line |
The first relevant log line to be absent is @astefanutti does this give any clues? What should be generating this file? What does it depend on? |
When the certs become ready, it doesn't take more than 5 seconds since the first kueue log line. So a greater timeout is unlikely to fix the issue. I'm currently testing cert-controller 0.9 in an attempt to narrow down a culprit #1489 |
The E2E test is also failing with 0.9. I wanted to try 0.7, but there are some dependency issues that prevented me from trying. I'm curious about whether the issue is related to open-policy-agent/cert-controller#81 |
Uhm... I don't think so. I do see the message:
Meaning that an initial write was sent to the apiserver. It looks like what should happen next is that the kubelet watches the update of the Secret and updates the file. |
@alculquicondor That file has the following lifecycle (simplified with the relevant bits):
In the failing runs, it's step 2.i.b. that fails to proceed. I initially though it could be the kubelet that takes too much time to propagate the Secret update to the volume mount (depending on the I still fail to understand what could cause to break what seems to be the two invariants of the
I'll keep digging, but it may be more productive to create a "DO NOT MERGE" PR with some added data points that could help nailing down why that |
I realise with the exponential backoff configuration that's used in the
The tests time out before the backoff in the So it's still a possibility the update made to the Secret by cert-controller isn't propagated as expected into the volume mount by the kubelet. |
I've just found https://ahmet.im/blog/kubernetes-secret-volumes-delay/ that seems to corroborate the long Secret volume mount update propagation hypothesis. @alculquicondor Based on that article, to validate the test timeout increase approach as in #1491, a value as long as two minutes might be needed. |
That article is very useful, thanks! Yes, it sounds like 2 minutes should be the target (how annoying). Otherwise, I wonder if I can cause a Pod status update (to trigger kubelet sync) using the readiness probes more smartly. But I'll leave that as a follow up. It sounds like something like this might be happening most of the time, as we get the secrets updated within 5s when the test passes. |
During e2e tests webhooks are not ready which results in failing e2e tests
What happened:
E2E test failed on testing kueue readiness
What you expected to happen:
E2E test to succeed
How to reproduce it (as minimally and precisely as possible):
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/1365/pull-kueue-test-e2e-main-1-28/1729486888244350976
Anything else we need to know?:
Environment:
kubectl version
):git describe --tags --dirty --always
):cat /etc/os-release
):uname -a
):The text was updated successfully, but these errors were encountered: