New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite} #30548
Comments
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30844. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30865. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30940. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #26057. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30713. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/8513/ Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
|
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #28921. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31039. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #29741. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/8533/ Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
|
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30730. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/8594/ Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
|
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31139. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31139. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30931. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31177. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30787. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31330. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31159. |
@euank I think you are right that the issue is not limited to coreos image. Re-assign it back to me. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31642. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/8860/ Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
|
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31769. |
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #31644. |
Should we remove the |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/8882/ Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
|
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30739. |
From comment in kubelet.go:
I think that's probably what's happening here. |
Sounds like increasing the wait time should help? |
It should, but we would need to wait before the pod is created. Is there a way to tell whether the Kubelet has found the service? |
Not that I know of, unless we expose the information through an endpoint, which sounds like an overkill : |
We could also modify the test to take the advice of "detect the missing env-var and restart". If the test doesn't fatal out when the service environment variables aren't there, but instead re-runs the client up to some duration of retries, that would make it less flaky. And we can at least assert that "within 1 minute the environment variables definitely should have propagated, else this is a bug" I'd think. |
We might be able to do something fancy with InitContainers, but I think @euank 's suggestion is probably the way to go. |
I also thought about letting the container fail and restart, but decided against it since I didn't like relying on kubelet's restart backoff interval. Recreating the pod sounds ok. |
What about making the pod restart on failure, and changing the command to |
That's what I originally wanted, but then decided not to suggest it because we don't have control over kubelet restart's backoff. Although I think it should work in general. |
Ok, I'll send a PR to add manual retries. |
Automatic merge from submit-queue Add retries to service environment variable test Fixes (hopefully): #30548 (comment) /cc @euank @dchen1107
Hopefully this is fixed. We'll reopen if it happens again. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30200/node-pull-build-e2e-test/18598/
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite}
Happened on a presubmit run in #30200.
The text was updated successfully, but these errors were encountered: