KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT weren't set in pod env

[hongchaodeng]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  GKE

What happened:

KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT weren't set in pod env.

What you expected to happen:

KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT should be set by default in pod.

How to reproduce it (as minimally and precisely as possible):

Create in-cluster client in a pod.

This isn't easily reproducible. We encountered this issue when running extensive e2e tests and the logs showed that a pod crashed due to:

panic: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

[hongchaodeng]

@dchen1107 @Random-Liu
Potential node issue?

[calebamiles]

Could someone from @kubernetes/sig-node-bugs please take a look. Thanks!

cc: @dchen1107

[liggitt]

cc @stevekuznetsov

[stevekuznetsov]

Seeing this intermittently as you say, in Origin e2e CI.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2+43a9be4", GitCommit:"43a9be4", GitTreeState:"clean", BuildDate:"2017-02-07T16:24:34Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

[stevekuznetsov]

@pmorie was thinking this may be symptomatic of the kubernetes service not actually starting.

[stevekuznetsov]

This is occurring with pretty high incidence in our CI -- is someone from @kubernetes/sig-node-bugs assigned to triage this?

[resouer]

@hongchaodeng When generating ENVs for container, there is a known race that kubelet may generate ENV before service is started.

See this: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go#L412-L416

May be we should make client-go retry for more times?

[xiang90]

@resouer

May be we should make client-go retry for more times?

I feel this is more like a workaround. This bug should be fixed.

[calebamiles]

ping, @kubernetes/sig-node-bugs can we come up with a plan for either fixing the race condition in the kublet or are we planning on changing client-go @kubernetes/sig-api-machinery-misc?

[shiywang]

met same problem, do we have a workaround here ?

Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.914+b9e8d2aee6d593", GitCommit:"b9e8d2aee6d59330306cb458c1241f5e2578c40b", GitTreeState:"clean", BuildDate:"2017-06-02T04:13:59Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.914+b9e8d2aee6d593", GitCommit:"b9e8d2aee6d59330306cb458c1241f5e2578c40b", GitTreeState:"clean", BuildDate:"2017-06-02T04:13:59Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"}
2017/06/02 04:51:34 util.go:131: Step './hack/e2e-internal/e2e-status.sh' finished in 117.680893ms
2017/06/02 04:51:34 util.go:129: Running: ./hack/ginkgo-e2e.sh --ginkgo.focus=ThirdParty
Setting up for KUBERNETES_PROVIDER="local".
Local doesn't need special preparations for e2e tests
Jun  2 04:51:34.868: INFO: >>> kubeConfig: 
Jun  2 04:51:34.868: INFO: failed to load config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
panic: 

[qiujian16]

met the same problem, especially when we reboot the master node.

[StevenACoffman]

So this is biting us with some regularity, so I assume there is a recommended workaround. Any tips?

It is sometimes problematic to override environment variables in pod definitions as an IP address is expected by some things:

        env:
        - name: KUBERNETES_SERVICE_HOST
          value: "kubernetes.default.svc.cluster.local"
        - name: KUBERNETES_SERVICE_PORT
          value: "443"

I'm wondering if the IP address is available somewhere else? Downward API volume file?
Also, it would be great if there was some way to make the env override conditional on it not already existing.

[stevekuznetsov]

We worked around this by polling until the kubernetes service in namespace default was up and running before launching our pods.

[smarterclayton]

Re: conditional, we are unlikely to that because we want to move to explicit service env vars. What doesn't expect IPs? Nothing about _HOST promises not to be a name, so anything in Kube or it's ecosystem should fix assumptions like that (please help us open bugs for that wherever you've hit that).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[redbaron]

/remove-lifecycle stale

[virendrasuryavanshi]

Met the same issue, is there any workaround?
Kubernetes v1.18.10

FATA[0000] could not get config error="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined" exit status 1