New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubelet won't read apiserver from kubeconfig #36745

Closed
mattymo opened this Issue Nov 14, 2016 · 21 comments

Comments

Projects
None yet
10 participants
@mattymo

mattymo commented Nov 14, 2016

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No

What keywords did you search in Kubernetes issues before filing this one?
kubelet kubeconfig


Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3+coreos.0", GitCommit:"7819c84f25e8c661321ee80d6b9fa5f4ff06676f", GitTreeState:"clean", BuildDate:"2016-10-17T21:19:17Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3+coreos.0", GitCommit:"7819c84f25e8c661321ee80d6b9fa5f4ff06676f", GitTreeState:"clean", BuildDate:"2016-10-17T21:19:17Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration:

  • OS (e.g. from /etc/os-release): Ubuntu Xenial 16.04

  • Kernel (e.g. uname -a): Linux node1 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools: Kargo (Ansible-based)

  • Others:

What happened:
--api-servers option is deprecated for kubelet, so I am now trying to deploy with simply using --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml

The error in kubelet log is:
W1114 12:00:28.684692 31989 server.go:383] No API client: no api servers specified

What you expected to happen:
Kubelet should start and register itself to my kube apiserver

How to reproduce it (as minimally and precisely as possible):
Execution call:
hyperkube kubelet --v=2 --address=10.90.0.2 --hostname-override=node1 --allow-privileged=true --cluster_dns=10.233.0.2 --cluster_domain=cluster.local --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests --resolv-conf=/etc/resolv.conf --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --network-plugin=cni --network-plugin-dir=/etc/cni/net.d

kubeconfig:

kind: Config
clusters:
- name: local
  cluster:
    certificate-authority: /etc/kubernetes/ssl/ca.pem
    server: https://node1:443
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/node.pem
    client-key: /etc/kubernetes/ssl/node-key.pem
contexts:
- context:
    cluster: local
    user: kubelet
  name: kubelet-cluster.local
current-context: kubelet-cluster.local

Anything else do we need to know:
Kubelet works fine with --api-servers specified, but not when reading the server field in kubeconfig. The results are the same when trying to connect to http or https-based apiserver.

@mattymo

This comment has been minimized.

mattymo commented Nov 14, 2016

I repeated on 1.4.5 with the same results

@dims

This comment has been minimized.

Member

dims commented Nov 14, 2016

@mattymo already fixed in #30798 as part of issue #30515

@mattymo

This comment has been minimized.

mattymo commented Nov 14, 2016

@dims That issue is related and seems to suggest there is no bug. But I am still encountering the issue where Kubelet won't actually communicate with the server located in my kubeconfig. All that bug does is tolerate the absence of kubeconfig (and waits for it to be created, or die if --require-kubeconfig is set).

@mattymo

This comment has been minimized.

mattymo commented Nov 15, 2016

Another update. If you specify --kubeconfig, but omit --api-servers and --require-kubeconfig, kubelet won't connect to any apiserver. If you add --require-kubeconfig, it works. Is that expected behavior?

@liggitt

This comment has been minimized.

Member

liggitt commented Nov 16, 2016

@smarterclayton, I thought --require-kubeconfig was defaulting to true for 1.5

@dims

This comment has been minimized.

Member

dims commented Nov 16, 2016

@mattymo : please specify which kubernetes version you are trying under :)

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Nov 16, 2016

I recall agreeing to that, but I don't know that it was actually changed
yet.

On Nov 16, 2016, at 1:51 AM, Jordan Liggitt notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton, I thought
--require-kubeconfig was defaulting to true for 1.5


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#36745 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pylWDY0a79NbHWiV9QWoyWwctgDyks5q-qf5gaJpZM4KxR8Y
.

@mattymo

This comment has been minimized.

mattymo commented Nov 16, 2016

@dims I already wrote, but it is 1.4.3 and 1.4.6

@dims

This comment has been minimized.

Member

dims commented Nov 16, 2016

Apologies @mattymo !

@dove-young

This comment has been minimized.

dove-young commented Dec 19, 2016

I repeated this error in v1.4.5

If I specify --kubeconfig with --api-servers, it reports No api server defined - no events will be sent to API server. in server.go

/usr/local/bin/kubelet --api-servers=https://127.0.0.1:443 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-cidr=10.0.0.0/24 --kubeconfig=/tmp/kubeconfig.cfg --v=9
Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
I1219 13:00:35.812353   13122 loader.go:354] Config loaded from file /tmp/kubeconfig.cfg
I1219 13:00:35.812869   13122 docker.go:375] Connecting to docker on unix:///var/run/docker.sock
I1219 13:00:35.812892   13122 docker.go:395] Start docker client with request timeout=2m0s
E1219 13:00:35.812975   13122 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
I1219 13:00:35.824670   13122 manager.go:143] cAdvisor running in container: "/"
W1219 13:00:35.991953   13122 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1219 13:00:36.157101   13122 fs.go:117] Filesystem partitions: map[......]
I1219 13:00:36.161664   13122 manager.go:198] Machine: {......}
I1219 13:00:36.162408   13122 manager.go:204] Version: {KernelVersion:3.10.0-327.36.3.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.10.0 CadvisorVersion: CadvisorRevision:}
I1219 13:00:36.165922   13122 oom_linux.go:69] attempting to set "/proc/self/oom_score_adj" to "-999"
W1219 13:00:36.166093   13122 server.go:613] No api server defined - no events will be sent to API server.
I1219 13:00:36.166112   13122 server.go:644] Using root directory: /var/lib/kubelet
W1219 13:00:36.168915   13122 kubelet_network.go:71] Hairpin mode set to "promiscuous-bridge" but configureCBR0 is false, falling back to "hairpin-veth"
I1219 13:00:36.168948   13122 kubelet.go:513] Hairpin mode set to "hairpin-veth"

If I specify --kubeconfig only and omit --api-servers, it reports No API client: no api servers specified in server.go.

While I did not even saw this log Config loaded from file /tmp/kubeconfig.cfg in this situation

/usr/local/bin/kubelet  --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-cidr=10.0.0.0/24 --kubeconfig=/tmp/kubeconfig.cfg --v=9
W1219 13:03:55.090003   13158 server.go:383] No API client: no api servers specified
I1219 13:03:55.090138   13158 docker.go:375] Connecting to docker on unix:///var/run/docker.sock
I1219 13:03:55.090161   13158 docker.go:395] Start docker client with request timeout=2m0s
E1219 13:03:55.090295   13158 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
I1219 13:03:55.099712   13158 manager.go:143] cAdvisor running in container: "/"
W1219 13:03:55.268026   13158 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1219 13:03:55.434289   13158 fs.go:117] Filesystem partitions: map[......]
I1219 13:03:55.439401   13158 manager.go:198] Machine: {......}
I1219 13:03:55.440643   13158 manager.go:204] Version: {KernelVersion:3.10.0-327.36.3.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.10.0 CadvisorVersion:
CadvisorRevision:}
I1219 13:03:55.441871   13158 oom_linux.go:69] attempting to set "/proc/self/oom_score_adj" to "-999"
W1219 13:03:55.442022   13158 server.go:613] No api server defined - no events will be sent to API server.
I1219 13:03:55.442042   13158 server.go:644] Using root directory: /var/lib/kubelet
@workhardcc

This comment has been minimized.

workhardcc commented Jan 18, 2017

I have the same issue in 1.5.1.
Seems --require-kubeconfig default value is false?

@ronaldpetty

This comment has been minimized.

ronaldpetty commented Feb 9, 2017

I see the same behavior on 1.5.2 (node is on same box as api-server, all insecure mode.)

$ cat nodea.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    server: http://172.16.151.129:8080
  name: local
contexts:
- context:
    cluster: local
    user: ""
  name: local
current-context: local
kind: Config
preferences: {}
users: []

$ sudo ~/kubernetes-1.5.2/_output/bin/kubelet --kubeconfig=nodea.kubeconfig --allow-privileged=true
I0209 13:09:50.536340   20585 feature_gate.go:181] feature gates: map[]
W0209 13:09:50.536873   20585 server.go:400] No API client: no api servers specified
...
@liggitt

This comment has been minimized.

Member

liggitt commented Feb 9, 2017

correct, --require-kubeconfig is still needed in 1.5.x

@ronaldpetty

This comment has been minimized.

ronaldpetty commented Feb 9, 2017

Thanks for clarifying, I confirmed it worked with --require-kubeconfig.

@rikatz

This comment has been minimized.

rikatz commented Apr 17, 2017

Tested in 1.6.1, --require-kubeconfig is still required to make this work.

We need to be careful, as api-server in this version is being deprecated (still works, but issues a WARNING about this deprecation) and require-kubeconfig is not clear about being used also for api-server connection.

Is there any plan to make require-kubeconfig default to true in next versions?

Thanks!

@liggitt

This comment has been minimized.

Member

liggitt commented Apr 17, 2017

Yes. The plan is to completely remove --api-servers and default --require-kubeconfig to true in 1.7

@rikatz

This comment has been minimized.

rikatz commented Apr 17, 2017

@liggitt anyway, shouldn't we enable --require-kubeconfig for the next version of kubernetes 1.6, by default?

This way I think migrations are going to be softer than enabling one and disabling other directly by default in 1.7.

Just a suggestion :)

@liggitt

This comment has been minimized.

Member

liggitt commented Apr 18, 2017

Actually, I should speak more precisely. In 1.7, the plan is to remove --api-servers and make the presence of the --kubeconfig flag determine whether an API connection is made. The presence of --kubeconfig will require the specified file and the absence of --kubeconfig will mean the kubelet is in standalone mode.

@ChristopherHanson

This comment has been minimized.

ChristopherHanson commented Aug 1, 2017

Looks like --require-kubeconfig is still required in 1.7.2, just had it happen to me and came across this issue.

@liggitt

This comment has been minimized.

Member

liggitt commented Aug 1, 2017

correct, removing it didn't happen until 1.8 - #40050

@liggitt

This comment has been minimized.

Member

liggitt commented Aug 1, 2017

fixed in 1.8 in #40050

@liggitt liggitt closed this Aug 1, 2017

openstack-gerrit pushed a commit to openstack/magnum that referenced this issue Feb 13, 2018

k8s: Fix kubelet, add RBAC and pass e2e tests
Due to a few several small connected patches for the
fedora atomic driver, this patch includes 4 smaller patches.

Patch 1:
k8s: Do not start kubelet and kube-proxy on master

Patch [1], misses the removal of kubelet and kube-proxy from
enable-services-master.sh and therefore they are started if they
exist in the image or the script will fail.

https://review.openstack.org/#/c/533593/
Closes-Bug: #1726482

Patch 2:
k8s: Set require-kubeconfig when needed

From kubernetes 1.8 [1] --require-kubeconfig is deprecated and
in kubernetes 1.9 it is removed.

Add --require-kubeconfig only for k8s <= 1.8.

[1] kubernetes/kubernetes#36745

Closes-Bug: #1718926

https://review.openstack.org/#/c/534309/

Patch 3:
k8s_fedora: Add RBAC configuration

* Make certificates and kubeconfigs compatible
  with NodeAuthorizer [1].
* Add CoreDNS roles and rolebindings.
* Create the system:kube-apiserver-to-kubelet ClusterRole.
* Bind the system:kube-apiserver-to-kubelet ClusterRole to
  the kubernetes user.
* remove creation of kube-system namespaces, it is created
  by default
* update client cert generation in the conductor with
  kubernetes' requirements
* Add --insecure-bind-address=127.0.0.1 to work on
  multi-master too. The controller manager on each
  node needs to contact the apiserver (on the same node)
  on 127.0.0.1:8080

[1] https://kubernetes.io/docs/admin/authorization/node/

Closes-Bug: #1742420
Depends-On: If43c3d0a0d83c42ff1fceffe4bcc333b31dbdaab
https://review.openstack.org/#/c/527103/

Patch 4:
k8s_fedora: Update coredns config to pass e2e

To pass the e2e conformance tests, coredns needs to
be configured with POD-MODE verified. Otherwise, pods
won't be resolvable [1].

[1] https://github.com/coredns/coredns/tree/master/plugin/kubernetes

https://review.openstack.org/#/c/528566/
Closes-Bug: #1738633

Change-Id: Ibd5245ca0f5a11e1d67a2514cebb2ffe8aa5e7de

openstack-gerrit pushed a commit to openstack/magnum that referenced this issue Feb 14, 2018

k8s: Fix kubelet, add RBAC and pass e2e tests
Due to a few several small connected patches for the
fedora atomic driver, this patch includes 4 smaller patches.

Patch 1:
k8s: Do not start kubelet and kube-proxy on master

Patch [1], misses the removal of kubelet and kube-proxy from
enable-services-master.sh and therefore they are started if they
exist in the image or the script will fail.

https://review.openstack.org/#/c/533593/
Closes-Bug: #1726482

Patch 2:
k8s: Set require-kubeconfig when needed

From kubernetes 1.8 [1] --require-kubeconfig is deprecated and
in kubernetes 1.9 it is removed.

Add --require-kubeconfig only for k8s <= 1.8.

[1] kubernetes/kubernetes#36745

Closes-Bug: #1718926

https://review.openstack.org/#/c/534309/

Patch 3:
k8s_fedora: Add RBAC configuration

* Make certificates and kubeconfigs compatible
  with NodeAuthorizer [1].
* Add CoreDNS roles and rolebindings.
* Create the system:kube-apiserver-to-kubelet ClusterRole.
* Bind the system:kube-apiserver-to-kubelet ClusterRole to
  the kubernetes user.
* remove creation of kube-system namespaces, it is created
  by default
* update client cert generation in the conductor with
  kubernetes' requirements
* Add --insecure-bind-address=127.0.0.1 to work on
  multi-master too. The controller manager on each
  node needs to contact the apiserver (on the same node)
  on 127.0.0.1:8080

[1] https://kubernetes.io/docs/admin/authorization/node/

Closes-Bug: #1742420
Depends-On: If43c3d0a0d83c42ff1fceffe4bcc333b31dbdaab
https://review.openstack.org/#/c/527103/

Patch 4:
k8s_fedora: Update coredns config to pass e2e

To pass the e2e conformance tests, coredns needs to
be configured with POD-MODE verified. Otherwise, pods
won't be resolvable [1].

[1] https://github.com/coredns/coredns/tree/master/plugin/kubernetes

https://review.openstack.org/#/c/528566/
Closes-Bug: #1738633

Change-Id: Ibd5245ca0f5a11e1d67a2514cebb2ffe8aa5e7de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment