Register the kubelet on the master node with an apiserver. #12349

roberthbailey · 2015-08-06T17:16:55Z

This option is separated from the apiserver running locally on the master node so that it
can be optionally enabled or disabled as needed.

Also, fix the healthchecking configuration for the master components, which
was previously only working by coincidence:

If a kubelet doesn't register with a master, it never bothers to figure out
what its local address is. In which case it ends up constructing a URL like
http://:8080/healthz for the http probe. This happens to work on the master
because all of the pods are using host networking and explicitly binding to
127.0.0.1. Once the kubelet is registered with the master and it determines
the local node address, it tries to healthcheck on an address where the pod
isn't listening and the kubelet periodically restarts each master component
when the liveness probe fails.

roberthbailey · 2015-08-06T17:17:38Z

/cc @dchen1107

k8s-bot · 2015-08-06T17:59:01Z

GCE e2e build/test passed for commit 21628c6f4fc82746497279b5bc26837ac9e75bdd.

roberthbailey · 2015-08-06T18:48:46Z

The shippable error is

!!! [0806 18:09:17] Timed out waiting for kubelet(masterless) to answer at http://127.0.0.1:10248/healthz; tried 25 waiting 0.5 between each
!!! Error in ./hack/test-cmd.sh:48
'return 1' exited with status 1
Call stack:
1: ./hack/test-cmd.sh:48 main(...)
Exiting with status 1

which seems unrelated to this change. I've restarted shippable.

roberthbailey · 2015-08-06T18:56:01Z

Running a more complete e2e suite locally:

$ go run hack/e2e.go -v -test --test_args="--ginkgo.skip=Skipped|Restart|Etcd.*|Reboot.*packages|Reboot.*unclean|Reboot.*|Reboot.*network|Reboot.*kernel\spanic|Nodes\sNetwork.*|Shell.*services.sh|Addon\supdate"
...
Ran 97 of 136 Specs in 5137.354 seconds
SUCCESS! -- 97 Passed | 0 Failed | 2 Pending | 37 Skipped PASS

Ginkgo ran 1 suite in 1h25m43.05540199s
Test Suite Passed

zmerlynn · 2015-08-06T19:00:18Z

cluster/validate-cluster.sh

  found=$(cat "${MINIONS_FILE}" | sed '1d' | grep -c .) || true
  ready=$(cat "${MINIONS_FILE}" | sed '1d' | awk '{print $NF}' | grep -c '^Ready') || true

-  if (( ${found} == "${NUM_MINIONS}" )) && (( ${ready} == "${NUM_MINIONS}")); then
+  if (( ${found} == "${EXPECTED_NUM_NODES}" )) && (( ${ready} == "${EXPECTED_NUM_NODES}")); then


nit: "${found}" and "${ready}"? (pre-existing).

Fixed and pushed squashed changes.

zmerlynn · 2015-08-06T19:00:42Z

LGTM modulo nit

roberthbailey · 2015-08-06T20:34:14Z

Next shippable flake:

ok k8s.io/kubernetes/pkg/client/cache   1.702s  coverage: 75.9% of statements
I0806 19:06:37.638803 12059 portforward.go:260] Forwarding from localhost:12345 -> 12345
I0806 19:06:37.639627 12059 portforward.go:260] Forwarding from 127.0.0.1:12345 -> 12345
I0806 19:06:37.640164 12059 portforward.go:260] Forwarding from [::1]:12345 -> 12345
E0806 19:06:37.640593 12059 portforward.go:246] Unable to create listener: Error listen tcp4 [::1]:12345: non-IPv4 address
E0806 19:06:37.640964 12059 portforward.go:246] Unable to create listener: Error listen tcp6 127.0.0.1:12345: bind: invalid argument
E0806 19:06:37.641248 12059 portforward.go:246] Unable to create listener: Error listen tcp6: too many colons in address ::1:12345
I0806 19:06:37.649375 12059 portforward.go:260] Forwarding from 127.0.0.1:5000 -> 5000
I0806 19:06:37.649812 12059 portforward.go:260] Forwarding from [::1]:5000 -> 5000
I0806 19:06:37.656432 12059 portforward.go:260] Forwarding from 127.0.0.1:5000 -> 5000
I0806 19:06:37.657112 12059 portforward.go:260] Forwarding from [::1]:5000 -> 5000
I0806 19:06:37.657473 12059 portforward.go:260] Forwarding from 127.0.0.1:6000 -> 6000
I0806 19:06:37.672490 12059 portforward.go:260] Forwarding from [::1]:6000 -> 6000
I0806 19:06:37.674596 12059 portforward.go:293] Handling connection for 5000
--- FAIL: TestForwardPorts (0.04s)
portforward_test.go:373: 2: expected to read '1234', got ''
FAIL
coverage: 82.4% of statements
FAIL    k8s.io/kubernetes/pkg/client/portforward    1.328s

…n is separated from the apiserver running locally on the master node so that it can be optionally enabled or disabled as needed. Also, fix the healthchecking configuration for the master components, which was previously only working by coincidence: If a kubelet doesn't register with a master, it never bothers to figure out what its local address is. In which case it ends up constructing a URL like http://:8080/healthz for the http probe. This happens to work on the master because all of the pods are using host networking and explicitly binding to 127.0.0.1. Once the kubelet is registered with the master and it determines the local node address, it tries to healthcheck on an address where the pod isn't listening and the kubelet periodically restarts each master component when the liveness probe fails.

roberthbailey · 2015-08-06T20:42:28Z

Shippable is running again on the slightly modified changes. Third time is the charm?

k8s-bot · 2015-08-06T20:59:51Z

GCE e2e build/test passed for commit 8df33bc.

dchen1107 · 2015-08-06T21:54:14Z

cluster/saltbase/salt/kube-scheduler/kube-scheduler.manifest

-"metadata": {"name":"kube-scheduler"},
+"metadata": {
+  "name":"kube-scheduler",
+  "namespace": "kube-system"


Are we sure we want all master components are in the same namespaces as our addons?

Not sure, no. But it seemed weird to have them in the default namespace, and kube-system is where we've been putting the system components. The master components are also system components, so that seemed like the logical place to move them to.

I guess I can live with it in kube-system for now.

dchen1107 · 2015-08-06T22:20:30Z

LGTM.

Register the kubelet on the master node with an apiserver.

dchen1107 · 2015-08-06T22:21:31Z

cc/ @vishh Now we can make sure heapster has master components' stats. :-)

vishh · 2015-08-06T22:32:48Z

Yay! Thanks for this PR!

On Thu, Aug 6, 2015 at 3:22 PM, Dawn Chen notifications@github.com wrote:

cc/ @vishh https://github.com/vishh Now we can make sure heapster has
master components' stats. :-)

—
Reply to this email directly or view it on GitHub
#12349 (comment)
.

…-#11483-#12349-upstream-release-1.0 Automated cherry pick of #11483 #12349

derekwaynecarr · 2015-08-10T21:36:39Z

Ok, so something in this PR broke the vagrant setup.

I am guessing its the removal of kubernetes_auth, @justinsb - are you still working fine on AWS?

I am not sure why that had to be removed as part of this PR, but I need more time to dig into what is actually happening here and why.

derekwaynecarr · 2015-08-10T21:38:25Z

cluster/saltbase/salt/kubelet/init.sls

@@ -31,19 +31,6 @@
    - mode: 400
    - makedirs: true

-#


Was there a problem keeping /var/lib/kubelet/kubernetes_auth in addition to /var/lib/kubelet/kubeconfig?

I was trying to clean it up since I thought everyone had moved off of it ages ago.

I would also like to rip it out of https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cmd/kubelet/app/server.go FWIW.

steelbrain · 2016-10-14T02:40:37Z

Is it possible to run pods on kubes master on AWS (not GCE)? or is it at least planned?

roberthbailey · 2016-10-14T17:43:28Z

It's possible on any kubernetes deployment. It's just a matter of which flags you pass to the kubelet running on the master.

How are you creating your cluster on AWS? Are you using ./cluster/kube-up.sh or kops?

@justinsb

steelbrain · 2016-10-14T17:46:04Z

I'm using ./cluster/kube-up.sh with some env vars that determine the size of the aws ec2 instances spawned for both master and minions. Currently it has one master and two minions but I only see the minions when I goto kube's nodes dashboard

roberthbailey · 2016-10-14T18:19:20Z

I don't think this flag has been plumbed through the aws startup scripts (this PR only did it for the GCE ones, since that's all I can test).

steelbrain · 2016-10-14T19:29:19Z

Is there any help someone can provide (like manually testing your PRs) to make this feature available for aws users? :)

dwiyerr · 2019-05-07T17:16:58Z

What is/are the downside of scheduling pods on master?

googlebot added the cla: yes label Aug 6, 2015

roberthbailey assigned zmerlynn Aug 6, 2015

zmerlynn reviewed Aug 6, 2015
View reviewed changes

roberthbailey force-pushed the kubelet-on-master branch from 21628c6 to 8df33bc Compare August 6, 2015 20:39

zmerlynn added lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-merge labels Aug 6, 2015

dchen1107 reviewed Aug 6, 2015
View reviewed changes

dchen1107 added a commit that referenced this pull request Aug 6, 2015

Merge pull request #12349 from roberthbailey/kubelet-on-master

2fa3004

Register the kubelet on the master node with an apiserver.

dchen1107 merged commit 2fa3004 into kubernetes:master Aug 6, 2015

zmerlynn mentioned this pull request Aug 6, 2015

Automated cherry pick of #11483 #12349 #12367

Merged

zmerlynn added a commit that referenced this pull request Aug 6, 2015

Merge pull request #12367 from roberthbailey/automated-cherry-pick-of…

aebf6b5

…-#11483-#12349-upstream-release-1.0 Automated cherry pick of #11483 #12349

brendandburns mentioned this pull request Aug 7, 2015

Release 1.0.3 #12382

Merged

derekwaynecarr reviewed Aug 10, 2015
View reviewed changes

derekwaynecarr mentioned this pull request Aug 11, 2015

Fix vagrant kube-up #12514

Merged

roberthbailey mentioned this pull request Aug 13, 2015

Cluster creation on GCE failed with version error (but cluster seems OK) #12617

Closed

roberthbailey deleted the kubelet-on-master branch August 20, 2015 21:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Register the kubelet on the master node with an apiserver. #12349

Register the kubelet on the master node with an apiserver. #12349

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

k8s-bot commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

zmerlynn Aug 6, 2015

roberthbailey Aug 6, 2015

zmerlynn commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

k8s-bot commented Aug 6, 2015

dchen1107 Aug 6, 2015

roberthbailey Aug 6, 2015

dchen1107 Aug 6, 2015

dchen1107 commented Aug 6, 2015

dchen1107 commented Aug 6, 2015

vishh commented Aug 6, 2015

derekwaynecarr commented Aug 10, 2015

derekwaynecarr Aug 10, 2015

roberthbailey Aug 10, 2015

roberthbailey Aug 10, 2015

steelbrain commented Oct 14, 2016

roberthbailey commented Oct 14, 2016

steelbrain commented Oct 14, 2016

roberthbailey commented Oct 14, 2016

steelbrain commented Oct 14, 2016

dwiyerr commented May 7, 2019

Register the kubelet on the master node with an apiserver. #12349

Register the kubelet on the master node with an apiserver. #12349

Conversation

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

k8s-bot commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmerlynn commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

roberthbailey commented Aug 6, 2015

k8s-bot commented Aug 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dchen1107 commented Aug 6, 2015

dchen1107 commented Aug 6, 2015

vishh commented Aug 6, 2015

derekwaynecarr commented Aug 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steelbrain commented Oct 14, 2016

roberthbailey commented Oct 14, 2016

steelbrain commented Oct 14, 2016

roberthbailey commented Oct 14, 2016

steelbrain commented Oct 14, 2016

dwiyerr commented May 7, 2019