Salt reconfiguration to get rid of nginx on GCE #6618

roberthbailey · 2015-04-09T07:27:44Z

Instead, pass a bearer token for kubectl to the apiserver.

install libcap2-bin w/ apiserver
use setcap to allow the apiserver to bind to low numbered ports
set secure port for the apiserver to 443
Kubelet talk to port 443 on gce

roberthbailey · 2015-04-09T07:28:13Z

I've tested that cluster creation works (as does kubectl for the new cluster) but haven't tested cluster upgrade or run e2e tests yet.

roberthbailey · 2015-04-09T07:28:55Z

@smarterclayton @deads2k @liggitt - Hopefully this is more along the lines of what you were looking for than #6508.

lavalamp · 2015-04-09T17:36:47Z

heavy_breathing_cat.gif

zmerlynn · 2015-04-09T17:47:53Z

Surfacing briefly.. I'm still OOO and probably won't get to this until
tomorrow or so. Access and time has been spottier than anticipated.
On Apr 9, 2015 1:37 PM, "Daniel Smith" notifications@github.com wrote:

heavy_breathing_cat.gif

—
Reply to this email directly or view it on GitHub
#6618 (comment)
.

roberthbailey · 2015-04-09T17:49:00Z

@zmerlynn - don't worry. I wanted to get some early feedback from the RedHat folks on this. I don't want it merged before the 0.15 cut next week.

deads2k · 2015-04-09T18:39:44Z

cluster/common.sh

+  # is missing.
+  # Note: we save dot ('.') to $root because the 'with' action overrides it.
+  # See http://golang.org/pkg/text/template/.
+  local token='{{$root := .}}{{with index $root "current-context"}}{{with index $root "contexts" .}}{{with index . "user"}}{{with index $root "users" .}}{{index . "token"}}{{end}}{{end}}{{end}}{{end}}'


This format just changed (today #6476). See https://gist.github.com/deads2k/c294a882c9358ea7679f for examples of the new template format.

Thanks for the heads up @deads2k. I'm waiting on a few things before rebasing this:

We need to do a bit of work in GKE to make this a smooth transition. It'll take a couple of weeks for us to get things in place.

Once the apiserver runs in a container, then I won't need to install the libcap2-bin package or run /sbin/setcap on the apiserver, which will simplify things a bit.

I've fixed this to use the new template format.

erictune · 2015-04-13T20:36:59Z

This looks awesome.

zmerlynn · 2015-04-16T19:12:15Z

cluster/gce/configure-vm.sh

  if [ ! -e "${KNOWN_TOKENS_FILE}" ]; then
    mkdir -p /srv/salt-overlay/salt/kube-apiserver
    (umask 077;
-      echo "${KUBELET_TOKEN},kubelet,kubelet" > "${KNOWN_TOKENS_FILE}")
+      echo "${KUBE_BEARER_TOKEN},admin,admin" > "${KNOWN_TOKENS_FILE}";
+      echo "${KUBELET_TOKEN},kubelet,kubelet" >> "${KNOWN_TOKENS_FILE}")


If I'm being pedantic, this has to handle the upgrade case. See the note on line 273/267 below.

Maybe this should actually be split up into a token directory and reassembled here? (Like how .conf files are done with foo.d/* directories?). Just a random thought so that any future upgrades are less obnoxious.

I think your suggestions are orthogonal to this PR (seeing as how the token file already has many files with the service account tokens) but they are good points. Since we don't yet support upgrade, I'm leaning towards leaving this as a breaking change between 0.15.0 and 0.16.0.

roberthbailey · 2015-04-17T23:27:25Z

Rebased. PTAL.

zmerlynn · 2015-04-20T22:16:20Z

LGTM (sorry, rebase again)

roberthbailey · 2015-04-21T04:36:27Z

Rebased. I also updated some documentation files to talk about bearer tokens instead of basic auth. /cc @jlowdermilk

roberthbailey · 2015-04-21T06:08:36Z

Two issues I'm working on.

The create external load balancer e2e test is failing due to a timeout:

Summarizing 1 Failure:

[Fail] Services [It] should be able to create a functioning external load balancer 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:339

Ran 35 of 40 Specs in 1244.857 seconds
FAIL! -- 34 Passed | 1 Failed | 1 Pending | 4 Skipped F0420 22:45:12.452252   99306 driver.go:94] At least one test failed

go run hack/e2e.go -v --up is failing to validate the e2e cluster

Running: /Users/robertbailey/Work/go/src/github.com/roberthbailey/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/darwin/amd64/kubectl get cs
2015/04/20 23:05:23 Error running up: exit status 1
2015/04/20 23:05:23 Error starting e2e cluster. Aborting.
exit status 1

roberthbailey · 2015-04-21T21:58:13Z

go run hack/e2e.go --up is failing because of #7139 (and it's broken for me on the master branch as well).

roberthbailey · 2015-04-21T22:35:20Z

@a-robinson pointed out that the reason the e2e test was failing was that I didn't have a firewall to allow http requests in my project. It's now passing as well:

$ go run hack/e2e.go -v --test --test_args="--ginkgo.focus=Service.*functioning external load balancer.*"
...
Ran 1 of 40 Specs in 61.238 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 39 Skipped I0421 15:34:21.475133   42739 driver.go:96] All tests pass

- Configure the apiserver to listen securely on 443 instead of 6443. - Configure the kubelet to connect to 443 instead of 6443. - Update documentation to refer to bearer tokens instead of basic auth.

roberthbailey · 2015-04-22T18:48:56Z

Now that the latest gcloud release is out, we can move forward with this PR. I rebased on master and re-ran e2e tests. services.sh failed (with the same error I'm seeing in the latest run in Jenkins) but everything else passed.

zmerlynn · 2015-04-22T19:10:30Z

LGTM.

zmerlynn · 2015-04-22T19:11:41Z

Waiting on green, and feedback to https://groups.google.com/forum/#!topic/kubernetes-dev/nf9L-IeB56k

Salt reconfiguration to get rid of nginx on GCE

roberthbailey assigned zmerlynn Apr 9, 2015

googlebot added the cla: yes label Apr 9, 2015

deads2k reviewed Apr 9, 2015
View reviewed changes

ArtfulCoder mentioned this pull request Apr 9, 2015

Add an nginx docker image for use on the master. #6334

Merged

roberthbailey mentioned this pull request Apr 16, 2015

Remaining work for using tokens instead of basic auth #4007

Closed

zmerlynn reviewed Apr 16, 2015
View reviewed changes

roberthbailey force-pushed the no-nginx branch from f12d810 to 451d97b Compare April 17, 2015 23:22

roberthbailey mentioned this pull request Apr 21, 2015

Update accessing-the-cluster.md #7094

Closed

roberthbailey force-pushed the no-nginx branch 3 times, most recently from 7a23393 to 688f382 Compare April 21, 2015 04:34

roberthbailey force-pushed the no-nginx branch from 688f382 to 049e4ac Compare April 21, 2015 04:56

roberthbailey force-pushed the no-nginx branch 2 times, most recently from 540fb10 to a909838 Compare April 22, 2015 05:45

roberthbailey changed the title ~~WIP: Salt reconfiguration to get rid of nginx on GCE~~ Salt reconfiguration to get rid of nginx on GCE Apr 22, 2015

Remove nginx and replace basic auth with bearer token auth for GCE.

dc45f7f

- Configure the apiserver to listen securely on 443 instead of 6443. - Configure the kubelet to connect to 443 instead of 6443. - Update documentation to refer to bearer tokens instead of basic auth.

roberthbailey force-pushed the no-nginx branch from a909838 to dc45f7f Compare April 22, 2015 18:11

zmerlynn added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 22, 2015

zmerlynn added a commit that referenced this pull request Apr 22, 2015

Merge pull request #6618 from roberthbailey/no-nginx

26aeb6c

Salt reconfiguration to get rid of nginx on GCE

zmerlynn merged commit 26aeb6c into kubernetes:master Apr 22, 2015

roberthbailey deleted the no-nginx branch April 23, 2015 04:40

erictune mentioned this pull request Apr 24, 2015

kube-up.sh sometimes doesn't complete #7272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Salt reconfiguration to get rid of nginx on GCE #6618

Salt reconfiguration to get rid of nginx on GCE #6618

roberthbailey commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

lavalamp commented Apr 9, 2015

zmerlynn commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

deads2k Apr 9, 2015

roberthbailey Apr 13, 2015

roberthbailey Apr 17, 2015

erictune commented Apr 13, 2015

zmerlynn Apr 16, 2015

zmerlynn Apr 16, 2015

roberthbailey Apr 17, 2015

zmerlynn Apr 20, 2015

roberthbailey commented Apr 17, 2015

zmerlynn commented Apr 20, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 22, 2015

zmerlynn commented Apr 22, 2015

zmerlynn commented Apr 22, 2015

Salt reconfiguration to get rid of nginx on GCE #6618

Salt reconfiguration to get rid of nginx on GCE #6618

Conversation

roberthbailey commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

lavalamp commented Apr 9, 2015

zmerlynn commented Apr 9, 2015

roberthbailey commented Apr 9, 2015

deads2k Apr 9, 2015

Choose a reason for hiding this comment

roberthbailey Apr 13, 2015

Choose a reason for hiding this comment

roberthbailey Apr 17, 2015

Choose a reason for hiding this comment

erictune commented Apr 13, 2015

zmerlynn Apr 16, 2015

Choose a reason for hiding this comment

zmerlynn Apr 16, 2015

Choose a reason for hiding this comment

roberthbailey Apr 17, 2015

Choose a reason for hiding this comment

zmerlynn Apr 20, 2015

Choose a reason for hiding this comment

roberthbailey commented Apr 17, 2015

zmerlynn commented Apr 20, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 21, 2015

roberthbailey commented Apr 22, 2015

zmerlynn commented Apr 22, 2015

zmerlynn commented Apr 22, 2015