api-aggregation setup instructions #895

MHBauer · 2017-05-25T01:18:28Z

mad help from @DirectXMan12 and @liggitt. could not have been anywhere close without them. shoutout to @ivan4th for Mirantis/kubeadm-dind-cluster which has been wicket helpful.

These instructions allow me to set both kubeconfigs of e2e to the same file and successfully pass.

lots of todos embedded in the short doc

need to look at feasibility of I, myself doing the integration in jenkins or leaving for a follow up

apiregistration api is beta in 1.7, alpha in 1.6, so if we intend to support this on 1.6 we need to both show how to set up the aggregator (or at least link to it) and have a switch for the flag based on the k8s version.

RBAC was whack a mole for a while. I need to turn it on in local-up-cluster and finish figuring out all the permissions, or otherwise list them all out by reviewing the code with someone.

Additionally to figuring out our process, we need to do a general guide for other apiservers that may exist in the future. I think it sounds related to the apiserver starter kit. @pwittrock WDYT?

MHBauer · 2017-05-25T02:12:30Z

resolves #935 instead of #880

derekwaynecarr · 2017-05-25T03:35:37Z

+1 we need to get to a point where this is the default and that our e2es verify all content is removed from the namespace when deleted by virtue of this integration.

deads2k · 2017-05-25T18:45:05Z

Step by step directions and script here: https://github.com/openshift/origin/pull/13229/files

MHBauer · 2017-05-26T16:33:59Z

@deads2k what is oc command?

deads2k · 2017-05-26T16:36:30Z

@deads2k what is oc command?

kubectl with a few extras (most generally useful: login, project, and status). I think I made the directions reasonably generic. You'll have to generate your own certs instead of using the service serving cert signer.

DirectXMan12

The content mostly looks ok (a few comments inline). Your formatting is a bit off though, but you can probably clean that up at the end.

DirectXMan12 · 2017-06-07T16:02:07Z

docs/api-aggregation-setup.md

+Need to figure out all of the permissions we need, and group them into a role.
+
+# not sure if I still need to create this (incorrectly spelled) namespace
+kubectl create namespace catlog


not entirely sure what this is here for...

DirectXMan12 · 2017-06-07T16:04:09Z

docs/api-aggregation-setup.md

+You can see we use the generated cert and key, along with the cores CA
+certificate. (TODO: Is this good or bad practice? Is there an
+alternative delegated kind of CA we can use if we don't have access to
+kube itself on a cloud provider.)


so, it doesn't matter what CA you use here, since we explicitly tell the aggregator which CA to use (the caBundle field). Using the core CA is "easy" because it's already there in most setups.

I think that may be more 'obvious' to someone who does this kind of cert management.

I think we should also have a basic workflow beyond using an existing CA of "create a new CA" and proceed from there.

We can probably just point to the relevant section in auth.md.

Oh, brilliant. Did a copy or version of auth.md ever make it to somewhere more visible that we should link to it instead now?

Which section were you thinking, specifically?

There's a copy in the service catalog docs, and a slight more generic copy here: https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md

Either works for me. I was thinking this section: https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md#generating-certificates

Something like "follow the instructions here to generate the CA and equivalent serving certificates. Set PURPOSE to ..."

DirectXMan12 · 2017-06-07T16:08:59Z

docs/api-aggregation-setup.md

+# this won't show up if you are talking to the insecure port. the insecure port is not behind the aggregator.
+
+# Get the appropriate tls ca/cert/keys from kube
+# TODO how do I know which ones to get? Are they ones given to specific flags on the apiserver, etc etc?


You don't need to use the main CA, but if you want to, it's the cert passed with the --root-ca-file flag to the controller-manager.

MHBauer · 2017-06-07T22:49:05Z

docs/api-aggregation-setup.md

+export NAME_SPACE=catalog
+export SERVICE_NAME=catalog-catalog-apiserver
+export ALT_NAMES="\"${SERVICE_NAME}.${NAME_SPACE}\",\"${SERVICE_NAME}.${NAME_SPACE}.svc"\"
+echo '{"CN":"'${SERVICE_NAME}'","hosts":['${ALT_NAMES}'],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=${SERVINGCA_CERT} -ca-key=${SERVINGCA_KEY} -config=server-ca-config.json - | cfssljson -bare apiserver


@DirectXMan12 do -ca= and -ca-key= have to belong to the same CA set?
Is this where I would switch out one or both of them to change the CA in the bundle?

yeah, they should be each other's public/private cert/key. Switch out both to sign the serving certs with a different CA.

make sense, just checking. I wish there was a sort of 'bundled form' but that's probably bad practice somehow because you want to expose one and not the other. Better that they are different files.

yeah, it's like public and private SSH keys -- you (generally) wouldn't want them bundled together, because one shouldn't be passed around ;-)

MHBauer · 2017-06-08T20:11:18Z

failure is probably the missing header for my 'script'

Okay, I was under the misconception that the CA had to be the core CA, and so now the instructions have two paths.

create your own CA stuff and use that
pull the kube ca out and base on it.

I scriptified the first flow, but you need cfssl tools to run it. Not bulletproof. The 'user-configurable' parts are the variables HELM_NAME & SVCCAT_NAMESPACE. Everything else is generated from there and the final run of the actual helm install relies on new variables made during the run.

pmorie · 2017-06-08T21:12:44Z

@MHBauer travis failure is legit:

Running repo-infra verify scripts

Boilerplate header is wrong for: ././contrib/api-setup.sh

make: *** [verify] Error 1

MHBauer · 2017-06-08T22:24:21Z

charts/catalog/templates/apiregistration.yaml

@@ -1,5 +1,5 @@
 {{- if .Values.useAggregator }}
-apiVersion: apiregistration.k8s.io/v1alpha1
+apiVersion: apiregistration.k8s.io/v1beta1


@arschles @krancour is there some helm trick we can pull for checking for kubernetes v1.6 vs v1.7 and setting to alpha or beta?

@MHBauer I don't know of a trick, but it would have to but we can ask the user to specify it in another helm value.

Also, you could ask in the Kubernetes Slack #helm-dev channel to see if there's a trick

At this point, I am most interested in the 1.7 use-case. I would rather just keep this PR focused on 1.7

MHBauer · 2017-06-09T00:59:23Z

contrib/api-setup.sh

+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.


@pmorie what's wrong with the boilerplate now?

your shebang has no bang, it looks like?

pmorie

Looks like a promising start - I'd prefer to merge only once it has complete sentences. Left a couple suggestions

pmorie · 2017-06-09T17:23:36Z

contrib/api-setup.sh

+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.


your shebang has no bang, it looks like?

pmorie · 2017-06-09T17:37:20Z

contrib/api-setup.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export HELM_NAME=catalog


We should probably make a note that catalog is currently expected as the namespace.

Related: #926

pmorie · 2017-06-09T17:38:08Z

docs/api-aggregation-setup.md

@@ -0,0 +1,198 @@
+
+# generic aggregator information here


'Setting up the Kubernetes API aggregator'

pmorie · 2017-06-09T17:38:38Z

docs/api-aggregation-setup.md

+For background on why we're messing with certificates and CA's at all,
+please check [the auth doc](auth.md).
+
+# steps


Capitalize headers consistently, plz :)

pmorie · 2017-06-09T17:39:04Z

docs/api-aggregation-setup.md

+# steps
+
+## Prerequisites
+Need cfssl tools, install with `go get -u github.com/cloudflare/cfssl/cmd/...`.


pmorie · 2017-06-09T17:41:46Z

docs/api-aggregation-setup.md

+cp /var/run/kubernetes/${SERVINGCA_CERT} .
+```
+
+### TODO explain what we're setting up 


Are you planning to address these TODOs in this PR?

MHBauer · 2017-06-09T19:41:10Z

I took another pass through the doc. Should be closer to ready.

Removed the RBAC info to cover it in a separate document.

MHBauer · 2017-06-12T18:17:00Z

resolves #935 instead of #880

bradtopol · 2017-06-12T21:23:34Z

docs/api-aggregation-setup.md

+
+# Setting up Service Catalog for API Aggregation in Kubernetes
+
+The aggregator is a server that sits in front of everything. 


Might be nice to mention that what you mean is the aggregator sits in front of the apiserver

Also would be helpful if you had 1-2 sentences that motivate the need for an aggregator. What problem does having an aggregator solve?

bradtopol

Made some doc suggestions that might help folks to understand why an aggregator would benefit them. Great job!

duglin · 2017-06-12T23:34:52Z

docs/api-aggregation-setup.md

+`apiregistration.k8s.io/v1alpha1` if running in kubernetes v1.6)
+
+This won't show up if you are talking to the insecure port. The
+insecure port is not behind the aggregator.


I'd prefer to make this a little more concrete since it could imply that we chose to not put it behind the aggregator... how about As of now the aggregator only supports routing of requests sent to the secure port or something like that. Make it clear that its not our choice/config-option - its the design of the aggregator.

It's as concrete as is possible. It's a statement of fact.

duglin · 2017-06-12T23:38:28Z

docs/api-aggregation-setup.md

+echo '{"CN":"'${SERVICE_NAME}'","hosts":['${ALT_NAMES}'],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=${SERVINGCA_CERT} -ca-key=${SERVINGCA_KEY} -config=server-ca-config.json - | cfssljson -bare apiserver
+```
+
+### put the output we need for running our apiserver behind the aggregator


s/put/Put/ for consistency

duglin · 2017-06-12T23:40:40Z

docs/api-aggregation-setup.md

+## Install the Service Catalog with Helm Charts
+
+
+Use helm to install the Service Catalog into the configured name and


"into the configured name" sounds odd- almost like "name" is a resource. Did you mean "Use helm to install the Service Catalog, associating it with the configured name ${HELM_NAME}, and into the specified namespace." ?

duglin · 2017-06-12T23:41:55Z

docs/api-aggregation-setup.md

+
+If it doesn't show up the kubectl discovery cache is stale and needs
+to be deleted. It may be located in the `.kube` directory,
+approximately `.kube/cache/discovery/`.


perhaps add "$HOME" before ".kube/..." ?

duglin · 2017-06-12T23:42:27Z

docs/api-aggregation-setup.md

+to be deleted. It may be located in the `.kube` directory,
+approximately `.kube/cache/discovery/`.
+
+Now e2e should work with the same kubeconfig set for both the core


s/e2e/service catalogs's e2e/

MHBauer · 2017-06-12T23:54:17Z

docs/api-aggregation-setup.md

+helm install /root/go/src/github.com/kubernetes-incubator/service-catalog/charts/catalog \
+    --name ${HELM_NAME} --namespace ${SVCCAT_NAMESPACE} \
+    --set apiserver.auth.enabled=true \
+        --set useAggregator=true \


todo: check if this is only needed for rbac or otherwise what it does.

pmorie

Overall LGTM; I went through these instructions myself and got things working. Left a couple nits/suggestions.

pmorie · 2017-06-13T12:55:13Z

charts/catalog/templates/apiregistration.yaml

@@ -1,5 +1,5 @@
 {{- if .Values.useAggregator }}
-apiVersion: apiregistration.k8s.io/v1alpha1
+apiVersion: apiregistration.k8s.io/v1beta1


At this point, I am most interested in the 1.7 use-case. I would rather just keep this PR focused on 1.7

pmorie · 2017-06-13T12:55:46Z

contrib/api-setup.sh

@@ -0,0 +1,61 @@
+#!/bin/sh --


extremely minor nit: can we call this script apiserver-ssl-setup.sh ?

Also, I don't think this script is referenced from your write-up. Is that intentional?

I can call it whatever you want.

I pulled the script out at the last minute. Probably ought to mention it.

pmorie · 2017-06-13T13:08:50Z

docs/api-aggregation-setup.md

+
+### Create our own new CA and generate keys
+
+This is an example. It is written with zero understanding of the best


I would actually like to leave this comment in, and I think @MHBauer is right to advertise that this might not be good-practice-compliant.

pmorie · 2017-06-13T13:09:51Z

docs/api-aggregation-setup.md

+various files may be named differently and in different locations.
+
+
+```


I'm running through these instructions now.

I think it would be good to clarify what directory it's expected that these commands should be executed in.

pmorie · 2017-06-13T14:57:55Z

docs/api-aggregation-setup.md

+```
+
+### Use the existing keys plus the config file to generate the new signing certificate and key
+


I would note here that you must install the catalog into the right namespace for the certificate to be signed correctly for the service's DNS name.

Yea, I think I wrote that down earlier, but it doesn't seem super clear that all of these things have to match and why.

pmorie · 2017-06-13T17:42:02Z

docs/api-aggregation-setup.md

+export SC_SERVING_CA=${SVCCAT_CA_CERT}
+export SC_SERVING_CERT=apiserver.pem
+export SC_SERVING_KEY=apiserver-key.pem
+```


Nit: I think if you go this route the APIService resource needs to set InsecureSkipTLSVerify since the CA will necessarily not be in the aggregator's trust store.

I don't understand what you want me to do. I haven't encountered any problems with the script or setup as it is, but I do want to cover as wide of cases as possible.

I think that for now, what you have is actually fine. I have a specific thing that I realized is more for when we do this in the walkthrough

pmorie

Created kubernetes/kubectl#31 for an issue I found when I messed up the namespace running through these instruction

pmorie · 2017-06-14T04:33:08Z

docs/api-aggregation-setup.md

+
+### Create our own new CA and generate keys
+
+This is an example. It is written with zero understanding of the best


That should not block this PR being merged. I have no clue about best practices for SSL certs and this moves things in the right direction.

MHBauer · 2017-06-14T06:40:58Z

Updated and squashed.

MHBauer · 2017-06-19T21:45:16Z

charts/catalog/templates/apiregistration.yaml

@@ -1,5 +1,9 @@
 {{- if .Values.useAggregator }}
+{{- if .Capabilities.APIVersions.Has "apiregistration.k8s.io/v1beta1" }}
+apiVersion: apiregistration.k8s.io/v1beta1
+{{- else if .Capabilities.APIVersions.Has "apiregistration.k8s.io/v1alpha1" }}


This is ugly but it works.

arschles · 2017-06-19T22:55:35Z

docs/api-aggregation-setup.md

+and provides the keys we just generated inline.
+
+```
+helm install /root/go/src/github.com/kubernetes-incubator/service-catalog/charts/catalog \


let's use a relative path here, assuming the user is executing from the root of the repository (./charts/catalog)

Sure. Last vestiges of this being a script that I ran myself on my machine.

@arschles any thoughts on using ${GOPATH} as reference? Probably best to still just pretend we're in the project root.

MHBauer · 2017-06-19T22:55:52Z

@kibbles-n-bytes tiller didn't come up. How do I restart jenkins?

arschles · 2017-06-19T23:00:05Z

docs/api-aggregation-setup.md

+    --name ${HELM_NAME} --namespace ${SVCCAT_NAMESPACE} \
+    --set apiserver.auth.enabled=true \
+        --set useAggregator=true \
+        --set apiserver.tls.ca=$(base64 --wrap 0 ${SC_SERVING_CA}) \


@MHBauer there is no ca field in the values.yaml file under apiserver.tls. Can you add one with an empty value and a comment to describe what it is?

@DirectXMan12 any suggestions here? We use it in charts/catalog/templates/apiregistration.yaml

I'm thinking

# Base64-encoded certificate authority for the APIService object to register # the Service Catalog API with the apiregistration

@arschles Not sure what to use as a value. Should I generate a new set of matching stuff and replace everything tls related? I'm not sure what's in there now and don't want to break everything.

duglin · 2017-06-20T01:56:50Z

contrib/svc-cat-apiserver-aggregation-tls-setup.sh

@@ -0,0 +1,61 @@
+#!/bin/sh --


I'd prefer to remove the -- since it'll cause people to wonder what the heck we're up to when it serves no purpose what so ever. Unless we want to just mess with people's heads, then I'm all for it! ;-)

duglin · 2017-06-20T01:58:45Z

docs/api-aggregation-setup.md

+`source`ed to define all of the variables it contains in the current
+shell process.
+
+The aggregator is a new feature of kubernetes and is an alpha API


missing server after API. I'd move this paragraph up to the top where brad made his comments.

it's an api not a server.

@DirectXMan12

- A script to setup the certificates needed for API Aggregations - Change to the apiregistration as it is now a beta API - An informational file to describe what we are doing and why I wish to thank these people - @DirectXMan12 for the auth doc, and walking me through cert generation - @liggitt for help with troubleshooting RBAC and a way to disable RBAC - @ivan4th for Mirantis/kubeadm-dind-cluster

MHBauer · 2017-06-20T02:08:43Z

Lost some content from an earlier rebase. Only documentation changes. Ought to still be green.

arschles · 2017-06-20T16:25:48Z

charts/catalog/values.yaml

@@ -9,6 +9,9 @@ apiserver:
  # are some outstanding problems with the TLS-secured endpoint
  insecure: true
  tls:
+    # Base64-encoded certificate authority for the APIService object to register
+    # the Service Catalog API with the apiregistration
+    ca: ""


arschles

I haven't had the chance to thoroughly test this, but it's a great start and I would prefer to get it merged now and for us to build on this foundation.

Also, I'd like to reiterate how excited and happy I am for us to fully utilize the aggregator. Thanks for getting us started @MHBauer

LGTM

MHBauer · 2017-06-20T16:39:25Z

#952 to follow up on the additional CI work needed.

MHBauer added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. in-progress labels May 25, 2017

MHBauer requested a review from DirectXMan12 May 25, 2017 01:18

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 25, 2017

MHBauer changed the title ~~the short version~~ api-aggregation setup instructions May 25, 2017

pmorie added this to the 0.1.0 milestone May 27, 2017

DirectXMan12 reviewed Jun 7, 2017

View reviewed changes

MHBauer commented Jun 7, 2017

View reviewed changes

MHBauer commented Jun 8, 2017

View reviewed changes

MHBauer commented Jun 9, 2017

View reviewed changes

MHBauer removed help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. in-progress labels Jun 9, 2017

pmorie reviewed Jun 9, 2017

View reviewed changes

MHBauer requested review from arschles, DirectXMan12, jpeeler, duglin and vaikas June 12, 2017 20:56

bradtopol reviewed Jun 12, 2017

View reviewed changes

duglin reviewed Jun 12, 2017

View reviewed changes

MHBauer commented Jun 12, 2017

View reviewed changes

pmorie reviewed Jun 13, 2017

View reviewed changes

pmorie modified the milestones: 0.0.11, 0.1.0 Jun 13, 2017

pmorie reviewed Jun 14, 2017

View reviewed changes

MHBauer commented Jun 19, 2017

View reviewed changes

MHBauer mentioned this pull request Jun 19, 2017

RBAC setup behind the aggregator. #936

Merged

arschles reviewed Jun 19, 2017

View reviewed changes

duglin reviewed Jun 20, 2017

View reviewed changes

pmorie added the LGTM1 label Jun 20, 2017

arschles reviewed Jun 20, 2017

View reviewed changes

arschles approved these changes Jun 20, 2017

View reviewed changes

arschles added the LGTM2 label Jun 20, 2017

vaikas added the LGTM3 label Jun 20, 2017

vaikas merged commit 9f743b2 into kubernetes-retired:master Jun 20, 2017

MHBauer deleted the api-aggregation-setup branch June 21, 2017 00:04

MHBauer mentioned this pull request Jun 21, 2017

Support running in aggregated mode using apiserver-boot run --aggregated kubernetes-sigs/apiserver-builder-alpha#44

Closed

MHBauer mentioned this pull request Jul 19, 2017

document api aggregation setup #935

Closed


		# Setting up Service Catalog for API Aggregation in Kubernetes

		The aggregator is a server that sits in front of everything.

		## Install the Service Catalog with Helm Charts


		Use helm to install the Service Catalog into the configured name and


		### Create our own new CA and generate keys

		This is an example. It is written with zero understanding of the best

		various files may be named differently and in different locations.


		```

		```

		### Use the existing keys plus the config file to generate the new signing certificate and key

api-aggregation setup instructions #895

api-aggregation setup instructions #895

Conversation

MHBauer commented May 25, 2017

MHBauer commented May 25, 2017 • edited Loading

derekwaynecarr commented May 25, 2017

deads2k commented May 25, 2017

MHBauer commented May 26, 2017

deads2k commented May 26, 2017

DirectXMan12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer commented Jun 8, 2017

pmorie commented Jun 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer commented Jun 9, 2017

MHBauer commented Jun 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bradtopol left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer commented Jun 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer commented Jun 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer commented Jun 20, 2017

Choose a reason for hiding this comment

MHBauer commented May 25, 2017 •

edited

Loading

arschles left a comment •

edited

Loading