Master nodes behind AWS ELB and certificate manager

[pandeybk]

Use Case: Use AWS certificate manager (https://aws.amazon.com/certificate-manager/) as certificate authority for master node

"AWS Certificate Manager removes the time-consuming manual process of purchasing, uploading, and renewing SSL/TLS certificates"

As per current implementation, KOPS create self signed certificate and create A-Record entry sets in route53 for kubernetes api.

I wonder If we can add our masters nodes behind AWS ELB and assigned certificate using aws certificate manager instead of self signed certificate.

Looping : @justinsb

[justinsb]

My understanding is that the requirement is for a trusted certificate, and that elb is just a suggested means to achieve that

[zytek]

Configuring ELB with valid cert from certificate manager requires manual request validation (email sent to domain admin with auth link), no idea if this can be automated but it would definitely improve user experience.

[chrislovecnm]

rather than introducing aws cert manager support specifically I believe we need to look at a plugin model for certs. For instance people will have use cases for vault. That is a bit of a refactor, but has already been discussed.

[gopinatht]

Guys, is this still under active consideration for 1.5.1? Any dev effort already underway?

[chrislovecnm]

@gopinatht under consideration, would be awesome, you able to help?

[gopinatht]

@chrislovecnm willing to. What can I start with? Any documentation around plugin models you mention?

[chrislovecnm]

@gopinatht the plugin model is @kris-nova's idea. I will let her provide more details.

[vjm]

@chrislovecnm @kris-nova any updates on this?

[saracen]

If moving to some kind of plugin model, I wouldn't mind helping out with the implementation of it to provision certificates from Vault.

[vjm]

This is still useful for us too, would love to help out!

[nhumrich]

@zytek kops doesnt need to create the cert. It could use an already existing cert in the cert manager, such as the "bring your own vpc" option.

[chrislovecnm]

@nhumrich we have an issue open to provide that, but at this point, it has not been implemented. Cert generation for k8s is very very complicated, so we would also need a mechanism to validate that the certs would work as well.

If anyone wants to focus on this, please reach out via GitHub or slack.

[thegranddesign]

Can someone explain to me why it's simply not possible to have the cert terminate at the ELB and have the internal ELB port be 8080 and have all communication within the private subnets be on 8080?

There's no way for external traffic to get into the subnet other than through the ELB, which is forced to only be sent to the masters, so therefore requiring SSL between the nodes is not providing any benefit that I can see.

Are there any configuration options to do this?

As another PS, everyone seems to be talking about "cert generation" via kops with ACM. I personally don't care about this at all. Creating the cert manually is not a big deal. The "go small" would be:

• Add the ACM cert ARN to the config
• Use it
• Done

[thegranddesign]

For the record, I don't really care how the nodes inter-communicate. What I care about is being able to use ACM on my ELB and not getting a: Unable to connect to the server: x509: certificate signed by unknown authority error when using kubectl

[brutog]

+1 for a configurable certificate for the ELB. I also don't care as much about the inter-cluster communications. Even if those remain self-signed (and generally unexposed), I would still like the API and kubectl to use a real certificate chain. Whether this gets put into the api itself, or terminated on the ELB, probably doesn't matter to me.

Thanks.

[mtolan]

This is a better idea in principle than in practice.

While enabling SSL termination at the ELB in front of the masters would mostly allow the use of an ACM managed certificate to secure communications to your kube api, the limitations of Elastic Load Balancers creep up on you over time.

The first thing that breaks is the ability to use kubectl exec on any pods in your cluster. The reason for this is that the communication between kubectl and the k8s API relies on SPDY, which ELBs notably don't support. The end result of this is that requests to exec result in HTTP 400 responses.

$kubectl exec -it mypod /bin/bash
Error from server (BadRequest): Upgrade request required

[thegranddesign]

@mtolan If you have a cluster you should have a bastion host in front of it that SSH proxies your commands. You shouldn't be doing any kubectl commands through your ELB.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[thegranddesign]

/remove-lifecycle stale

[erks]

@mtolan i was able to get kubectl exec to work with ELB + ACM by setting the LB and instance protocols to ssl instead of https.

[sstarcher]

It should also work with ELB+ACM if the inbound is SSL and the outbound is also SSL

[Raffo]

As discussed in the kops office hours, I'll take a shot at identifying what is needed to implement this feature and create a PR after that.

[sstarcher]

@Raffo Thanks alot