New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impossible to completely eliminate insecure access to apiserver #13598

Closed
eparis opened this Issue Sep 4, 2015 · 18 comments

Comments

Projects
None yet
@eparis
Member

eparis commented Sep 4, 2015

The apiserver listens, by default, on 127.0.01:8080 in a completely insecure manor. Even if the scheduler, controller-manager, proxies, kubelets, etc are all configured to use the secure port the apiserver still talks to itself on the insecure port.

https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go#L340

This is how things like the admission controllers (part of the apiserver) talk to other parts of the apiserver. The problem with having the apiserver talk to itself over the insecure port is that means we cannot block the insecure port from everything else on the host. Any user able to log into the machine running the api-server can connect to the insecure port and do anything they want. Blocking all access to 8080 with iptables means admission controllers and such stop working.

There may be many solutions, but the basic problem is I want to turn off all insecure access to the apiserver.

@liggitt suggested some form of privileged kubeconfig the apiserver would use to talk to itself

It might also be possible to create a unix domain socket pair which only the apiserver could use to talk to itself in an 'insecure' manor similar to how it is currently using the insecure socket, but which everyone else on the host couldn't use.

@gmarek

This comment has been minimized.

Show comment
Hide comment
@gmarek

gmarek Sep 9, 2015

Member

The comment for --insecure-access flag says:

The port on which to serve unsecured, unauthenticated access. Default 8080. It is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. This is performed by nginx in the default setup.

Do we want to do something with this? @thockin

Member

gmarek commented Sep 9, 2015

The comment for --insecure-access flag says:

The port on which to serve unsecured, unauthenticated access. Default 8080. It is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. This is performed by nginx in the default setup.

Do we want to do something with this? @thockin

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Sep 9, 2015

Member

You can disable with --insecure-access=0, but it breaks admission controllers (at least). At minimum, need to plumb client config for the admission controllers to use (similar to the way you tell the controller manager what client config to use for the controllers)

Member

liggitt commented Sep 9, 2015

You can disable with --insecure-access=0, but it breaks admission controllers (at least). At minimum, need to plumb client config for the admission controllers to use (similar to the way you tell the controller manager what client config to use for the controllers)

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Sep 9, 2015

Member

It would be great to turn this off, but it's outside my area of expertise :)

On Wed, Sep 9, 2015 at 5:08 AM, Marek Grabowski notifications@github.com
wrote:

The comment for --insecure-access flag says:

The port on which to serve unsecured, unauthenticated access. Default 8080. It is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. This is performed by nginx in the default setup.

Do we want to do something with this? @thockin
https://github.com/thockin


Reply to this email directly or view it on GitHub
#13598 (comment)
.

Member

thockin commented Sep 9, 2015

It would be great to turn this off, but it's outside my area of expertise :)

On Wed, Sep 9, 2015 at 5:08 AM, Marek Grabowski notifications@github.com
wrote:

The comment for --insecure-access flag says:

The port on which to serve unsecured, unauthenticated access. Default 8080. It is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. This is performed by nginx in the default setup.

Do we want to do something with this? @thockin
https://github.com/thockin


Reply to this email directly or view it on GitHub
#13598 (comment)
.

@roberthbailey

This comment has been minimized.

Show comment
Hide comment
@roberthbailey

roberthbailey Sep 10, 2015

Member

@gmarek We should update the comment that talks about nginx, since we removed it quite a while ago.

Member

roberthbailey commented Sep 10, 2015

@gmarek We should update the comment that talks about nginx, since we removed it quite a while ago.

@lavalamp

This comment has been minimized.

Show comment
Hide comment
@lavalamp

lavalamp Oct 8, 2015

Member

UGH.

Stuff inside the apiserver shouldn't do this. Making a client pointing at yourself is a little nuts. Have to think about how to fix this.

Member

lavalamp commented Oct 8, 2015

UGH.

Stuff inside the apiserver shouldn't do this. Making a client pointing at yourself is a little nuts. Have to think about how to fix this.

@gmarek

This comment has been minimized.

Show comment
Hide comment
@gmarek
Member

gmarek commented Oct 9, 2015

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Mar 18, 2016

Member

Would really like to see this fixed for 1.3

Member

liggitt commented Mar 18, 2016

Would really like to see this fixed for 1.3

@liggitt liggitt added this to the next-candidate milestone Mar 18, 2016

@roberthbailey

This comment has been minimized.

Show comment
Hide comment
@roberthbailey

roberthbailey Mar 18, 2016

Member

Me too. Are you guys going to have any bandwidth to work on it?

Member

roberthbailey commented Mar 18, 2016

Me too. Are you guys going to have any bandwidth to work on it?

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Mar 18, 2016

Member

Stuff inside the apiserver shouldn't do this. Making a client pointing at yourself is a little nuts

I like how you can make sure all the same paths apply (authnz, admission, etc). I like the ability to give different components different permissions.

@smarterclayton was experimenting with a go library that let you do "loopback" calls via a channel so that you wouldn't pay network stack overhead, but could treat the rest of the handler chain consistently

Member

liggitt commented Mar 18, 2016

Stuff inside the apiserver shouldn't do this. Making a client pointing at yourself is a little nuts

I like how you can make sure all the same paths apply (authnz, admission, etc). I like the ability to give different components different permissions.

@smarterclayton was experimenting with a go library that let you do "loopback" calls via a channel so that you wouldn't pay network stack overhead, but could treat the rest of the handler chain consistently

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Mar 18, 2016

Contributor

The network stack overhead isn't that bad on modern kernels. Even the
loopback channels (emulated net.Conn) weren't substantially faster,
although tail latency was much better.

On Thu, Mar 17, 2016 at 10:36 PM, Jordan Liggitt notifications@github.com
wrote:

Stuff inside the apiserver shouldn't do this. Making a client pointing at
yourself is a little nuts

I like how you can make sure all the same paths apply (authnz, admission,
etc). I like the ability to give different components different
permissions.

@smarterclayton https://github.com/smarterclayton was experimenting
with a go library that let you do "loopback" calls via a channel so that
you wouldn't pay network stack overhead, but could treat the rest of the
handler chain consistently


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13598 (comment)

Contributor

smarterclayton commented Mar 18, 2016

The network stack overhead isn't that bad on modern kernels. Even the
loopback channels (emulated net.Conn) weren't substantially faster,
although tail latency was much better.

On Thu, Mar 17, 2016 at 10:36 PM, Jordan Liggitt notifications@github.com
wrote:

Stuff inside the apiserver shouldn't do this. Making a client pointing at
yourself is a little nuts

I like how you can make sure all the same paths apply (authnz, admission,
etc). I like the ability to give different components different
permissions.

@smarterclayton https://github.com/smarterclayton was experimenting
with a go library that let you do "loopback" calls via a channel so that
you wouldn't pay network stack overhead, but could treat the rest of the
handler chain consistently


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#13598 (comment)

@discordianfish

This comment has been minimized.

Show comment
Hide comment
@discordianfish

discordianfish Jul 20, 2016

Contributor

I'm surprised that this doesn't get address with higher priority.

For the sake of a safer world, can you please at least require token or basic auth on http endpoints?

Currently this behavior by default opens what is effectively a root shell to anyone who can reach 127.0.0.1. That could be containers if the apiserver runs on systems with regular workloads but also local, unprivileged users or remote attackers exploiting a https://httpoxy.org/ like vulnerability. Or use DNS-rebinding to trigger api calls etc.

Contributor

discordianfish commented Jul 20, 2016

I'm surprised that this doesn't get address with higher priority.

For the sake of a safer world, can you please at least require token or basic auth on http endpoints?

Currently this behavior by default opens what is effectively a root shell to anyone who can reach 127.0.0.1. That could be containers if the apiserver runs on systems with regular workloads but also local, unprivileged users or remote attackers exploiting a https://httpoxy.org/ like vulnerability. Or use DNS-rebinding to trigger api calls etc.

@k3a

This comment has been minimized.

Show comment
Hide comment
@k3a

k3a Aug 18, 2016

Yes this really needs to be addressed

k3a commented Aug 18, 2016

Yes this really needs to be addressed

@wied03

This comment has been minimized.

Show comment
Hide comment
@wied03

wied03 Aug 19, 2016

This may be already addressed above, but I also see no way to make `kube-controller-manager`` work with the API Server in SSL only mode unless the API Server's cert is signed by a trusted CA. I don't know enough about Go off the top of my head to know if you can customize that like Java without changing code.

wied03 commented Aug 19, 2016

This may be already addressed above, but I also see no way to make `kube-controller-manager`` work with the API Server in SSL only mode unless the API Server's cert is signed by a trusted CA. I don't know enough about Go off the top of my head to know if you can customize that like Java without changing code.

@wied03

This comment has been minimized.

Show comment
Hide comment
@wied03

wied03 Aug 19, 2016

Unless the kubeconfig approach discussed here can work with the controller-manager, and it looks like it does

wied03 commented Aug 19, 2016

Unless the kubeconfig approach discussed here can work with the controller-manager, and it looks like it does

@k3a

This comment has been minimized.

Show comment
Hide comment
@k3a

k3a Aug 19, 2016

No, it doesn't seem to be fixed in v1.3.5 release. The problem are admission controllers inside kube-apiserver. They seem to communicate with api-server over insecure connection only.

1 . I have apiserver run with: --insecure-port=0
2. Every kubernetes process use --kubeconfig and --master with secure ports specified
3. Yet when I run the apiserver, it logs out:

reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://127.0.0.1:0/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 127.0.0.1:0: getsockopt: connection refused

Always tries insecure bind + insecure port. While it is mostly fine to use insecure port on machines dedicated for kubernetes only, it makes admission controllers unusable at the moment in environments where other processes are running next to the kubernetes processes for security reasons (I don't want anyone with access to the local machine as a regular user to use 127.0.0.1:8080 to manage the kube master).

k3a commented Aug 19, 2016

No, it doesn't seem to be fixed in v1.3.5 release. The problem are admission controllers inside kube-apiserver. They seem to communicate with api-server over insecure connection only.

1 . I have apiserver run with: --insecure-port=0
2. Every kubernetes process use --kubeconfig and --master with secure ports specified
3. Yet when I run the apiserver, it logs out:

reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://127.0.0.1:0/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 127.0.0.1:0: getsockopt: connection refused

Always tries insecure bind + insecure port. While it is mostly fine to use insecure port on machines dedicated for kubernetes only, it makes admission controllers unusable at the moment in environments where other processes are running next to the kubernetes processes for security reasons (I don't want anyone with access to the local machine as a regular user to use 127.0.0.1:8080 to manage the kube master).

@du2016

This comment has been minimized.

Show comment
Hide comment
@du2016

du2016 Aug 23, 2016

Contributor

yes ,my k8s version is v1.3.3,in my log of apiserver: k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://10.10.10.150:0/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 10.10.10.150:0: getsockopt: connection refused

Contributor

du2016 commented Aug 23, 2016

yes ,my k8s version is v1.3.3,in my log of apiserver: k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://10.10.10.150:0/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 10.10.10.150:0: getsockopt: connection refused

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598
@dims

This comment has been minimized.

Show comment
Hide comment
@dims

dims Aug 26, 2016

Member

Am not sure if the approach i came up with in #31491 is acceptable or not. PTAL

Member

dims commented Aug 26, 2016

Am not sure if the approach i came up with in #31491 is acceptable or not. PTAL

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 26, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 27, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 27, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 27, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 27, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Aug 28, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 18, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 18, 2016

[WIP] Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 18, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 18, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 19, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

dims added a commit to dims/kubernetes that referenced this issue Sep 20, 2016

Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598

k8s-merge-robot added a commit that referenced this issue Sep 22, 2016

Merge pull request #31491 from dims/fixes-issue-13598
Automatic merge from submit-queue

Allow secure access to apiserver from Admission Controllers

* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
  if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
  to talk to the api server
* Add a new authenticator that checks for this specific bearer token

Fixes #13598
@discordianfish

This comment has been minimized.

Show comment
Hide comment
@discordianfish

discordianfish Sep 22, 2016

Contributor

@dims Awesome! Thanks for this!

Contributor

discordianfish commented Sep 22, 2016

@dims Awesome! Thanks for this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment