Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2018-1002105: proxy request handling in kube-apiserver can leave vulnerable TCP connections #71411

Closed
liggitt opened this Issue Nov 26, 2018 · 49 comments

Comments

Projects
None yet
@liggitt
Copy link
Member

liggitt commented Nov 26, 2018

CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H (9.8, critical)

With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection.

Thanks to Darren Shepherd for reporting this problem.

CVE-2018-1002105 is fixed in the following Kubernetes releases:

Affected components:

  • Kubernetes API server

Affected versions:

  • Kubernetes v1.0.x-1.9.x
  • Kubernetes v1.10.0-1.10.10 (fixed in v1.10.11)
  • Kubernetes v1.11.0-1.11.4 (fixed in v1.11.5)
  • Kubernetes v1.12.0-1.12.2 (fixed in v1.12.3)

Note: If you are using binaries or packages provided by a distributor (not the ones provided in the open source release artifacts), you should contact them to determine what versions resolve this CVE. Distributors may choose to provide support for older releases beyond the ones maintained by the open source project.

Affected configurations:

  • Clusters >= 1.6.x that run aggregated API servers (like the metrics server) that are directly accessible from the Kubernetes API server’s network. If there are aggregated API servers configured in a cluster, the following command will return the names of the associated APIService objects (if no names are listed, or the kube-apiserver is an older version that does not have the apiservices API, then the cluster has no aggregated API servers configured):
    kubectl get apiservices \
      -o 'jsonpath={range .items[?(@.spec.service.name!="")]}{.metadata.name}{"\n"}{end}'
    
  • Clusters >= 1.0.x that grant pod exec/attach/portforward permissions to users that are not expected to have full access to kubelet APIs

Vulnerability impact:

  • An API call to any aggregated API server endpoint can be escalated to perform any API request against that aggregated API server, as long as that aggregated API server is directly accessible from the Kubernetes API server’s network. Default RBAC policy allows all users (authenticated and unauthenticated) to perform discovery API calls that allow this escalation against any aggregated API servers configured in the cluster.
  • A pod exec/attach/portforward API call can be escalated to perform any API request against the kubelet API on the node specified in the pod spec (e.g. listing all pods on the node, running arbitrary commands inside those pods, and obtaining the command output). Pod exec/attach/portforward permissions are included in the admin/edit RBAC roles intended for namespace-constrained users.

Mitigations:

This section lists possible mitigations to use prior to upgrading. Note that many of the mitigations are likely to be disruptive, and upgrading to a fixed version is strongly recommended.

Mitigations for the anonymous user -> aggregated API server escalation include:

  • suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server)
  • disable anonymous requests by passing --anonymous-auth=false to the kube-apiserver (note that this may disrupt load balancer or kubelet health checks of the kube-apiserver, and breaks kubeadm join setup flows)
  • remove all anonymous access to all aggregated APIs (including discovery permissions granted by the default discovery role bindings)

Mitigations for the authenticated user -> aggregated API server escalation include:

  • suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server)
  • remove all access to all aggregated APIs (including discovery permissions granted by the default discovery role bindings) from users that should not have full access to the aggregated APIs (note that this may disrupt users and controllers that make use of discovery information to map API types to URLs)

Mitigation for the authorized pod exec/attach/portforward -> kubelet API escalation:

  • Remove pod exec/attach/portforward permissions from users that should not have full access to the kubelet API

Detection:

There is no simple way to detect whether this vulnerability has been used. Because the unauthorized requests are made over an established connection, they do not appear in the Kubernetes API server audit logs or server log. The requests do appear in the kubelet or aggregated API server logs, but are indistinguishable from correctly authorized and proxied requests via the Kubernetes API server.

Post-mortem:

@liggitt liggitt added the needs-sig label Nov 26, 2018

@liggitt liggitt changed the title Proxy connection gets stuck on error response CVE-2018-1002105: proxy request handling in kube-apiserver can leave vulnerable TCP connections Dec 3, 2018

teemow added a commit to giantswarm/retagger that referenced this issue Dec 3, 2018

update hyperkube images
CVE-2018-1002105: proxy request handling in kube-apiserver can leave
vulnerable TCP connections

kubernetes/kubernetes#71411

teemow added a commit to giantswarm/retagger that referenced this issue Dec 3, 2018

update hyperkube images
CVE-2018-1002105: proxy request handling in kube-apiserver can leave
vulnerable TCP connections

kubernetes/kubernetes#71411

teemow added a commit to giantswarm/retagger that referenced this issue Dec 3, 2018

update hyperkube images (#159)
CVE-2018-1002105: proxy request handling in kube-apiserver can leave
vulnerable TCP connections

kubernetes/kubernetes#71411

teemow added a commit to giantswarm/k8scloudconfig that referenced this issue Dec 3, 2018

upgrade hyperkube images
CVE-2018-1002105: proxy request handling in kube-apiserver can leave
vulnerable TCP connections

kubernetes/kubernetes#71411

teemow added a commit to giantswarm/k8scloudconfig that referenced this issue Dec 3, 2018

upgrade hyperkube images (#470)
CVE-2018-1002105: proxy request handling in kube-apiserver can leave
vulnerable TCP connections

kubernetes/kubernetes#71411
@redbaron

This comment has been minimized.

Copy link
Contributor

redbaron commented Dec 3, 2018

@liggitt , why doesn't kubelet's webhook authorization help here? Shouldn't it reject subsequent requests on still open TCP connection from APIServer?

@liggitt

This comment has been minimized.

Copy link
Member Author

liggitt commented Dec 3, 2018

@liggitt , why doesn't kubelet's webhook authorization reject subsequent requests on still open TCP connection from APIServer?

Because the kube-apiserver <-> kubelet connection was established with the kube-apiserver's TLS credentials, which are broadly authorized against the kubelet API. The kubelet would authorize the kube-apiserver to make that request, and allow it.

@liggitt

This comment has been minimized.

Copy link
Member Author

liggitt commented Dec 20, 2018

Updated the description with a link to the document and recording of the post-mortem for this incident. Links to action items are in the document and reference this issue.

Post-mortem:

brengarajalu added a commit to brengarajalu/kubernetes that referenced this issue Jan 5, 2019

Merge pull request #1 from brengarajalu/CVE-2018-1002105
backport of pr#71415 fixing CVE-2018-1002105 kubernetes#71411

brengarajalu pushed a commit to brengarajalu/kubernetes that referenced this issue Jan 5, 2019

brengarajalu added a commit to brengarajalu/kubernetes that referenced this issue Jan 5, 2019

brengarajalu added a commit to brengarajalu/kubernetes that referenced this issue Jan 5, 2019

flaper87 added a commit to flaper87/cluster-api-provider-openstack that referenced this issue Jan 9, 2019

Update examples to use 1.12.3 (kubernetes-sigs#123)
Mitigation of CVE-2018-1002105
See kubernetes/kubernetes#71411 for more details

@liggitt liggitt added area/security and removed area/security labels Mar 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.