Support pod ip and service cluster ip lookups in the destination service

[adleong]

Fixes #3444
Fixes #3443

Background and Behavior

This change adds support for the destination service to resolve Get requests which contain a service clusterIP or pod ip as the Path parameter. It returns the stream of endpoints, just as if Get had been called with the service's authority. This lays the groundwork for allowing the proxy to TLS TCP connections by allowing the proxy to do destination lookups for the SO_ORIG_DST of tcp connections. When that ip address corresponds to a service cluster ip or pod ip, the destination service will return the endpoints stream, including the pod metadata required to establish identity.

Prior to this change, attempting to look up an ip address in the destination service would result in a InvalidArgument error.

Updating the GetProfile method to support ip address lookups is out of scope and attempts to look up an ip address with the GetProfile method will result in InvalidArgument.

Implementation

We do this by creating a IPWatcher which wraps the EndpointsWatcher and supports lookups by ip. IPWatcher maintains a mapping up clusterIPs to service ids and translates subscriptions to an IP address into a subscription to the service id using the underlying EndpointsWatcher.

Since the service name is no longer always infer-able directly from the input parameters, we restructure EndpointTranslator and PodSet so that we propagate the service name from the endpoints API response.

Testing

This can be tested by running the destination service locally, using the current kube context to connect to a Kubernetes cluster:

go run controller/cmd/main.go destination -kubeconfig ~/.kube/config

Then lookups can be issued using the destination client:

go run controller/script/destination-client/main.go -path 192.168.54.78:80 -method get -addr localhost:8086

Service cluster ips and pod ips can be used as the path argument.

[zaharidichev]

Maybe this questions is quite silly, but just trying to understand things a little better. Why is this error not handled here. Is it because the Svc().Informer() is guaranteed to not return services of type ExternalName? What happens if this service changes to an external type right before you hit the k8s API?

[adleong]

This is not a silly question, it's actually a great point.

We should probably handle errors here by sending a NoEndpoints(exists = false) message to the subscribers.

[alpeb]

Looking good 👍

I was wondering about support for pods with hostNetwork: true... Calling Get with an IP for such a pod will return nothing because the pod indexer in getOrNewServiceSubscriptions might return more than one pod.
Would it make sense to map things not only by clusterIP but also by port?

Besides that, I just found a couple of nits described below, plus the comment of the NewServer func in server.go that needs to be updated.

[alpeb]

To clarify over my previous comment, the log shows the "Pod IP conflict" at startup, and the Get call does actually return an answer, but not necessarily corresponding to the right pod.

Note that hostNetwork: true pods are never meshed, which I guess no mTLS can be established. Not sure if that would require a special treatment.

[adleong]

Nice catch, @alpeb. I think the simplest solution would be to omit pods which have hostNetwork true.

[olix0r]

I've built and pushed an image from this branch, including the required proxy changes under git-9f78d7e6

[olix0r]

Appears to work as expected; however it seemed that the emojivoto scrapes weren't discovered properly until emojivoto was reinjected...

:; linkerd metrics -n linkerd deploy/linkerd-prometheus | grep ^response_total
response_total{direction="outbound",tls="no_identity",no_tls_reason="not_provided_by_service_discovery",classification="failure",error="unclassified"} 45
response_total{direction="outbound",tls="no_identity",no_tls_reason="not_provided_by_service_discovery",status_code="200",classification="success"} 1303
response_total{direction="outbound",tls="no_identity",no_tls_reason="not_provided_by_service_discovery",status_code="502",classification="failure"} 2
response_total{direction="inbound",tls="no_identity",no_tls_reason="not_provided_by_remote",status_code="200",classification="success"} 93
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-destination",dst_pod="linkerd-destination-67fc4995d4-5b6z8",dst_pod_template_hash="67fc4995d4",dst_serviceaccount="linkerd-destination",tls="true",server_id="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 92
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-tap",dst_pod="linkerd-tap-9d56f9d78-bdbm4",dst_pod_template_hash="9d56f9d78",dst_serviceaccount="linkerd-tap",tls="true",server_id="linkerd-tap.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 92
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-controller",dst_pod="linkerd-controller-59b98cd9f-xh6fw",dst_pod_template_hash="59b98cd9f",dst_serviceaccount="linkerd-controller",tls="true",server_id="linkerd-controller.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 138
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-sp-validator",dst_pod="linkerd-sp-validator-6d6d66f959-jlhgf",dst_pod_template_hash="6d6d66f959",dst_serviceaccount="linkerd-sp-validator",tls="true",server_id="linkerd-sp-validator.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 92
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-proxy-injector",dst_pod="linkerd-proxy-injector-665676f8f4-s6kcv",dst_pod_template_hash="665676f8f4",dst_serviceaccount="linkerd-proxy-injector",tls="true",server_id="linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 46
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-destination",dst_pod="linkerd-destination-67fc4995d4-2dg4r",dst_pod_template_hash="67fc4995d4",dst_serviceaccount="linkerd-destination",tls="true",server_id="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 91
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-controller",dst_pod="linkerd-controller-59b98cd9f-h45g6",dst_pod_template_hash="59b98cd9f",dst_serviceaccount="linkerd-controller",tls="true",server_id="linkerd-controller.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 135
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-proxy-injector",dst_pod="linkerd-proxy-injector-665676f8f4-c98z8",dst_pod_template_hash="665676f8f4",dst_serviceaccount="linkerd-proxy-injector",tls="true",server_id="linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 89
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-sp-validator",dst_pod="linkerd-sp-validator-6d6d66f959-c49hn",dst_pod_template_hash="6d6d66f959",dst_serviceaccount="linkerd-sp-validator",tls="true",server_id="linkerd-sp-validator.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 88
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-proxy-injector",dst_pod="linkerd-proxy-injector-665676f8f4-75ql9",dst_pod_template_hash="665676f8f4",dst_serviceaccount="linkerd-proxy-injector",tls="true",server_id="linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 87
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-tap",dst_pod="linkerd-tap-9d56f9d78-hr4vl",dst_pod_template_hash="9d56f9d78",dst_serviceaccount="linkerd-tap",tls="true",server_id="linkerd-tap.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 86
response_total{authority="linkerd-prometheus.linkerd.svc.cluster.local:9090",direction="inbound",tls="true",client_id="linkerd-controller.linkerd.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 6317
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="vote-bot",dst_pod="vote-bot-55fb746c54-4kntg",dst_pod_template_hash="55fb746c54",dst_serviceaccount="default",tls="true",server_id="default.emojivoto.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 6
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="emoji",dst_pod="emoji-847d6d58d4-6j2jm",dst_pod_template_hash="847d6d58d4",dst_serviceaccount="emoji",tls="true",server_id="emoji.emojivoto.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 6
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="voting",dst_pod="voting-7c8c96b45c-b2kvq",dst_pod_template_hash="7c8c96b45c",dst_serviceaccount="voting",tls="true",server_id="voting.emojivoto.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 6
response_total{direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="web",dst_pod="web-bf8469c6b-4dvdb",dst_pod_template_hash="bf8469c6b",dst_serviceaccount="web",tls="true",server_id="web.emojivoto.serviceaccount.identity.linkerd.cluster.local",status_code="200",classification="success"} 6

[adleong]

Appears to work as expected; however it seemed that the emojivoto scrapes weren't discovered properly until emojivoto was reinjected...

Perhaps prometheus was reusing long lived connections? I would expect that restarting prometheus would also cause everything to get picked up properly.

[olix0r]

On Mon, Nov 4, 2019 at 14:01 Alex Leong ***@***.***> wrote: Appears to work as expected; however it seemed that the emojivoto scrapes weren't discovered properly until emojivoto was reinjected... Perhaps prometheus was reusing long lived connections? I would expect that restarting prometheus would also cause everything to get picked up properly.

This was following a control plane upgrade, so Prometheus should have been restarted? —

You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#3595?email_source=notifications&email_token=AAB2YYW2PYDUGOEDFJJSGYDQSCLT5A5CNFSM4JBTSZX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDA3XGQ#issuecomment-549567386>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB2YYU5JJB4POXN37ITXLLQSCLT5ANCNFSM4JBTSZXQ> .

-- Oliver Gould <ver@buoyant.io>

[adleong]

To see this in action:

• Install or upgrade to this verison
• kubectl -n linkerd port-forward deploy/linkerd-prometheus 9090
• Browse metrics like request_total{deployment="linkerd-prometheus",direction="outbound"}

This will show metrics for requests from Prometheus to other pods in the cluster. Notice that these metrics have full dst metadata including pod name, etc, and have tls=true. This is true even though Prometheus addresses requests directly to pod IPs.

[ihcsim]

I added some comments and questions regarding the new IP watcher. I'll take look at the unit test and do more testing tomorrow.

[ihcsim]

How does this work with the AddEventHandler in the endpoint_watcher.go? Will one override the other? Or is this a separate Informer instance?

[adleong]

it's the same imformer with multiple handler funcs registered.

[ihcsim]

Do we need to perform a iw.deletePod(obj) before the iw.addPod(obj)?

[adleong]

since pod IP addresses are immutable once assigned, I don't think so.

[ihcsim]

Do we need to add the corresponding k8sAPI.Pod().Informer().AddIndexers() to the NewAPIWatcher() constructor, like what you did for Svc?

Also, how come the pod informer knows about clusterIP?

[adleong]

the endpoints watcher has already added the pod ip indexer so we don't need to do it.

I'm using the name clusterIP a bit more broadly to mean the ip address of a resource in the cluster. This can be the ip address of a pod, or the cluster ip of a service.

[ihcsim]

IIUC, the writer lock is only needed when if !ok, right? Is this the same as:

ss, ok := iw.getServiceSubscription()
if ! ok {
  iw.Lock()
  defer iw.Unlock()
  ss = &serviceSubscriptions{
    clusterIP: clusterIP,
    listeners: make(map[EndpointUpdateListener]Port),
    endpoints: iw.endpoints,
    log:       iw.log.WithField("clusterIP", clusterIP),
  }
...
}
return ss

[adleong]

I'm not sure, but we might need a read lock even in the ok case. The way we have it is slightly more conservative, but as far as I can tell it's idiomatic and makes things easier to reason about.

[ihcsim]

Won't ss.pod return the same thing? This seems like a repetition of what iw.getOrNewServiceSubscriptions() did to initialize the PodSet in the serviceSubscription, with the exception that iw.getOrNewServiceSubscriptions() also checks for pod.spec.HostNetwork.

[adleong]

good call

[ihcsim]

These use test cases seem identical(-ish) to those defined in endpoints_watcher_test.go. Any reasons why we need to re-run them here? Do the behaviour of the endpoints_watcher changes after it's being wrapped?

[adleong]

the way the lookups works is different, with lookups by ip instead of by service name. I think it's worth testing these conditions.

[ihcsim]

This works great! Previously, if I used an injected curl pod to curl a pod IP, tap will display the requests with TLS not provided. With this change, requests with IP authority is now TLS'ed 👍.

⚡  linkerd tap po                        
req id=2:0 proxy=out src=10.244.0.17:57658 dst=10.244.0.16:80 tls=true :method=GET :authority=10.244.0.16 :path=/api/list
rsp id=2:0 proxy=out src=10.244.0.17:57658 dst=10.244.0.16:80 tls=true :status=200 latency=7609µs
end id=2:0 proxy=out src=10.244.0.17:57658 dst=10.244.0.16:80 tls=true duration=374µs response-length=4513B
...
req id=2:5 proxy=out src=10.244.0.17:57972 dst=10.244.0.16:80 tls=true :method=GET :authority=web-sts-svc.emojivoto :path=/api/list
rsp id=2:5 proxy=out src=10.244.0.17:57972 dst=10.244.0.16:80 tls=true :status=200 latency=3653µs
end id=2:5 proxy=out src=10.244.0.17:57972 dst=10.244.0.16:80 tls=true duration=64µs response-length=4513B

Also, can confirm using Wireshark to show that the traffic is truly encrypted.

[alpeb]

It seems there's still an issue with pods with hostNetwork: true. If clusterIP corresponds to one or more than one such pods, even though they'll be ignored below, ss still gets set and will get returned at the end of this method.

[adleong]

hmmm, what do you think the correct behavior should be here?

getOrNewServiceSubscriptions should always return a serviceSubscriptions even if the ip address doesn't correspond to any pods or services in the cluster. (or it only corresponds to hostNetwork pods). In either of these cases, the returned serviceSubscriptions should have ss.pod = PodSet{}.

This causes updatePod to send a listener.Add with a size 0 pod set. Perhaps we simply need a check here to skip the add if the size of the pod set is 0.

Is there another issue that I'm missing?