linkerd 2.10 regression when meshing a forward-proxy

[Esardes]

Bug Report

What is the issue?

In linkerd < 2.10, I could mesh a pod and a forward proxy and use my proxy from my pod as any other service.
Now I get different errors in different cases.

How can it be reproduced?

Have a meshed forward proxy, an unmeshed one, a meshed pod, and an unmeshed one at the ready.
Errors mentioned below can be found here
Tests return:

• from client-unmeshed: curl -x proxy-unmeshed:80 --insecure https://httpbin.org/ip WORKS
• from client-unmeshed: curl -x proxy-meshed:80 --insecure https://httpbin.org/ip FAILS with ERROR1
• from client-meshed: curl -x proxy-unmeshed:80 --insecure https://httpbin.org/ip FAILS with ERROR2
• from client-meshed: curl -x proxy-meshed:80 --insecure https://httpbin.org/ip FAILS with ERROR2

Now same if I try to reach http://httpbin.org/ip (no ssl)

• client-meshed with proxy-meshed: fails with Unauthorized - my proxy auth headers seem to never reach our proxy-pod behind linkerd-proxy
• client-meshed with proxy-unmeshed: works
• client-unmeshed with proxy-meshed: works
• client-unmeshed with proxy-unmeshed: works

Logs, error output, etc

I capture some logs with level debug when trying to reach an unmeshed proxy from a meshed-pod

linkerd check output

ontrol plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

Environment

• Kubernetes Version: v1.18.9-eks-d1db3c
• Cluster Environment: EKS
• Host OS: ubuntu 20.04LTS
• Linkerd version: 2.10

Possible solution

@olix0r mentionned it could be "I'm not immediately sure what's going on but we might be handling CONNECT poorly?" but I am clueless

Additional context

[kleimkuhler]

@Esardes It would be helpful if you are able to test a more recent edge such as edge-21.4.1. It includes a fix which was causing issues with inbound TLS detection. It's possible the forward proxy's Linkerd proxy is hitting this issue. I'm not too confident about that, but it would be worth testing.

Meanwhile I started working on trying to reproduce this myself and don't have a significant update yet. While I work on setting up a repro, it'd be helpful if you can update me that this issue does or does not persist on edge-21.4.1. Thanks!

[kleimkuhler]

@Esardes I have created an Nginx forward proxy and tested this on stable-2.10.0. Unfortunately I'm not seeing the errors that you have run in to. I'll leave some steps on how to setup my forward proxy in a cluster; maybe you can use that to see what happens for you?

The forward proxy image is ghcr.io/kleimkuhler/nginx-forward-proxy:v0.0.1. You can run this by itself an confirm that curl -x works as expected with the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-1
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - name: nginx-1
        image: ghcr.io/kleimkuhler/nginx-forward-proxy:v0.0.1
        imagePullPolicy: Always
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

$ kubectl exec nginx-1-bff6c67cf-m84vs -it -- bash -il
# curl httpbin.org directly
root@nginx-1-bff6c67cf-m84vs:/# curl https://httpbin.org/ip
{
  "origin": "34.86.52.153"
}
# curl httpbin.org through forward-proxy port 3128
root@nginx-1-bff6c67cf-m84vs:/# curl -x localhost:3128 https://httpbin.org/ip
{
  "origin": "34.86.52.153"
}
# pass -v to confirm CONNECT frame is sent/received correctly
root@nginx-1-bff6c67cf-m84vs:/# curl -x localhost:3128 -v https://httpbin.org/ip
...
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.64.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection Established
< Proxy-agent: nginx
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
...

So this confirms -x works for this basic forward proxy. Now installed Linkerd stable-2.10.0 and apply this manifest. This creates an uninjected/injected client and uninjected/injected forward proxy.

kubectl exec into either of the clients (curl-1 or curl-2) and send requests to the pod IPs of nginx-1 or nginx-2. Here, you'll notice that the requests continue to go through correctly. This works for all combinations of client -> forward proxy.

Please let me know if you can provide any additional details or confirm that this works for you as well. Also, it would be really helpful if you can still test edge-21.4.1. Thanks!

[adleong]

Closing due to inactivity.