Sporadic HTTP 403 Response Codes Due to TLS Handshake Failures

[soma-kurisu]

What is the issue?

We are encountering sporadic HTTP 403 response codes across all our clusters in the Linkerd-proxy setup. The attached logs capture the conversation flow from gojira-pod's Linkerd-proxy sidecar <-> gojira-pod's Linkerd-debug sidecar <-> mosura-pod's Linkerd-debug sidecar <-> mosura-pod's Linkerd-proxy sidecar, illustrating one such occurrence. Approximately 5 promille of all requests fail due to unsuccessful TLS handshakes, resulting in HTTP 403 response codes.

How can it be reproduced?

To reproduce the issue, you can generate approximately 1000 requests to a linkerd-proxy setup. A small percentage of these requests will fail, with the log message "Peer does not support TLS" occurring during the TLS handshake. The client and target are situated in different namespaces, with the client being gojira-pod and the server being mosura-pod, where mosura serves the kani-api. This issue can be replicated using any client communicating with any server that provides a REST API.

For detailed configuration, please refer to the "Additional context" section, where the configurations of authorizationpolicies.v1alpha1.policy.linkerd.io, servers.v1beta3.policy.linkerd.io, and meshtlsauthentications.v1alpha1.policy.linkerd.io can be reviewed.

Logs, error output, etc

gojira-debug.log.csv
gojira-proxy.log.csv
mosura-debug.log.csv
mosura-proxy.log.csv

output of linkerd check -o short

kurisu@linkerd-worries$ linkerd check -n linkerd --linkerd-namespace linkerd --cni-namespace linkerd  -o short
linkerd-identity
----------------
‼ trust anchors are valid for at least 60 days
    Anchors expiring soon:
        * 4116 BlueOysterCA will expire on 2025-05-23T16:53:17Z
    see https://linkerd.io/2/checks/#l5d-identity-trustAnchors-not-expiring-soon for hints
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2025-05-01T08:56:18Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
    Anchors expiring soon:
        * 4116 BlueOysterCA will expire on 2025-05-23T16:53:17Z
        * 558509666953049899959298119593666570293338784147 root.linkerd.cluster.local will expire on 2025-04-13T15:29:32Z
    see https://linkerd.io/2/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
    Anchors expiring soon:
        * 4116 BlueOysterCA will expire on 2025-05-23T16:53:17Z
        * 558509666953049899959298119593666570293338784147 root.linkerd.cluster.local will expire on 2025-04-13T15:29:32Z
    see https://linkerd.io/2/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
‼ policy-validator cert is valid for at least 60 days
    Anchors expiring soon:
        * 4116 BlueOysterCA will expire on 2025-05-23T16:53:17Z
        * 558509666953049899959298119593666570293338784147 root.linkerd.cluster.local will expire on 2025-04-13T15:29:32Z
    see https://linkerd.io/2/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    is running version 25.2.3 but the latest edge version is 25.3.3
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 25.1.2 but the latest edge version is 25.3.3
    see https://linkerd.io/2/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running edge-25.1.2 but cli running edge-25.2.3
    see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-5c7f864fcc-2zqp6 (edge-25.1.2)
        * linkerd-destination-5c7f864fcc-k2k7t (edge-25.1.2)
        * linkerd-identity-7554d6fdd-pjhvd (edge-25.1.2)
        * linkerd-identity-7554d6fdd-qlwph (edge-25.1.2)
        * linkerd-proxy-injector-6d47f9cc8-jht52 (edge-25.1.2)
        * linkerd-proxy-injector-6d47f9cc8-splcs (edge-25.1.2)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-5c7f864fcc-2zqp6 running edge-25.1.2 but cli running edge-25.2.3
    see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints

Status check results are √

Environment

kurisu@linkerd-worries$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.7

Possible solution

No response

Additional context

kurisu@linkerd-worries$ kubectl get -n mosura-land authorizationpolicies.v1alpha1.policy.linkerd.io
NAME                            AGE
kani-auth-gojira-land           16d
kani-auth-mosura-land           16d
kurisu@linkerd-worries$ kubectl get -n gojira-land authorizationpolicies.v1alpha1.policy.linkerd.io
No resources found in gojira-land namespace.

kurisu@linkerd-worries$ linkerd authz -n mosura-land deploy/kani-api
ROUTE   SERVER              AUTHORIZATION_POLICY     SERVER_AUTHORIZATION
*       kani-api-server     kani-auth-gojira-land
*       kani-api-server     kani-auth-mosura-land

kurisu@linkerd-worries$ kubectl describe servers.v1beta3.policy.linkerd.io -n mosura-land kani-api-server
Name:         kani-api-server
Namespace:    mosura-land
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: kani-api
              meta.helm.sh/release-namespace: mosura-land
API Version:  policy.linkerd.io/v1beta3
Kind:         Server
Metadata:
  Creation Timestamp:  <redacted>
  Generation:          <redacted>
  Resource Version:    <redacted>
  UID:                 <redacted>
Spec:
  Access Policy:  deny
  Pod Selector:
    Match Labels:
      App:         kani-api
  Port:            8080
  Proxy Protocol:  unknown
Events:            <none>

kurisu@linkerd-worries$ kubectl describe authorizationpolicies.v1alpha1.policy.linkerd.io -n mosura-land kani-auth-gojira-land
Name:         kani-auth-gojira-land
Namespace:    mosura-land
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: kani-api
              meta.helm.sh/release-namespace: mosura-land
API Version:  policy.linkerd.io/v1alpha1
Kind:         AuthorizationPolicy
Metadata:
  Creation Timestamp:  <redacted>
  Generation:          <redacted>
  Resource Version:    <redacted>
  UID:                 <redacted>
Spec:
  Required Authentication Refs:
    Group:  policy.linkerd.io
    Kind:   MeshTLSAuthentication
    Name:   kani-gojira-meshauth
  Target Ref:
    Group:  policy.linkerd.io
    Kind:   Server
    Name:   kani-api-server
Events:     <none>

kurisu@linkerd-worries$ kubectl describe authorizationpolicies.v1alpha1.policy.linkerd.io -n mosura-land kani-auth-mosura-land
Name:         kani-auth-mosura-land
Namespace:    mosura-land
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: kani-api
              meta.helm.sh/release-namespace: mosura-land
API Version:  policy.linkerd.io/v1alpha1
Kind:         AuthorizationPolicy
Metadata:
  Creation Timestamp:  <redacted>
  Generation:          <redacted>
  Resource Version:    <redacted>
  UID:                 <redacted>
Spec:
  Required Authentication Refs:
    Group:  policy.linkerd.io
    Kind:   MeshTLSAuthentication
    Name:   kani-mosura-meshauth
  Target Ref:
    Group:  policy.linkerd.io
    Kind:   Server
    Name:   kani-api-server
Events:     <none>

kurisu@linkerd-worries$ kubectl describe -n mosura-land meshtlsauthentications.v1alpha1.policy.linkerd.io kani-gojira-meshauth
Name:         kani-gojira-meshauth
Namespace:    mosura-land
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: kani-api
              meta.helm.sh/release-namespace: mosura-land
API Version:  policy.linkerd.io/v1alpha1
Kind:         MeshTLSAuthentication
Metadata:
  Creation Timestamp:  <redacted>
  Generation:          <redacted>
  Resource Version:    <redacted>
  UID:                 <redacted>
Spec:
  Identity Refs:
    Kind:       ServiceAccount
    Name:       gojira-proxy
    Namespace:  gojira-land
Events:         <none>

kurisu@linkerd-worries$ kubectl describe -n mosura-land meshtlsauthentications.v1alpha1.policy.linkerd.io kani-mosura-meshauth
Name:         kani-mosura-meshauth
Namespace:    mosura-land
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: kani-api
              meta.helm.sh/release-namespace: mosura-land
API Version:  policy.linkerd.io/v1alpha1
Kind:         MeshTLSAuthentication
Metadata:
  Creation Timestamp:  <redacted>
  Generation:          <redacted>
  Resource Version:    <redacted>
  UID:                 <redacted>
Spec:
  Identity Refs:
    Kind:       ServiceAccount
    Name:       default
    Namespace:  mosura-land
Events:         <none>

Would you like to work on fixing this bug?

None

[soma-kurisu]

related issues #13013 and #6548

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[soma-kurisu]

I'd like to share an update based on some recent testing I conducted:

I ran a series of experiments using various combinations of Linkerd versions (edge-24.8.2, edge-25.1.2, and edge-25.4.2), Gateway API configurations (as provided by the Linkerd installation), and NGINX Ingress Controller versions (4.11.2, 4.12.0, and 4.12.1) across multiple kind clusters running Kubernetes (versions 1.30.7 and 1.31.8). In each scenario, I simulated traffic by sending a substantial number of requests (two parallel consumers, each issuing 1000 sequential requests) through the ingress to a backend pod within a Linkerd service mesh.

In none of these test cases was I able to reproduce the 403 Forbidden errors.

That said, since we continue to observe sporadic 403 responses in our production environment - and considering that @cratelyn kindly removed the wontfix label - I assume there may be some level of ongoing investigation or interest in this issue.

Would you recommend any interim mitigation strategies to help reduce the frequency or impact of these errors while the root cause is still being explored?

[adleong]

Hi @soma-kurisu thanks for all the work you've already done investigating this. We've tried to track down these types of sporadic TLS issues in the past but they've been hard to nail down without a consistent way to reproduce them. In some cases, they can happen when a node is CPU constrained, so the overall health of your node might be something to look at.

I'll also point out that Linkerd's TLS handling has changed significantly between edge-25.2.1 (where you reported the issue) and now (e.g. edge-25.6.4). I would recommend upgrading to a more recent edge release and seeing if the TLS errors occur there.

[soma-kurisu]

Thanks for the input, @adleong - much appreciated.

We'll take a closer look at potential CPU constraints on the nodes and will also try upgrading to a more recent edge release like edge-25.6.4 to see if the issue persists.

[micke-post]

I'll also point out that Linkerd's TLS handling has changed significantly between edge-25.2.1 (where you reported the issue) and now (e.g. edge-25.6.4). I would recommend upgrading to a more recent edge release and seeing if the TLS errors occur there.

Do you happen if there is a specific version where the change (or the bulk of changes) happened? We since upgraded to edge-25.4.2 but are still seeing the issues.

[bkittinger]

@adleong some updates on this while @soma-kurisu is away:

We've encountered the issue again this week during a production deployment. At the time, we were running edge-25.4.2. In response to the incident, we upgraded our clusters to edge-25.6.2 as any newer version is affected by #14298 which is also an issue for our workloads. Also, at the time of occurrence, the nodes were not CPU constrained at all.

Next Monday there will be another attempt at the production deployment that casued the most recent incident. We'll observe and let you know if we have finally found a way for reliable reproduction of the issue or if the upgrade seems to have fixed things.

[micke-post]

Unfortunately the issue occurred despite the upgrade to 25.6.2. This time we were lucky enough to be able to get a look at the in-progress issue and could gather some more information.

• We observed the behavior in a workload with 6 replicas on the receiving side.
• All replicas received load balanced traffic, but only one of the replicas had the 403-issue.
• All replicas were on separate nodes.
• While the issue happened we didn't observe any kind of CPU, memory, or IO pressure on the node, pod, or linkerd-proxy container
• No other pod on the same node, or any other node on the cluster, generated 403 messages while the incident was in progress
• Restarting the failing replica fixed the issue

[bkittinger]

@adleong we have a major update on the behavior observed. During the past week, the issue has occurred in production after two different application deployments. In one case, the deployment was rolled back, in the other case, the affected pod (1/6) was restarted. What is noteworthy is that once the issue occurs, it persists for some time until it potentially goes away itself.

Since we cannot reliably reproduce the start of the problem, troubleshooting has been quite difficult as all we had to work with were logs and never the pods showing the odd behavior. Yesterday, I was working on configuring linkerd-viz in a sandbox environment with no external load at all when the problem randomly occurred on the pod I had the log stream open on (talk about luck). Here's the log file: demotenant-api-application-apiapp-deployment-7695c8bcc8-b7hcc-1754322047371503600.log

Observations from the debug session

• The pod from which the logs are extracted was restarted as part of a scaling and immediately started to show the 403 with NoClientHello behavior (log file starts at 44 seconds, the issue was there before). The IP 10.207.204.11 is prometheus trying to scrape the metrics from the admin API.
• Prometheus continued to re-try the calls every few seconds, every time with the same failure
• At 143s, I tried to call the pod from a test pod in another namespace which did not have any AuthorizationPolicy to allow that connection. As expected, this resulted in a 403 but the main difference is that in this case the client is identified.
• Between 184s and 194s, I restarted the prometheus pod which resulted in the IP changing to 10.207.204.44. From that point on, the prometheus connections also fail with 403 but now with identity which is expected as there was no AuthorizationPolicy in place yet.
• While this was happening, prometheus was also trying to scrape other pods where it did not encounter the NoClientHello problem.

What we know so far

• The issue appears randomly, most likely after workload deployments
• It only affects a single pairing of src / dst - the client is able to establish other connections and the server is able to receive other connections
• Restarting either the client or the service pod will bring things back to normal
• The issue tends to go away itself in the evening (where we have much lower load on the system)

What we believe is happening

• Service discovery fails for a connection due to a unknown reason
• The client proxy caches this discovery result for 5 seconds due to the default setting of outboundDiscoveryCacheUnusedTimeout
• All subsequent calls to the same destination from this client will fail unless there is a period of time where no call happens for at least 5 seconds.

This would explain why only a single pairing of src / dst is affected. Also, why restarting any of the two pods solves the problem (in one case the cache is destroyed, in the other the destination changes and the cache is not accessed anymore) and why the problem tends to go away when load decreases (less calls = more likely to make 5s without outbound call).

Conclusion
We still do not know the reason for the initial discovery failure. However, it is clearly a transient fault of sorts.

I see two possible solutions for this problem (without being familiar with the codebase):

• Adapt the caching logic to not cache discovery results where the TLS handshake failed
• Implement a second cache timeout (e.g. outboundDiscoveryCacheTimeout) that is not specific to unused cache items. This way, regular re-discovery could be forced to take place, making sure transient discovery faults do not turn into persistent ones.

[alpeb]

Thanks for the detailed report @bkittinger . Quick question just to rule out something similar to #14103, which was fixed on edge-25.6.3: are you using native sidecars?

[bkittinger]

@alpeb We are using native sidecars.

However, the issue here does not lie within the policy engine - it is doing what it is supposed to.
As @soma-kurisu pointed out in his initial post, the problem is a failed TLS handshake between the proxies, resulting in a fallback to plain HTTP communications and then a block by the inbound policy as the client is unauthenticated (since there is no certificate). And because the discovery result is cached, a single failed handshake can cause a persisting issue.

[alpeb]

It seems we may need a bit more information to form a hypothesis about the cause. Could you help clarify why a fallback to plain text might occur if the TLS handshake fails? That is something that the proxy on the inbound side wouldn't do. In the reproduction steps you provided, it appears that Prometheus, as the client, is encountering the issue. Have you set up different scrape configurations using both HTTPS and HTTP schemes?