host header is not being sent with probes after upgrading to 1.18.1 #46087

mmerickel · 2023-07-20T05:07:00Z

Is this the right place to submit this?

This is not a security vulnerability or a crashing bug
This is not a question about how to use Istio

Bug Description

After upgrading from istio 1.16.1 to 1.18.1 my pods are no longer receiving the correct Host header on their startup/readiness/liveness probes. I have inconsistent results with other custom headers. I have a few pods that are receiving custom headers, but most of them are not receiving those as well. All with the same pod spec.

This configuration worked with istio 1.16.1. I have not tested intermediate versions.

I modified the app to ignore the host header to debug this and can confirm that the probe is hitting the correct endpoint on our app. However, we expect the host header to be sent in the request.

Readiness probe in Deployment

        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Host
              value: example.com
            - name: X-Request-Id
              value: readiness-probe
            path: /__health
            port: 5000
            scheme: HTTP
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1

Readiness probe in auto-injected Pod

    readinessProbe:
      failureThreshold: 3
      httpGet:
        httpHeaders:
        - name: Host
          value: example.com
        - name: X-Request-Id
          value: readiness-probe
        path: /app-health/web/readyz
        port: 15020
        scheme: HTTP
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 1

ISTIO_KUBE_APP_PROBERS

    - name: ISTIO_KUBE_APP_PROBERS
      value: '{"/app-health/web/livez":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"liveness-probe"}]},"timeoutSeconds":1},"/app-health/web/readyz":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"readiness-probe"}]},"timeoutSeconds":1},"/app-health/web/startupz":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"startup-probe"}]},"timeoutSeconds":1}}'

Captured tcpdump

You can see that the Host header being sent is 10.34.54.17:5000.
You can also see the X-Request-Id is the expected readiness-probe value.

$ tcpdump -i lo -X port 5000
<snip>
        0x0030:  f3a7 1fa5 4745 5420 2f5f 5f68 6561 6c74  ....GET./__healt
        0x0040:  6820 4854 5450 2f31 2e31 0d0a 486f 7374  h.HTTP/1.1..Host
        0x0050:  3a20 3130 2e33 342e 3534 2e31 373a 3530  :.10.34.54.17:50
        0x0060:  3030 0d0a 5573 6572 2d41 6765 6e74 3a20  00..User-Agent:.
        0x0070:  6b75 6265 2d70 726f 6265 2f31 2e32 362b  kube-probe/1.26+
        0x0080:  0d0a 4163 6365 7074 3a20 2a2f 2a0d 0a43  ..Accept:.*/*..C
        0x0090:  6f6e 6e65 6374 696f 6e3a 2063 6c6f 7365  onnection:.close
        0x00a0:  0d0a 582d 5265 7175 6573 742d 4964 3a20  ..X-Request-Id:.
        0x00b0:  7265 6164 696e 6573 732d 7072 6f62 650d  readiness-probe.
        0x00c0:  0a41 6363 6570 742d 456e 636f 6469 6e67  .Accept-Encoding
        0x00d0:  3a20 677a 6970 0d0a 0d0a                 :.gzip....

Version

$ istioctl version
client version: 1.17.3
control plane version: 1.18.1
data plane version: 1.16.1 (1 proxies), 1.18.1 (47 proxies)

Note that the 1 proxy on 1.16.1 is the old version of the pod I'm trying to redeploy with 1.18.1 but the startup probe is failing due to this issue.

$ kubectl version --short
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.26.6-eks-a5565ad

Additional Information

Probably related: #45632 and #45482

I should probably emphasize again that I have only 1 pod where the x-request-id custom headers are working out of 4 running examples (across 2 separate kubernetes clusters). And every pod has a broken host header.

Affected product area

The text was updated successfully, but these errors were encountered:

howardjohn · 2023-07-20T16:27:11Z

cc @jaellio

mmerickel · 2023-07-20T17:15:56Z

I feel confident at this point that the issue is specific to the Host header. I was able to find some inconsistent config in our app related to the other custom headers that was throwing me off between a couple different deployments I was comparing... The tcpdump in this issue proves that the Host header itself is being sent incorrectly as part of the probe and that's the focus of this issue.

jaellio · 2023-07-20T19:00:10Z

I'll take a look at this! Thanks for creating the issue!

mmerickel · 2023-07-20T19:14:17Z

Awesome, thank you @jaellio! Let me know if there's some more info I can try to provide. I had to redact things a little bit and not show the full pod because it's on our internal staging deployments, but if you gave me a less sensitive deployment/pod you wanted me to try to run I could probably do that as well. I have identical deployments atm in both 1.16.1 and 1.18.1 to compare.

hzxuzhonghu · 2023-07-21T08:23:37Z

I am not sure how you take example.com. In kubelet, the prober will send request to HttpGet.Host

mmerickel · 2023-07-21T16:10:11Z

I am not sure how you take example.com. In kubelet, the prober will send request to HttpGet.Host

Apologies I'm not clear what point you're making here.

The tl;dr of the issue is that istio injection is rewriting the main container such that probes go to istio-proxy over port 15020. The proxy is then re-sending its own probe query to the main container over the loopback interface. The request to the main container from the istio-proxy (over the loopback interface) is showing the wrong Host header in the tcpdump.

mmerickel · 2023-07-21T22:33:18Z

I was poking at this a bit and I feel like it's worth pointing out that if I set the httpHeaders value to host or HOST, etc I do not get the liveness probe sent to my app at all (nothing shows in tcpdump). It seems to only work if I set the name: Host specifically. This surprises me a lot and figure something weird is going on. The casing is reflected in the ISTIO_KUBE_APP_PROBERS.

I also tried things like x-request-id and that did work and got translated to X-Request-Id in the main container - it feels like some header normalization is going on somewhere? I wish I had a way to debug istio but I have no clue how to setup a harness for that, just to see the values in this code path https://github.com/istio/istio/blob/1.18.1/pilot/cmd/pilot-agent/status/server.go#L689-L722.

howardjohn · 2023-07-21T22:37:08Z

Kubernetes itself also does some similar/weird/broken casing things as well that were recently fixed. Will try to find a link

…

On Fri, Jul 21, 2023, 3:33 PM Michael Merickel ***@***.***> wrote: I was poking at this a bit and I feel like it's worth pointing out that if I set the httpHeaders value to host or HOST, etc I do not get the liveness probe sent to my app at all (nothing shows in tcpdump). It seems to only work if I set the name: Host specifically. This surprises me a lot and figure something weird is going on. The casing is reflected in the ISTIO_KUBE_APP_PROBERS. I also tried things like x-request-id and that did work and got translated to X-Request-Id in the main container - it feels like some header normalization is going on somewhere? I wish I had a way to debug istio but I have no clue how to setup a harness for that, just to see the values in this code path https://github.com/istio/istio/blob/1.18.1/pilot/cmd/pilot-agent/status/server.go#L689-L722 . — Reply to this email directly, view it on GitHub <#46087 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXJMDMB7F5W3YAB6MJ3XRL7TVANCNFSM6AAAAAA2Q3MX5E> . You are receiving this because you commented.Message ID: ***@***.***>

mmerickel · 2023-07-22T04:23:52Z

Quick update, I reverted to 1.18.0 and things are working as expected there. So something changed between 1.18.0 and 1.18.1 that is causing this behavior. I also tried 1.17.4 and it is working fine as well.

Also wrt the x-request-id casing, on 1.17.4 if I use the lowercase version then the runtime sends the app header twice (both keys normalized to X-Request-Id) but as John said it may not be istio doing that and it's not a problem if I specify the case correctly in the Deployment. On 1.18.0 I do not observe the header being sent twice. Same version of kubernetes (1.26 on EKS).

hzxuzhonghu · 2023-07-24T02:08:04Z

I was poking at this a bit and I feel like it's worth pointing out that if I set the httpHeaders value to host or HOST, etc I do not get the liveness probe sent to my app at all (nothing shows in tcpdump).

Without istio enabled, if you set HTTP Header Host: example.com, the readiness/liveness Probe will fail too.

Please take a look at the kubelet probe code kubernetes/pkg/kubelet/prober/prober.go

mmerickel · 2023-07-24T02:45:06Z

Without istio enabled, if you set HTTP Header Host: example.com, the readiness/liveness Probe will fail too.

This issue is about setting a httpHeaders item for the host header. Not about the HttpGet.Host attribute. I'm aware that if you set HttpGet.Host that it will try to open a connection to that address, but that setting is not about the header specifically.

hzxuzhonghu · 2023-07-24T03:06:38Z

Setting host header will override HttpGet.Host

hzxuzhonghu · 2023-07-24T03:08:33Z

PTAL https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/prober/prober.go#L143-L156, and track down the NewRequestForHTTPGetAction function

hzxuzhonghu · 2023-07-24T03:50:21Z

I know what happened here.

mmerickel · 2023-07-24T03:50:25Z

What part of this issue are you referring to? I’ve tested my claims about the header vs HttpGet.Host as well as being super confident our config has been working for the last couple years through many versions of istio and k8s using the example posted above. I have repro’d the literal example.com header working in 1.16.1, 1.17.4, 1.18.0 and failing in 1.18.1. It does fail when using HttpGet.Host for reasons stated above which is expected.

mmerickel · 2023-07-24T03:51:45Z

All recent testing has been on k8s 1.26 but we’ve been using the same config since at least 1.19.

hzxuzhonghu · 2023-07-24T04:03:14Z

	// Header contains the request header fields either received
	// by the server or to be sent by the client.
	//
	// If a server received a request with header lines,
	//
	//	Host: example.com
	//	accept-encoding: gzip, deflate
	//	Accept-Language: en-us
	//	fOO: Bar
	//	foo: two
	//
	// then
	//
	//	Header = map[string][]string{
	//		"Accept-Encoding": {"gzip, deflate"},
	//		"Accept-Language": {"en-us"},
	//		"Foo": {"Bar", "two"},
	//	}
	//
	// For incoming requests, the Host header is promoted to the
	// Request.Host field and removed from the Header map.

In golang the Host is removed from req.headers

hzxuzhonghu · 2023-07-24T04:03:30Z

Will file a fix

hzxuzhonghu · 2023-08-02T00:58:14Z

can be closed

istio-policy-bot added the area/networking label Jul 20, 2023

howardjohn added this to P0 in Prioritization Jul 20, 2023

jaellio self-assigned this Jul 20, 2023

hzxuzhonghu mentioned this issue Jul 24, 2023

Fix HTTP probe Host propagation #46130

Merged

howardjohn mentioned this issue Jul 25, 2023

Fix HTTP Probe Host propagation #46172

Closed

mmerickel changed the title ~~host header is not being sent with probes after upgrading to 1.18.1 from 1.16.1~~ host header is not being sent with probes after upgrading to 1.18.1 Jul 25, 2023

hzxuzhonghu closed this as completed Aug 2, 2023

Prioritization automation moved this from P0 to Done Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

host header is not being sent with probes after upgrading to 1.18.1 #46087

host header is not being sent with probes after upgrading to 1.18.1 #46087

mmerickel commented Jul 20, 2023 •

edited

howardjohn commented Jul 20, 2023

mmerickel commented Jul 20, 2023

jaellio commented Jul 20, 2023

mmerickel commented Jul 20, 2023

hzxuzhonghu commented Jul 21, 2023

mmerickel commented Jul 21, 2023 •

edited

mmerickel commented Jul 21, 2023

howardjohn commented Jul 21, 2023 via email

mmerickel commented Jul 22, 2023 •

edited

hzxuzhonghu commented Jul 24, 2023 •

edited

mmerickel commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

mmerickel commented Jul 24, 2023

mmerickel commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Aug 2, 2023

host header is not being sent with probes after upgrading to 1.18.1 #46087

host header is not being sent with probes after upgrading to 1.18.1 #46087

Comments

mmerickel commented Jul 20, 2023 • edited

Is this the right place to submit this?

Bug Description

Readiness probe in Deployment

Readiness probe in auto-injected Pod

ISTIO_KUBE_APP_PROBERS

Captured tcpdump

Version

Additional Information

Affected product area

howardjohn commented Jul 20, 2023

mmerickel commented Jul 20, 2023

jaellio commented Jul 20, 2023

mmerickel commented Jul 20, 2023

hzxuzhonghu commented Jul 21, 2023

mmerickel commented Jul 21, 2023 • edited

mmerickel commented Jul 21, 2023

howardjohn commented Jul 21, 2023 via email

mmerickel commented Jul 22, 2023 • edited

hzxuzhonghu commented Jul 24, 2023 • edited

mmerickel commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

mmerickel commented Jul 24, 2023

mmerickel commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Jul 24, 2023

hzxuzhonghu commented Aug 2, 2023

mmerickel commented Jul 20, 2023 •

edited

mmerickel commented Jul 21, 2023 •

edited

mmerickel commented Jul 22, 2023 •

edited

hzxuzhonghu commented Jul 24, 2023 •

edited