Requests proxied from API Server to APIService obfuscate original request hostname #107435

rynowak · 2022-01-09T23:44:08Z

What happened?

Requests that are proxied by the APIServer to a custom APIService (aggregation) don't generate the X-Forwarded-Host header correctly. Each request will have both Host and X-Forwarded-Host set to ClusterIP IP address of their service. X-Forwarded-Host should be set to the original Host value sent by the client.

The result is that the APIService will not be able to determine the original host and scheme used to make the request. An APIService that wants to generate URLs that are routable for an external client cannot do so.

As an example here's some network psuedocode that reflects what I observe:

Client makes an HTTP request to the API Server using a public API Server endpoint. Let's say mycluster.mycloud.net:6443.

Initial request:

GET /apis/api.example.com/v1alpha1
Host: mycluster.mycloud.net:6443

This is routed to aggregation layer of the API Server which then reverse-proxies the request. In this case the APIService is using a ClusterIP service, which happens to have Cluster IP 10.43.176.179.

Request received by APIService:

GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: 10.43.176.179:443
<other headers added by API Server>

The thing I am saying is the bug is that X-Forwarded-Host is 10.43.176.179:443 when it should be mycluster.mycloud.net:6443.

What did you expect to happen?

I would expect X-Forwarded-Host to reflect the original hostname.

Expected: (request sent to APIService)

GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: mycluster.mycloud.net:6443
<other headers added by API Server>

Actual: (request sent to APIService)

GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: 10.43.176.179:443
<other headers added by API Server>

How can we reproduce it (as minimally and precisely as possible)?

I have put together a sample that demonstrates this by echoing the values it observes for the Host and X-Forwarded-Host headers:

Clone https://github.com/rynowak/apiservice-repro
(option) If you prefer, build a container from src/ and edit deploy.yaml to use yours
(with a Kubernetes cluster you don't mind getting dirty)

run:

kubectl apply -f deploy.yaml

Make sure APIService starts successfully

run:

kubectl get apiservice v1alpha1.api.example.com

You should see:

NAME                       SERVICE                   AVAILABLE   AGE
v1alpha1.api.example.com   default/demo-apiservice   True        1m

Open a local proxy

run:

kubectl proxy

Make a request

run:

curl http://127.0.0.1:8001/apis/api.example.com/v1alpha1

you should see something like (you will have different IP addresses):

{"host":"10.43.176.179:443","forwardedHost":"10.43.176.179:443"}

Based on this repro (kubectl proxy) I would expect 127.0.0.1:8001 to be the value of X-Forwarded-Host

Check your service

run:

kubectl get service demo-apiservice

you should see:

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-apiservice   ClusterIP   10.43.176.179   <none>        443/TCP   5m

Note that your value for CLUSTER-IP should match the IP addresses from step 6. This confirms that the APIServer proxy is using the downstream value for X-Forwarded-Host instead of the original value.

Based on this repro (kubectl proxy) I would expect 127.0.0.1:8001 to be the value of X-Forwarded-Host

This issue reproduces with and without kubectl proxy. I chose kubectl proxy for the repro steps here because it simpler to set up a repro.

Anything else we need to know?

I suspect this root cause with this code:

https://github.com/kubernetes/kube-aggregator/blob/578a094fa7fa8cdba22411e9ab008757f3ef1cb2/pkg/apiserver/handler_proxy.go#L156

At Line 156 the a new request is created with the downstream as its destination URL

At Line 167 a roundtripper is created that understands how to rewrite host headers to match the downstream for proxying. It seems like the request should use the original host value, so that it can be rewritten by the roundtripper, not the value of the downstream.

Kubernetes version

Using k3d (have also reproduced with Kind)

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7+k3s1", GitCommit:"ac70570999c566ac3507d2cc17369bb0629c1cc0", GitTreeState:"clean", BuildDate:"2021-11-29T16:40:13Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Cloud provider

(n/a)

OS version

(n/a)

Install tools

(n/a)

Container runtime (CRI) and and version (if applicable)

(n/a)

Related plugins (CNI, CSI, ...) and versions (if applicable)

(n/a)

The text was updated successfully, but these errors were encountered:

rynowak · 2022-01-09T23:46:52Z

/sig api-machinery

aojea · 2022-01-11T06:19:43Z

IIUIC the header is added by this roundtripper

kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy/transport.go

Lines 91 to 93 in a0dfd95

    
           if len(t.Host) > 0 { 
        
           	req.Header.Set("X-Forwarded-Host", t.Host) 
        
           }

The roundtripper is added in

kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go

Lines 239 to 241 in a0dfd95

    
           if h.Transport == nil || h.WrapTransport { 
        
           	h.Transport = h.defaultProxyTransport(req.URL, h.Transport) 
        
           }

by

kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go

Line 176 in a0dfd95

    
           handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})

I think you are right, but my naive question is, is there any value in the X-Forwarded-Host header?
in a kubernetes cluster all these requests will be forwarded by the apiserver, is there any case you have to discriminate by the forwarded host?

ping @liggitt for historical context, the code seems to have more than 7 years

rynowak · 2022-01-11T17:09:50Z

in a kubernetes cluster all this request will be forwarded by the apiserver, is there any case you have to discriminate by the forwareded host

The scenario that I can think of where this matters is for an APIService that needs to generate a URL that the client can understand. That's what I was trying to do but I'll readily admit that it was a niche scenario.

The blocking issue is that the server has no knowledge what host values are routable for the client. The APIService only sees the internally routable hostname (in this case the ClusterIP of its own service) which isn't meaningful outside the cluster.

liggitt · 2022-01-11T19:02:41Z

Tracked back to original addition here: https://github.com/kubernetes/kubernetes/pull/3612/files

The original motivation was a lot like what you described, though the main use I'm aware of at the time was for backing pods returning html pages to browser-based applications. That was never very successful, and dropped off sharply when basic auth support was removed.

Additionally, there's no guarantee the API host is the one the client contacted, so even if the API server forwarded the request host, that may or may not be the original one the client contacted, right?

fedebongio · 2022-01-11T21:21:20Z

/cc @deads2k @lavalamp
/triage accepted

k8s-triage-robot · 2022-04-11T21:58:20Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

lavalamp · 2022-04-11T23:56:59Z

FWIW I don't see any reason not to fix this? It'd be reasonable to set X-Forwarded-Host to the Host value from the original request, I think. Regardless of whether that e.g. names an apiserver or a load balancer.

If we're setting both to the same thing that clearly seems like a bug.

k8s-triage-robot · 2022-05-12T00:00:23Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-06-11T00:39:11Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-06-11T00:39:26Z

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lavalamp · 2022-06-13T16:56:46Z

/reopen
/lifecycle frozen

k8s-ci-robot · 2022-06-13T16:57:02Z

@lavalamp: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…served via the api-gw currently due to k8s apiserver issue: kubernetes/kubernetes#107435 it modifies the x-forwarded-host and the k8sapiserver also rewrites the html response and updates href section removing the rewrite from k8s apiserver(staging/src/k8s.io/apimachinery/pkg/util/proxy/transport.go) file and running make release-image locally pushed the result image to gcr.io/nsx-sm bumping up tag for nexus runtime Closes NPT-656 See merge request nsx-allspark_users/nexus-sdk/cli!174

…served via the api-gw currently due to k8s apiserver issue: kubernetes/kubernetes#107435 it modifies the x-forwarded-host and the k8sapiserver also rewrites the html response and updates href section removing the rewrite from k8s apiserver(staging/src/k8s.io/apimachinery/pkg/util/proxy/transport.go) file and running make release-image locally pushed the result image to gcr.io/nsx-sm Closes NPT-656 See merge request nsx-allspark_users/nexus-sdk/nexus-runtime-manifests!33

k8s-triage-robot · 2023-06-13T17:35:59Z

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

Confirm that this issue is still relevant with /triage accepted (org members only)
Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

alexzielenski · 2023-06-13T19:21:27Z

/triage accepted

To mark issue as still relevant. Tagging current api machinery leads to make them aware of it again to see if it is worth fixing soon:
/cc @jpbetz @deads2k

rynowak added the kind/bug Categorizes issue or PR as related to a bug. label Jan 9, 2022

k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 9, 2022

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 9, 2022

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 11, 2022

rynowak mentioned this issue Jan 14, 2022

Add workaround for proxy in front of API Server radius-project/radius#1732

Merged

YuxinZhou mentioned this issue Feb 25, 2022

Watcher doesn't receive expected event nor EOF for resources defined by aggregation sometimes #108344

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 11, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 12, 2022

k8s-ci-robot closed this as completed Jun 11, 2022

k8s-ci-robot reopened this Jun 13, 2022

k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Jun 13, 2022

sk593 mentioned this issue Jan 10, 2023

Parse Referer header to preserve the casing of URL Path. radius-project/radius#4930

Merged

5 tasks

rynowak mentioned this issue Jan 12, 2023

Add SDK package as home for client code radius-project/radius#4952

Merged

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. and removed triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Jun 13, 2023

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Requests proxied from API Server to APIService obfuscate original request hostname #107435

Requests proxied from API Server to APIService obfuscate original request hostname #107435

rynowak commented Jan 9, 2022 •

edited

rynowak commented Jan 9, 2022

aojea commented Jan 11, 2022 •

edited

rynowak commented Jan 11, 2022

liggitt commented Jan 11, 2022

fedebongio commented Jan 11, 2022

k8s-triage-robot commented Apr 11, 2022

lavalamp commented Apr 11, 2022

k8s-triage-robot commented May 12, 2022

k8s-triage-robot commented Jun 11, 2022

k8s-ci-robot commented Jun 11, 2022

lavalamp commented Jun 13, 2022

k8s-ci-robot commented Jun 13, 2022

k8s-triage-robot commented Jun 13, 2023

alexzielenski commented Jun 13, 2023

Requests proxied from API Server to APIService obfuscate original request hostname #107435

Requests proxied from API Server to APIService obfuscate original request hostname #107435

Comments

rynowak commented Jan 9, 2022 • edited

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Kubernetes version

Cloud provider

OS version

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

rynowak commented Jan 9, 2022

aojea commented Jan 11, 2022 • edited

rynowak commented Jan 11, 2022

liggitt commented Jan 11, 2022

fedebongio commented Jan 11, 2022

k8s-triage-robot commented Apr 11, 2022

lavalamp commented Apr 11, 2022

k8s-triage-robot commented May 12, 2022

k8s-triage-robot commented Jun 11, 2022

k8s-ci-robot commented Jun 11, 2022

lavalamp commented Jun 13, 2022

k8s-ci-robot commented Jun 13, 2022

k8s-triage-robot commented Jun 13, 2023

alexzielenski commented Jun 13, 2023

rynowak commented Jan 9, 2022 •

edited

aojea commented Jan 11, 2022 •

edited