API proxy strips URL query params when proxying websocket connections

[thediveo]

Tested on Kubernetes 1.14 (minikube 1.0). To reproduce:

1. deploy pod with a websocket server running:

   kubectl run wsserver --generator=run-pod/v1 --rm -i --tty --image ubuntu:disco -- bash -c \
     "apt-get update && apt-get install -y wget && \
     wget https://github.com/vi/websocat/releases/download/v1.4.0/websocat_1.4.0_ssl1.1_amd64.deb && \
     dpkg -i webso*.deb && \
     websocat -vv -s 0.0.0.0:8000"

2. in host shell outside minikube run the following websocat command to connect to the websocket server in your wsserver pod:

   websocat --binary --ws-c-uri=wss://192.168.99.100:8443/api/v1/namespaces/default/pods/wsserver:8000/proxy/test?foo=what - ws-c:cmd:'socat - ssl:192.168.99.100:8443,verify=0,cafile=$HOME/.minikube/ca.crt,cert=$HOME/.minikube/client.crt,key=$HOME/.minikube/client.key'

3. check the output of the server-side websocat and notice that the query params are missing: [DEBUG websocat::ws_server_peer] Incoming { version: Http11, subject: (Get, AbsolutePath("/test")), headers: ...

4. exec into your pod with the websocket server, and issue a local websocket connect.

   kubectl exec wsserver websocat ws://localhost:8000/test?foo=what

5. check output of the service-side websocat, now the query params are present as to be expected: [DEBUG websocat::ws_server_peer] Incoming { version: Http11, subject: (Get, AbsolutePath("/test?foo=what")), headers: ...

(note: updated instructions to be much simpler now)

[thediveo]

/sig api-machinery

[fedebongio]

/assign @cheftako

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[thediveo]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thediveo]

/remove-lifecycle stale

This is still an issue even with 1.16.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thediveo]

/remove-lifecycle stale

Still an issue.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thediveo]

/remove-lifecycle stale

[tarokkk]

Is there anything we can help about this issue?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thediveo]

/remove-lifecycle stale

[thediveo]

To reproduce with a recent version of Minikube/k8s/Ubuntu, here the updated instructions:

kubectl run wsserver --rm -i --tty --image ubuntu:jammy -- bash -c   "apt-get update && apt-get install -y wget && \
  wget https://github.com/vi/websocat/releases/download/v1.9.0/websocat_linux64 && \
  chmod u+x ./websocat_linux64 && ./websocat_linux64 -vv -s 0.0.0.0:8000"

See how the socat ws server doesn't get the query params passed:

./websocat --binary --ws-c-uri=wss://192.168.49.2:8443/api/v1/namespaces/default/pods/wsserver:8000/proxy/test?foo=what - ws-c:cmd:'socat - ssl:192.168.49.2:8443,verify=0,cafile=$HOME/.minikube/ca.crt,cert=$HOME/.minikube/profiles/minikube/client.crt,key=$HOME/.minikube/profiles/minikube/client.key'

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale

seriously, this bug is not going to puff off into thin air because some bot bots around.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale

[aojea]

one naive question, why is the query params relevant if AFAIK they are not used in any place?

[thediveo]

Finally I can give more background information since Siemens opened its Edgeshark project for capturing container traffic from stand-alone container hosts as well as k8s nodes.

Captured packets are streamed via a websocket pcapng stream. In order to establish the correct stream, some parameters need to be passed to the our packet capture streaming service ("packetflix") on the host where the target container is located. In case of k8s the plugin connecting Wireshark with the node's streaming service goes through the k8s API, using its built-in proxy functionality.

And as we found out some years ago and reported here, the k8s API proxy verb strips URL query params instead of correctly forwarding them to the service.

[aojea]

Ok, so this only happens with websockets urls 🤔 , normal request are working ok

kubectl run webapp --command --image=registry.k8s.io/e2e-test-images/agnhost:2.47 -- /agnhost netexec --http-port=8080

kubectl get --raw http://1/api/v1/namespaces/default/pods/webapp:8080/proxy/echo?msg=echoed_msg -v7
I0422 13:17:39.333422 3767678 loader.go:373] Config loaded from file:  /usr/local/google/home/aojea/.kube/config
I0422 13:17:39.333989 3767678 round_trippers.go:463] GET https://127.0.0.1:34563/api/v1/namespaces/default/pods/webapp:8080/proxy/echo?msg=echoed_msg
I0422 13:17:39.334006 3767678 round_trippers.go:469] Request Headers:
I0422 13:17:39.334015 3767678 round_trippers.go:473]     Accept: application/json, */*
I0422 13:17:39.334023 3767678 round_trippers.go:473]     User-Agent: kubectl/v1.26.15 (linux/amd64) kubernetes/aae4c29
I0422 13:17:39.336562 3767678 round_trippers.go:574] Response Status: 200 OK in 2 milliseconds
echoed_msg

based on this #120290 it seems it get lost during the upgrade

Is the apiserver doing the upgrade to websockets or is passing it through?

cc: @liggitt @seans3

[liggitt]

The proxy handlers are set up here:

• pod

  kubernetes/pkg/registry/core/pod/rest/subresources.go

  Lines 82 to 84 in 0d8f996

  	location.Path = net.JoinPreservingTrailingSlash(location.Path, proxyOpts.Path)
  	// Return a proxy handler that uses the desired transport, wrapped with additional proxy handling (to get URL rewriting, X-Forwarded-* headers, etc)
  	return newThrottledUpgradeAwareProxyHandler(location, transport, true, false, responder), nil

• service

  kubernetes/pkg/registry/core/service/proxy.go

  Lines 75 to 77 in 0d8f996

  	location.Path = net.JoinPreservingTrailingSlash(location.Path, proxyOpts.Path)
  	// Return a proxy handler that uses the desired transport, wrapped with additional proxy handling (to get URL rewriting, X-Forwarded-* headers, etc)
  	return newThrottledUpgradeAwareProxyHandler(location, transport, true, false, responder), nil

• node

  kubernetes/pkg/registry/core/node/rest/proxy.go

  Lines 79 to 81 in 0d8f996

  	location.Path = net.JoinPreservingTrailingSlash(location.Path, proxyOpts.Path)
  	// Return a proxy handler that uses the desired transport, wrapped with additional proxy handling (to get URL rewriting, X-Forwarded-* headers, etc)
  	return newThrottledUpgradeAwareProxyHandler(location, transport, true, false, responder), nil

Those only preserve the path from the incoming request, construct the target location within the kubelet handler, and construct a UpgradeAwareHandler with AppendLocationPath and UseRequestLocation set to false.

Handling for non-upgrade requests looks like this and explicitly copies the RawQuery from the request:

• kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go

  Lines 222 to 223 in dd68c5f

  	loc := *h.Location
  	loc.RawQuery = req.URL.RawQuery

Handling for upgrade requests looks like this and does not use any part of the request location unless UseRequestLocation is true:

• kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go

  Lines 321 to 339 in dd68c5f

  	location := *h.Location
  	if h.UseRequestLocation {
  	location = *req.URL
  	location.Scheme = h.Location.Scheme
  	location.Host = h.Location.Host
  	if h.AppendLocationPath {
  	location.Path = singleJoiningSlash(h.Location.Path, location.Path)
  	}
  	}
  	clone := utilnet.CloneRequest(req)
  	// Only append X-Forwarded-For in the upgrade path, since httputil.NewSingleHostReverseProxy
  	// handles this in the non-upgrade path.
  	utilnet.AppendForwardedForHeader(clone)
  	klog.V(6).Infof("Connecting to backend proxy (direct dial) %s\n Headers: %v", &location, clone.Header)
  	if h.UseLocationHost {
  	clone.Host = h.Location.Host
  	}
  	clone.URL = &location

That's why query params get dropped for upgraded requests when proxying through the API server.

[liggitt]

So... now that we know why the issue occurs ... what can we do about it?

It would be easy enough to just modify the upgradeaware.go upgrade-handling path to copy over RawQuery the same way, but we'd have to be exceptionally careful about the following things:

• are there query params already being set in h.location by the upgradeaware proxy constructor that will be stomped by whatever is in the request?
• is using parts of the incoming request even when UseRequestLocation is false breaking expectations of the caller? does that expose the backend to manipulation by the caller getting control over query params?

Another possibility would be to copy over the query from the request to the location before constructing the upgradeawareproxy (in the core//rest/ handlers). That appeals to me more, since we can reason about those and know none of them are setting query params, and we'd avoid changing low-level upgradeaware.go behavior for all callers.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thediveo]

/remove-lifecycle stale