kubectl port-forward and kubectl exec not sending "Upgrade" header

[puru86]

What happened:
Running kubectl port-forward or kubectl exec is failing with:

error: error upgrading connection: unable to upgrade connection: unable to upgrade: missing upgrade headers in request: http.Header{"Accept-Encoding":[]string{"gzip"}, "Connection":[]string{"Upgrade"}, "Content-Length":[]string{"0"}, "Kubectl-Command":[]string{"kubectl port-forward"}, "Kubectl-Session":[]string{"9cd5e8fe-beb5-4d9e-b2f8-44d4c28f27b8"}, "User-Agent":[]string{"kubectl/v1.22.16 (windows/amd64) kubernetes/b28e1f3"}, "X-Forwarded-For":[]string{"10.241.0.9, 10.105.0.12"}, "X-Stream-Protocol-Version":[]string{"portforward.k8s.io"}}

Notice that "Connection: Upgrade" has been sent, however "Upgrade: SPDY/3.1" is missing

What you expected to happen:
"Upgrade: SPDY/3.1" or equivalent upgrade header should be sent, allowing a connection upgrade to occur.

How to reproduce it (as minimally and precisely as possible):

Against Kubernetes Cluster version 1.23.8, run kubectl port-forward -n namespace services/service-name :80

Anything else we need to know?:
For some reason, if i start a kubectl proxy to this cluster, the proxying kubectl starts sending the header.
This is what i did:

1. Created a new kubeconfig context called 'proxy-local' that uses server as http://localhost:8001
2. Run kubectl proxy -v=99 in cmd-prompt-1, connecting to my actual cluster
3. In cmd-prompt-2 run kubectl port-forward --context=proxy-local -n namespaces services/service-name :80 -v=99
4. The proxying kubectl in cmd-prompt-1 now shows that it is sending the "Upgrade: SPDY/3.1" header.

I1128 10:44:46.094942    7860 proxy_server.go:91] /api/v1/namespaces/argocd/pods/argocd-server-65fdcbf96d-54b94/portforward matched ^.*
I1128 10:44:46.095475    7860 proxy_server.go:91] localhost matched ^localhost$
I1128 10:44:46.098143    7860 proxy_server.go:131] Filter accepting POST /api/v1/namespaces/argocd/pods/argocd-server-65fdcbf96d-54b94/portforward localhost
I1128 10:44:46.104152    7860 upgradeaware.go:322] Connecting to backend proxy (direct dial) https://REDACTED/api/v1/namespaces/argocd/pods/argocd-server-65fdcbf96d-54b94/portforward
  Headers: map[Connection:[Upgrade] Content-Length:[0] Kubectl-Command:[kubectl port-forward] Kubectl-Session:[b2a4d350-5558-4212-b814-03b9a9717f32] Upgrade:[SPDY/3.1] User-Agent:[kubectl/v1.22.16 (windows/amd64) kubernetes/b28e1f3] X-Forwarded-For:[127.0.0.1] X-Stream-Protocol-Version:[portforward.k8s.io]]
I1128 10:44:46.105211    7860 round_trippers.go:435] curl -v -XPOST  -H "Connection: Upgrade" -H "Kubectl-Command: kubectl port-forward" -H "Kubectl-Session: b2a4d350-5558-4212-b814-03b9a9717f32" -H "Upgrade: SPDY/3.1" -H "X-Forwarded-For: 127.0.0.1" -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.22.16 (windows/amd64) kubernetes/b28e1f3" -H "Content-Length: 0" -H "Authorization: Bearer <masked>" 'https://REDACTED/api/v1/namespaces/argocd/pods/argocd-server-65fdcbf96d-54b94/portforward'
I1128 10:44:46.105818    7860 round_trippers.go:454] POST https://REDACTED/api/v1/namespaces/argocd/pods/argocd-server-65fdcbf96d-54b94/portforward 200 OK in 0 milliseconds
I1128 10:44:46.106322    7860 round_trippers.go:460] Response Headers:
I1128 10:44:46.451529    7860 upgradeaware.go:376] Proxy upgrade error, status code 400

6. However the kubectl in cmd-prompt-2 is still not sending this header.

C:\Users\REDACTED>kubectl --context=proxy-local port-forward -n argocd services/argocd-server :80
error: error upgrading connection: unable to upgrade connection: unable to upgrade: missing upgrade headers in request: http.Header{"Accept-Encoding":[]string{"gzip"}, "Connection":[]string{"Upgrade"}, "Content-Length":[]string{"0"}, "Kubectl-Command":[]string{"kubectl port-forward"}, "Kubectl-Session":[]string{"b2a4d350-5558-4212-b814-03b9a9717f32"}, "User-Agent":[]string{"kubectl/v1.22.16 (windows/amd64) kubernetes/b28e1f3"}, "X-Forwarded-For":[]string{"127.0.0.1, REDACTED, REDACTED"}, "X-Stream-Protocol-Version":[]string{"portforward.k8s.io"}}

Environment:

• Kubernetes client and server versions (use kubectl version):
  Kubernetes Server: 1.23.8
  Kubernetes Clients checked:
  1.13.5
  1.19.0
  1.23.0
  1.23.8
  1.25.0

C:\Users\REDACTED>kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.16", GitCommit:"b28e1f370a4a4c428ddbeababcaf0198f048fcac", GitTreeState:"clean", BuildDate:"2022-11-09T13:37:37Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"83d00b7cbf10e530d1d4b2403f22413220c37621", GitTreeState:"clean", BuildDate:"2022-11-09T19:50:11Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}

C:\Users\REDACTED>kubectl_1238 version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"a12b886b1da059e0190c54d09c5eab5219dd7acf", GitTreeState:"clean", BuildDate:"2022-06-16T05:57:43Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"83d00b7cbf10e530d1d4b2403f22413220c37621", GitTreeState:"clean", BuildDate:"2022-11-09T19:50:11Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}

C:\Users\REDACTED>kubectl_1230 version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"83d00b7cbf10e530d1d4b2403f22413220c37621", GitTreeState:"clean", BuildDate:"2022-11-09T19:50:11Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}

C:\Users\REDACTED>kubectl_1250 version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"83d00b7cbf10e530d1d4b2403f22413220c37621", GitTreeState:"clean", BuildDate:"2022-11-09T19:50:11Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.23) exceeds the supported minor version skew of +/-1

• Cloud provider or hardware configuration:
  Azure Kubernetes Server (1.23.8)
• OS (e.g: cat /etc/os-release):

C:\Users\REDACTED>systeminfo

Host Name:                 REDACTED
OS Name:                   Microsoft Windows 11 Enterprise
OS Version:                10.0.22000 N/A Build 22000
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:          REDACTED
Registered Organization:   REDACTED
Product ID:                REDACTED
Original Install Date:     11/10/2022, 6:42:50 PM
System Boot Time:          11/28/2022, 6:44:21 AM
System Manufacturer:       Dell Inc.
System Model:              Latitude 5521
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 141 Stepping 1 GenuineIntel ~2496 Mhz
BIOS Version:              Dell Inc. 1.15.0, 8/9/2022
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-06:00) Central Time (US & Canada)
Total Physical Memory:     32,495 MB
Available Physical Memory: 18,027 MB
Virtual Memory: Max Size:  37,359 MB
Virtual Memory: Available: 21,859 MB
Virtual Memory: In Use:    15,500 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    REDACTED
Logon Server:              REDACTED
Hotfix(s):                 4 Hotfix(s) Installed.
                           [01]: REDACTED
                           [02]: REDACTED
                           [03]: REDACTED
                           [04]: REDACTED
Network Card(s):           3 NIC(s) Installed.
                           [01]: Cisco AnyConnect Secure Mobility Client Virtual Miniport Adapter for Windows x64
                                 Connection Name: Ethernet 3
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: REDACTED
                                 [02]: REDACTED
                                 [03]: REDACTED
                           [02]: Intel(R) Wi-Fi 6 AX201 160MHz
                                 Connection Name: Wi-Fi
                                 DHCP Enabled:    Yes
                                 DHCP Server:     REDACTED
                                 IP address(es)
                                 [01]: REDACTED
                                 [02]: REDACTED
                                 [03]: REDACTED
                                 [04]: REDACTED
                           [03]: Intel(R) Ethernet Connection (14) I219-LM
                                 Connection Name: Ethernet 2
                                 Status:          Media disconnected
Hyper-V Requirements:      VM Monitor Mode Extensions: Yes
                           Virtualization Enabled In Firmware: Yes
                           Second Level Address Translation: Yes
                           Data Execution Prevention Available: Yes

[k8s-ci-robot]

@puru86: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[puru86]

Some additional information:

1. This is only happening for users on windows 11.
2. This is only happening for clusters that are going through a bastion IP in the middle - i.e. network hopped from corporate-vpn over to Azure network.

[ardaguclu]

Some additional information:

1. This is only happening for users on windows 11.
2. This is only happening for clusters that are going through a bastion IP in the middle - i.e. network hopped from corporate-vpn over to Azure network.

Server version looks old. It would be better to try these steps on cluster whose server is up to date(1.25)

[puru86]

@ardaguclu - thanks for the response.
AKS wont release 1.25.0 as GA till Dec 2022.
Additionally - upgrading is not in my control since this is managed by an Enterprise team that has its own certification roadmap

What's weird is that this is working for other developers in my team. I have ruled out networking and routing, since we are all ultimately entering through the same ip-addresses.

What i can see is that my kubectl is not sending the Upgrade header - which is what the server seems to be rejecting it for.

The only difference b/w my setup and the others is that i am on Windows 11.

[puru86]

forgot to provide the AKS release calendar link:
AKS Release Calendar

[puru86]

Any news on this? i cant seem to get past this header issue.

[eddiezane]

@marosset @claudiubelu have y'all heard anything about this yet?

[marosset]

I haven't seen this but I normally use WSL for everything kubernetes related.
@puru86 do the headers show up on Windows 11 without going through Azure Bastion?

/sig windows
@brasmith-ms @jsturtevant in case either of you have tried this on Windows 11

[marosset]

I'll also try to set up a repo but I'll need to find a Windows 11 machine first

[puru86]

I haven't seen this but I normally use WSL for everything kubernetes related. @puru86 do the headers show up on Windows 11 without going through Azure Bastion?

/sig windows @brasmith-ms @jsturtevant in case either of you have tried this on Windows 11

Well the weird thing is that it sends the headers if i try to do this against our old on-premise clusters.

These are k8s 1.13.x (very very very old)

Since these are on-premise they dont go via Azure Bastion.

[puru86]

@marosset - whats making me scratch my head is why it doesnt work on Azure.

Its almost as if its asking the cluster about its capabilities and then doing something that breaks.

I remember an old post (maybe on MSDN?) somewhere, where they mentioned that with IE11, the "option to enable SPDY/3" had been replaced with an "option to enable HTTP/2".

I dont know what that affects though.

PS: I would love to use WSL for this :( but i have to use Cisco Any Connect, as well a MITM style corporate proxy. So virtualization comes with a lot of pain for me (and tbh quite a few road blocks)

[jsturtevant]

I also typically use WSL as well. I tried from a Windows 11 PowerShell session and was successful but I don't have proxy machine through a corporate network that I can try from.

Microsoft Windows [Version 10.0.22621.819]

 kubectl port-forward nginx-deployment-9456bbbf9-5kvdt :80
Forwarding from 127.0.0.1:61334 -> 80
Forwarding from [::1]:61334 -> 80
Handling connection for 61334
Handling connection for 61334