OpenAPI spec fetching ignores apiserver URL path

[ash2k]

What happened?

This was originally reported in https://gitlab.com/gitlab-org/gitlab/-/issues/407161. GitLab Agent for Kubernetes, apart from other things, is a Kubernetes API reverse proxy. Users recently started getting errors when using it with kubectl v1.27.x.

Users' CI jobs get a generated kubectl-compatible config file, where server URL has a path component. For GitLab.com the address is https://kas.gitlab.com/k8s-proxy/. Looks like kubectl v1.27.x ignores the path component for some OpenAPI requests - here is an output for a kubectl apply run:

I0419 06:40:40.335640      21 loader.go:373] Config loaded from file:  /builds/jeff127/master-site.tmp/KUBECONFIG
I0419 06:40:40.337257      21 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v2?timeout=32s
I0419 06:40:40.337609      21 round_trippers.go:469] Request Headers:
I0419 06:40:40.337808      21 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:40.337996      21 round_trippers.go:473]     Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0419 06:40:40.338175      21 round_trippers.go:473]     Authorization: Bearer <masked>
I0419 06:40:40.705094      21 round_trippers.go:574] Response Status: 200 OK in 366 milliseconds
I0419 06:40:40.705471      21 round_trippers.go:577] Response Headers:
I0419 06:40:40.705674      21 round_trippers.go:580]     Cache-Control: no-cache, private
I0419 06:40:40.705825      21 round_trippers.go:580]     Audit-Id: 48e3c227-8e96-4688-8e44-c39c89f13588
I0419 06:40:40.706023      21 round_trippers.go:580]     X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0419 06:40:40.706200      21 round_trippers.go:580]     Date: Wed, 19 Apr 2023 06:40:40 GMT
I0419 06:40:40.706367      21 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: d8cba8a0-47c4-4945-ac5f-79f4e5e47b25
I0419 06:40:40.706555      21 round_trippers.go:580]     Accept-Ranges: bytes
I0419 06:40:40.706732      21 round_trippers.go:580]     Content-Type: application/octet-stream
I0419 06:40:40.706874      21 round_trippers.go:580]     Etag: "90C689D93C2CBA4FB3BB7599D386DB48F5A580DFCE512B5E5594DBE2ACC8A95791BD41AE58413A6F57288874F25A4F0A7123AF5F28101087E840B3D413D8CBA4"
I0419 06:40:40.707064      21 round_trippers.go:580]     Via: 2.0 gitlab-agent/v15.11.0/996c5f37
I0419 06:40:40.707231      21 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:40.707382      21 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 80394f72-88ac-405d-b429-678a7820d227
I0419 06:40:40.707672      21 round_trippers.go:580]     Last-Modified: Sat, 15 Apr 2023 06:42:23 GMT
I0419 06:40:40.707844      21 round_trippers.go:580]     Vary: Accept-Encoding
I0419 06:40:40.707980      21 round_trippers.go:580]     Vary: Accept
I0419 06:40:40.708169      21 round_trippers.go:580]     Vary: Accept-Encoding
I0419 06:40:40.708315      21 round_trippers.go:580]     Vary: Accept
I0419 06:40:41.204663      21 request.go:1186] Response Body:
00000000  0a 03 32 2e 30 12 16 0a  0a 4b 75 62 65 72 6e 65  |..2.0....Kuberne|
00000010  74 65 73 12 08 76 31 2e  32 34 2e 31 30 42 80 a6  |tes..v1.24.10B..|
00000020  a1 01 12 8c 02 0a 22 2f  2e 77 65 6c 6c 2d 6b 6e  |......"/.well-kn|
00000030  6f 77 6e 2f 6f 70 65 6e  69 64 2d 63 6f 6e 66 69  |own/openid-confi|
00000040  67 75 72 61 74 69 6f 6e  2f 12 e5 01 12 e2 01 0a  |guration/.......|
00000050  09 57 65 6c 6c 4b 6e 6f  77 6e 1a 57 67 65 74 20  |.WellKnown.Wget |
00000060  73 65 72 76 69 63 65 20  61 63 63 6f 75 6e 74 20  |service account |
00000070  69 73 73 75 65 72 20 4f  70 65 6e 49 44 20 63 6f  |issuer OpenID co|
00000080  6e 66 69 67 75 72 61 74  69 6f 6e 2c 20 61 6c 73  |nfiguration, als|
00000090  6f 20 6b 6e 6f 77 6e 20  61 73 20 74 68 65 20 27  |o known as the '|
000000a0  4f 49 44 43 20 64 69 73  63 6f 76 65 72 79 20 64  |OIDC discovery d|
000000b0  6f 63 27 2a 2a 67 65 74  53 65 72 76 69 63 65 41  |oc'**getServiceA|
000000c0  63 63 6f 75 6e 74 49 73  73 75 65 72 4f 70 65 6e  |ccountIssuerOpe [truncated 17871695 chars]
I0419 06:40:41.278595      21 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v3?timeout=32s
I0419 06:40:41.278962      21 round_trippers.go:469] Request Headers:
I0419 06:40:41.279159      21 round_trippers.go:473]     Accept: application/json, */*
I0419 06:40:41.279317      21 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:41.279531      21 round_trippers.go:473]     Authorization: Bearer <masked>
I0419 06:40:41.567776      21 round_trippers.go:574] Response Status: 200 OK in 288 milliseconds
I0419 06:40:41.568253      21 round_trippers.go:577] Response Headers:
I0419 06:40:41.568463      21 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: d8cba8a0-47c4-4945-ac5f-79f4e5e47b25
I0419 06:40:41.568742      21 round_trippers.go:580]     Accept-Ranges: bytes
I0419 06:40:41.568917      21 round_trippers.go:580]     Audit-Id: dc49e1ee-77b5-4dea-a139-9ee2156a637e
I0419 06:40:41.569081      21 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0419 06:40:41.569249      21 round_trippers.go:580]     Date: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.569373      21 round_trippers.go:580]     Last-Modified: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.569587      21 round_trippers.go:580]     Via: 2.0 gitlab-agent/v15.11.0/996c5f37
I0419 06:40:41.569751      21 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:41.569913      21 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 80394f72-88ac-405d-b429-678a7820d227
I0419 06:40:41.570078      21 round_trippers.go:580]     Cache-Control: no-cache, private
I0419 06:40:41.570252      21 round_trippers.go:580]     Content-Length: 14539
I0419 06:40:41.571553      21 request.go:1188] Response Body: {"paths":{".well-known/openid-configuration":{"serverRelativeURL":"/openapi/v3/.well-known/openid-configuration?hash=BF1C8CB94CEB72F9853EDF554520390667FFB0AA92BDAD649C8B70C28C0F7594D8E877E9A0682773E8301D0289D288DF118C90A0155A647A7A27AF26C02CBEC2"},"api":{"serverRelativeURL":"/openapi/v3/api?hash=2DCF3BF7BF6FAAB0A535BC865BE6C3613FAB37EB2A5650DAD1FA4DCCF5658086E5D15896598112DAB5591BF089A327C60E7A98F88D37AEB5F99DAB7E1B2704EC"},"api/v1":{"serverRelativeURL":"/openapi/v3/api/v1?hash=17C7EEA4DFF6156193C13E441C8A8BA5D552BCA97017070ADC5688AF4B3FFCA305B4708D83CF883B522E33958A3812631C33E90C5556DDF482DB7CC4D5286C6F"},"apis":{"serverRelativeURL":"/openapi/v3/apis?hash=3D55D9A351386B8FD26B41F214DD2D0C34328BA69DB2E4E0FC54EE6FC5693DF7B5094439ABE2A3ADA31CCCBB22A9FC9372769CB2ACCD3A165CC30A6D281251D6"},"apis/admissionregistration.k8s.io":{"serverRelativeURL":"/openapi/v3/apis/admissionregistration.k8s.io?hash=D3B2D7397ED883146045CE57DD02E66B442A7ED8FF25480FAFB1238044B0F1241140493C55AADB6C87AC8BB317DB28B15CBA569930FB85747AC2EF6 [truncated 13515 chars]
I0419 06:40:41.572270      21 round_trippers.go:463] GET https://kas.gitlab.com/openapi/v3/apis/apps/v1?hash=BF53D008DE76FB14B787193AFA503792A5ABCF349C100FCAC5E22D8179272576DB7C657B60A4B6D932F1E09A3B6CFE13A74E95FF2FB91F92B1D34E6621262606&timeout=32s
I0419 06:40:41.572507      21 round_trippers.go:469] Request Headers:
I0419 06:40:41.572754      21 round_trippers.go:473]     Accept: application/json
I0419 06:40:41.572926      21 round_trippers.go:473]     User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:41.573138      21 round_trippers.go:473]     Authorization: Bearer <masked>
I0419 06:40:41.596856      21 round_trippers.go:574] Response Status: 426 Upgrade Required in 23 milliseconds
I0419 06:40:41.597220      21 round_trippers.go:577] Response Headers:
I0419 06:40:41.597377      21 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0419 06:40:41.597557      21 round_trippers.go:580]     Server: gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:41.597716      21 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0419 06:40:41.597900      21 round_trippers.go:580]     Date: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.598104      21 round_trippers.go:580]     Content-Length: 76
I0419 06:40:41.598559      21 request.go:1188] Response Body: WebSocket protocol violation: Connection header "" does not contain Upgrade
error: error validating "kubernetes/staging/app.yml": error validating data: the server responded with the status code 426 but did not return more information; if you choose to ignore these errors, turn validation off with --validate=false

Important bits - request is made to https://kas.gitlab.com/openapi/v3/apis/apps/v1 but should be https://kas.gitlab.com/k8s-proxy/openapi/v3/apis/apps/v1:

I0419 06:40:41.571553      21 request.go:1188] Response Body: {"paths":{".well-known/openid-configuration":{"serverRelativeURL":"/openapi/v3/.well-known/openid-configuration?hash=BF1C8CB94CEB72F9853EDF554520390667FFB0AA92BDAD649C8B70C28C0F7594D8E877E9A0682773E8301D0289D288DF118C90A0155A647A7A27AF26C02CBEC2"},"api":{"serverRelativeURL":"/openapi/v3/api?hash=2DCF3BF7BF6FAAB0A535BC865BE6C3613FAB37EB2A5650DAD1FA4DCCF5658086E5D15896598112DAB5591BF089A327C60E7A98F88D37AEB5F99DAB7E1B2704EC"},"api/v1":{"serverRelativeURL":"/openapi/v3/api/v1?hash=17C7EEA4DFF6156193C13E441C8A8BA5D552BCA97017070ADC5688AF4B3FFCA305B4708D83CF883B522E33958A3812631C33E90C5556DDF482DB7CC4D5286C6F"},"apis":{"serverRelativeURL":"/openapi/v3/apis?hash=3D55D9A351386B8FD26B41F214DD2D0C34328BA69DB2E4E0FC54EE6FC5693DF7B5094439ABE2A3ADA31CCCBB22A9FC9372769CB2ACCD3A165CC30A6D281251D6"},"apis/admissionregistration.k8s.io":{"serverRelativeURL":"/openapi/v3/apis/admissionregistration.k8s.io?hash=D3B2D7397ED883146045CE57DD02E66B442A7ED8FF25480FAFB1238044B0F1241140493C55AADB6C87AC8BB317DB28B15CBA569930FB85747AC2EF6 [truncated 13515 chars]
I0419 06:40:41.572270      21 round_trippers.go:463] GET https://kas.gitlab.com/openapi/v3/apis/apps/v1?hash=BF53D008DE76FB14B787193AFA503792A5ABCF349C100FCAC5E22D8179272576DB7C657B60A4B6D932F1E09A3B6CFE13A74E95FF2FB91F92B1D34E6621262606&timeout=32s

The error that is reported is a red herring. The problem is that the call is made not to the proxy URL but talks to something that expects a WebSocket request.

What did you expect to happen?

Command should work normally.

How can we reproduce it (as minimally and precisely as possible)?

Maybe write a unit test that starts apiserver with a URL path and test how discovery works?

Anything else we need to know?

This is probably related to kubernetes/enhancements#3352, which was released as beta in v1.27.

Kubernetes version

v1.27.x

Cloud provider

N/A

OS version

N/A

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[ash2k]

/sig api-machinery
/sig cli

[ash2k]

FYI @seans3

[ardaguclu]

/triage accepted

[ardaguclu]

/assign

[lacibus]

I thought that this was going to be fixed in Kubernetes 1.27.2 - but this doesn't seem to be the case.

[liggitt]

I thought that this was going to be fixed in Kubernetes 1.27.2 - but this doesn't seem to be the case.

are you using updated v0.27.2 client libraries?

[lacibus]

I just updated the cluster per https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ and rebuilt a container image and ran the Gitlab agent to re-deploy it. I'm not sure where the client libraries are used in this process, but will do some research to find out.

[lacibus]

Thanks, @ligett - I had carefully updated the server, but neglected to update kubectl. Now that I have done that, the Gitlab build works fine with Kubernetes 1.27.2.