Redirect issues with Workspaces and standalone virtual-workspace server

[ncdc]

When the partial metadata informer is trying to request /clusters/*/apis/tenancy.kcp.dev/v1beta1/workspaces from the kcp process, this gets redirected to the front proxy, and we see this error

W0819 11:41:35.676289       1 reflector.go:324] k8s.io/client-go@v0.0.0-20220803191238-b6a732dbd013/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: Get "https://<external url>/services/workspaces/%2A/apis/tenancy.kcp.dev/v1beta1/workspaces?limit=500&resourceVersion=0": x509: certificate is valid for [REDACTED], not apiserver-loopback-client

We need to figure out how to handle situations where the loopback client might get redirected to another URL where it can't validate the certificate.

Originally posted by @ncdc in #1654 (comment)

[ncdc]

FYI @sttts @p0lyn0mial
@stevekuznetsov @csams from our discussion today

[p0lyn0mial]

unfortunately, this issue hits also any controller that needs to deal with an external vw server, like the apibinding_deletion_controller (controllers use c.identityConfig = rest.CopyConfig(c.GenericConfig.LoopbackClientConfig)

I0824 13:15:01.340989   13424 apibinding_deletion_controller.go:328] "patching APIBinding" reconciler="kcp-apibindingdeletion" key="root:e2e-org-hchc2|tenancy.kcp.dev" apibinding.workspace="root:e2e-org-hchc2" apibinding.namespace="" apibinding.name="tenancy.kcp.dev" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"1573\",\"uid\":\"b6ee73d3-806c-4a65-818c-90db80ba406a\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"APIExportValid\"},{\"lastTransitionTime\":\"2022-08-24T11:15:01Z\",\"message\":\"Get \\\"https://127.0.0.1:6444/services/workspaces/root:e2e-org-hchc2/apis/tenancy.kcp.dev/v1beta1/workspaces\\\": x509: certificate is valid for localhost, not apiserver-loopback-client\",\"reason\":\"ResourceDeletionFailed\",\"severity\":\"Error\",\"status\":\"False\",\"type\":\"BindingResourceDeleteSuccess\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"BindingUpToDate\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}]}}"
E0824 13:15:01.360752   13424 apibinding_deletion_controller.go:178] deletion of apibinding root:e2e-org-hchc2|tenancy.kcp.dev failed: Get "https://127.0.0.1:6444/services/workspaces/root:e2e-org-hchc2/apis/tenancy.kcp.dev/v1beta1/workspaces": x509: certificate is valid for localhost, not apiserver-loopback-client
I0824 13:15:01.579106   13424 apibinding_deletion_controller.go:328] "patching APIBinding" reconciler="kcp-apibindingdeletion" key="root:e2e-org-hchc2:e2e-workspace-25cmk|tenancy.kcp.dev" apibinding.workspace="root:e2e-org-hchc2:e2e-workspace-25cmk" apibinding.namespace="" apibinding.name="tenancy.kcp.dev" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"1588\",\"uid\":\"21aec9fe-5d52-47c9-9e5e-134e37b8f13e\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"APIExportValid\"},{\"lastTransitionTime\":\"2022-08-24T11:15:01Z\",\"message\":\"Get \\\"https://127.0.0.1:6444/services/workspaces/root:e2e-org-hchc2:e2e-workspace-25cmk/apis/tenancy.kcp.dev/v1beta1/workspaces\\\": x509: certificate is valid for localhost, not apiserver-loopback-client\",\"reason\":\"ResourceDeletionFailed\",\"severity\":\"Error\",\"status\":\"False\",\"type\":\"BindingResourceDeleteSuccess\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"BindingUpToDate\"},{\"lastTransitionTime\":\"2022-08-24T11:14:54Z\",\"status\":\"True\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-08-24T11:14:55Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-08-24T11:14:55Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}]}}"
E0824 13:15:01.581461   13424 apibinding_deletion_controller.go:178] deletion of apibinding root:e2e-org-hchc2|tenancy.kcp.dev failed: Get "https://127.0.0.1:6444/services/workspaces/root:e2e-org-hchc2/apis/tenancy.kcp.dev/v1beta1/workspaces": x509: certificate is valid for localhost, not apiserver-loopback-client
E0824 13:15:01.666039   13424 apibinding_deletion_controller.go:178] deletion of apibinding root:e2e-org-hchc2:e2e-workspace-25cmk|tenancy.kcp.dev failed: Get "https://127.0.0.1:6444/services/workspaces/root:e2e-org-hchc2:e2e-workspace-25cmk/apis/tenancy.kcp.dev/v1beta1/workspaces": x509: certificate is valid for localhost, not apiserver-loopback-client

it also affects users accessing workspaces, i.e.

 k get --raw '/clusters/root/apis/tenancy.kcp.dev/v1beta1/workspaces'
Unable to connect to the server: x509: certificate is valid for 192.168.32.104, not 127.0.0.1

[p0lyn0mial]

Maybe we need a proxy rather than a simple redirection? The proxy could verify the vw server cert.

[ncdc]

it also affects users accessing workspaces, i.e.

k get --raw '/clusters/root/apis/tenancy.kcp.dev/v1beta1/workspaces'
Unable to connect to the server: x509: certificate is valid for 192.168.32.104, not 127.0.0.1

This is dependent upon the deployment topology. We have a topology where this URL redirects to the front proxy (which then maps it into the virtual workspaces container). In this setup, a client is able to validate the front proxy's certificate correctly and everything works.

[sttts]

This is dependent upon the deployment topology. We have a topology where this URL redirects to the front proxy (which then maps it into the virtual workspaces container)

Do we really do that? That's wrong. We must go directly to the vw address.

[sttts]

When the partial metadata informer is trying to request /clusters/*/apis/tenancy.kcp.dev/v1beta1/workspaces from the kcp process, this gets redirected to the front proxy, and we see this error

One step back: do we actually want that the ddsif lists projections?

[p0lyn0mial]

To sum up a discussion with Stefan in slack. The ddsif and the apibinding_deletion_controller shouldn't use projection resources. Instead of workspaces they should use ClusterWorkspaces.

For local development, we should use some domain name so that we can validate the server. It should be possible since we have self-signed certs and CAs.