Admission Webhook cannot work on private GKE clusters

[fcantournet]

On private GKE cluster the 0.9 version of the operator doesn't work because the validation webhook cannot be called by the kubernetes API server.

The reason is that GKE create a firewall rule to isolated the api-server and the cluster. The only traffic authorized from api-server to cluster is on ports 15090 (which is kubelets) and 443 for everything else including admission webhooks.

This is sort of related to #1335.

from this commit :
7d778e8

I understand that the change in ports to 9443 was done in order to not run as root, which is indeed a better practice.
However I suggest that we use cap_net_bind_service so the operator can bind on 443 without being root.

this could be done like so in the Dockerfile
RUN setcap 'cap_net_bind_service=+ep' /path/to/binary

[barkbay]

Thanks for reporting.
This is a known issue with GKE. GKE is configured with --enable-aggregator-routing=true for some reasons described here: kubernetes/kubernetes#79739 (comment)

The webhook is supposed to be accessed through the service IP (on port 443) but on GKE it's internally translated to endpoint IPs and routed to the relevant pods. This is where it's failing in your case.

A quick workaround is to add a firewall rule as described here: https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules You have to allow connections to 9443 from the master IP range as it is already done for the kubelet and 443.

[barkbay]

Closing for now as it is related and specific to the way GKE is configured and Google provides some documentation on this point.