Support LoadBalancerAddress for mesh gateways

[lkysow]

Add support for using the external address of Kubernetes load balancer
for the mesh gateway wan address.

This change uses the new consul-k8s load-balancer-address command to get
the address of the load balancer.

It also removes meshGateway.wanAddress.{useNodeIP, useNodeName, host}
config values in favour of meshGateway.wanAddress.{source, static}. The
new source value allows selecting NodeIP, NodeName, LoadBalancerAddress
or Static. This is more extensible than the previous boolean values.

[lkysow]

Reviewers:

• Review Add lifecycle sidecar to mesh gateway #380 first because this PR relies on the new structure for the mesh-gateway deployment.
• Needs consul-k8s image built from Add new load-balancer-address command consul-k8s#234 and that has Add -token-sink-file flag to acl-init consul-k8s#232. I've built lkysow/consul-k8s-dev:mar18-2020-lb-address that has that.

[ishustava]

Luke, I haven't looked at the code yet, but decided to try it out first.

So far, it didn't work because the mesh gateway clusterrole needs to be updated to allow it to read services. The error I get is:

2020-03-19T22:14:46.604Z [ERROR] getting service iryna-consul-mesh-gateway: services "iryna-consul-mesh-gateway" is forbidden: User "system:serviceaccount:default:iryna-consul-mesh-gateway" cannot get resource "services" in API group "" in the namespace "default"

I also had a thought on the UX. I initially looked at the config and tried to figure out which values I need to set to enable the Load Balancer. I set meshGateway.service.enabled to true and meshGateway.service.type to LoadBalancer. But that didn't trigger the mesh gateway deployment to fetch the load balancer address. I then realized that I need to the meshGateway.wanAddress.source to LoadBalancerAddress. I'm wondering if my stupidity is an indicator that the UX could be a bit more obvious. It would be super nice if we could infer from the service settings what you most likely would want.

One idea on how to do that could be enabling service by default and adding an option to wanAddress.source called something like service. This option would imply that we'll use the service for wan address, and perhaps, even the port. It might also make sense to change the default service type from ClusterIP to something else. IIRC routable cluster IPs are pretty niche, and so I'm not sure if this is a sane default for the mesh gateway. Just an idea, but I'm curious what you think (and apologies for the long story).

[lkysow]

Ack sorry I missed that in my commit, I had it locally. You need

  - apiGroups: [""]
    resources:
      - services
    resourceNames:
      - {{ template "consul.fullname" . }}-mesh-gateway
    verbs:
      - get

I'll update the PR.

[ishustava]

Update: my tests were successful after adding the clusterrole rules. Barring what I said about UX improvements, I think this is good to go, but would like to hear what you think first.

[lkysow]

Updated to have wanAddress.source=Service be smarter. See values.yaml docs for new behaviour. Needs hashicorp/consul-k8s#235 since we now support ClusterIP services via the load-balancer-address => service-address command

[ishustava]

@lkysow I found a problem with my own suggestion to add wan_ipv4 tagged address 🤦‍♀ . When I tried it on EKS, I got this error in the service-init container:

Address "a143257ad85f14ba6a4e526cf2ae9760-2083317689.us-west-2.elb.amazonaws.com" written to address.txt successfully
Error registering service "mesh-gateway": Unexpected response code: 500 (Service tagged address "wan_ipv4" must be a valid ipv4 address)

[ishustava]

Thanks for making the changes to support the different kinds of services! I've left a couple of thoughts so far.

[lkysow]

Removed all _ipv4 addresses. I've also tested on EKS, AKS and GKE now. I've set the default containerPort to be 8443 instead of 443 because I was getting bind permission errors on GKE.

[ishustava]

Looks good (except for some failing unit tests)!