Support ACL replication

[lkysow]

Summary

• Adds flag -acl-replication-token-file for setting the ACL replication
  token. This token is used by secondary dc's to create ACL policies.
  This turns on ACL replication mode.
  In this mode we will not bootstrap ACLs since we expect replication
  to be running.
• Modifies various policies and tokens to only be applicable to the
  local datacenter. These policies should have been only local before.
• If running in a secondary DC, append the datacenter name to the
  policy name. This is required because policies must be globally
  unique.
• Note: we aren't sharing policies between datacenters because
  each server-acl-init could modify the policy depending on
  its local config.
• Adds agent:read permissions to the replication token which is needed to get the current datacenter

Test Description

I took an incremental approach to the tests rather than having all the existing tests take another permutation of replication enabled. I did this because the replication changes are incremental and because I thought adding that permutation to all the tests would make them overly complicated and harder to debug when they failed since multiple consul servers are involved and there's a ton of logs to pick through.

How to test

DC1

• Bring up two kube clusters
• Checkout the wan-fed-acls Helm branch
• Run helm install using primary-config.yaml (see below)
• Wait for mesh gateway svc to get external ip (kubectl get svc consul-mesh-gateway )
• Edit primary-dc.yaml and update meshGateway.wanAddress.host to that IP. Run helm upgrade.
• Export SSL certs and acl token

  kubectl get secret consul-ca-cert -o yaml > consul-ca-cert.yaml
  kubectl get secret consul-ca-key -o yaml > consul-ca-key.yaml
  kubectl get secret consul-acl-replication-acl-token -o yaml > consul-acl-replication-acl-token.yaml
  

• Note down mesh gateway IP

DC2

• Switch kubectl context to dc2
• Edit secondary-dc.yaml and update server.extraConfig.primaryGateways to the primary's mesh gateway IP
• Import the secrets: kubectl apply -f consul-ca-cert.yaml -f consul-ca-key.yaml -f consul-acl-replication-acl-token.yaml
• Run helm install using secondary-dc.yaml (see below)
• Wait for mesh gateway svc to get external ip (kubectl get svc consul-mesh-gateway )
• Edit secondary-dc.yaml and update meshGateway.wanAddress.host to that IP. Run helm upgrade.
• 🤞 it works!

primary-dc.yaml

global:
  image: lkysow/consul:1.8.0-beta1-mar10
  imageK8S: lkysow/consul-k8s-dev:mar10-2020-wanfed
  tls:
    enabled: true
  bootstrapACLs: true
  acls:
    createReplicationToken: true
  federation:
    enabled: true
  name: consul
server:
  replicas: 1
  bootstrapExpect: 1
  extraConfig: |
    {
      "log_level": "debug"
    }
client:
  extraConfig: |
    {
      "log_level": "debug"
    }
connectInject:
  enabled: true
  imageEnvoy: envoyproxy/envoy-alpine:v1.13.0
meshGateway:
  enabled: true
  replicas: 1
  enableHealthChecks: false
  wanAddress:
    useNodeIP: false
    host: "52.188.143.194"
  service:
    enabled: true
    type: LoadBalancer
  imageEnvoy: envoyproxy/envoy-alpine:v1.13.0

secondary-dc.yaml

global:
  datacenter: dc2
  image: lkysow/consul:1.8.0-beta1-mar10
  imageK8S: lkysow/consul-k8s-dev:mar10-2020-wanfed
  tls:
    enabled: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
    caKey:
      secretName: consul-ca-key
      secretKey: tls.key
  bootstrapACLs: true
  acls:
    replicationToken:
      secretName: consul-acl-replication-acl-token
      secretKey: token
  federation:
    enabled: true
  name: consul
server:
  replicas: 1
  bootstrapExpect: 1
  extraConfig: |
    {
      "primary_datacenter": "dc1",
      "primary_gateways": ["52.188.143.194:443"],
      "log_level": "debug"
    }
client:
  extraConfig: |
    {
      "log_level": "debug"
    }
connectInject:
  enabled: true
  imageEnvoy: envoyproxy/envoy-alpine:v1.13.0
meshGateway:
  enabled: true
  replicas: 1
  enableHealthChecks: false
  wanAddress:
    useNodeIP: false
    host: "52.143.82.105"
  service:
    enabled: true
    type: LoadBalancer
  imageEnvoy: envoyproxy/envoy-alpine:v1.13.0

[lkysow]

NOTE: I will put up a follow-up PR for configuring the anonymous policy to allow cross-dc calls to work.

[ishustava]

Luke, awesome work so far! I've reviewed the code and left some comments inline.

Now onto testing it with the Helm chart!

[ishustava]

Approving since my testing with the Helm chart PR were all successful 🎉 There are a couple of comments that are still unresolved, but I don't think they are blocking. Great job!