Add create-federation-secret command

[lkysow]

This command will be run as a Kubernetes Job via a Helm hook. It creates
a Kubernetes secret that contains data needed by secondary datacenters
to federate with the primary. To set up a secondary dc, users will
export this secret from their primary and import it into secondaries.
They will then reference the secret in their Helm config for
secondaries.

The command works with ACLs enabled/disabled and with gossip encryption
enabled/disabled. The command only works when TLS is enabled because
federation requires TLS be enabled.

Flags worth noting:

• -export-replication-token will be set to true if ACLs are enabled. Helm chart will ensure this job can't run unless createReplicationToken: true is set (when acls are enabled) to ensure this secret exists
• -gossip-key-file will be set if gossip encryption is enabled (mounted in via helm chart)

CA differences with autoencrypt:

• When auto-encrypt is enabled, the CA we use to talk to the local consul client is different from the one we need to include in our federation secret. The fed secret needs the server ca because federation is server => server comms. But we need to use the auto-encrypt CA to talk to our client. This is handled via the -ca-file and -server-ca-cert-file flags. When auto-encrypt is not enabled, these will point to the same files but when auto-encrypt is enabled, they'll point to different files with the -ca-file flag pointing to the file created by the auto-encrypt init container.

Design requirements: Needs to be able to tolerate Consul clients not running yet, mesh gateways not created yet and replication secret not created yet because these will all be happening concurrently.

Refactoring:

• server-acl-init creates the replication secret in kube but we pull it from this job so i wanted to link the name of the kube secret between the jobs. I pulled a constant out into a new common package.
• I needed the TLS generation for my tests so I pulled it out into a test_util file in common.

Tests cover:

• base-level validation: required flags, caCert/caKey/gossipKey files not existing, replication secret missing expected key
• main run-through testing with permutations: kube namespace, acls on/off, gossip flag on/off, replication flag on/off, resource prefixes
• delayed dependencies
  • no mesh gateway instances yet created
  • replication not yet created
  • consul client not yet running
• mesh gateway specific
  • mesh gateway instances without a WAN address tag
  • confirm that mesh gateway addresses are uniqued
  • custom mesh gateway name
• auto-encrypt
• if being re-run we update the secret, e.g. in the case where the mesh gateway ips change.

Testing

Locally

You can actually run this command locally pretty easily:

• spin up cluster with mesh gateways enabled and tls enabled
• optionally add in gossip encryption, acls enabled (with createreplicationtoken=true)
• port-forward to a consul client on 8501
• pull down the generated ca cert into tls.crt
• run

  consul-k8s create-federation-secret -ca-file=tls.crt -server-ca-cert-file=tls.crt -server-ca-key-file=tls.crt -http-addr=https://localhost:8501 -resource-prefix=consul-consul -k8s-namespace=default -mesh-gateway-service-name=mesh-gateway
  

In Cluster

• Use the https://github.com/hashicorp/consul-helm/tree/create-federation-secret branch
• Use base config:

  global:
    imageK8S: lkysow/consul-k8s-dev:may1-1
    image: ishustava/consul-dev:apr-1-2020
    federation:
      enabled: true
      createFederationSecret: true
    tls:
      enabled: true
      enableAutoEncrypt: false
    acls:
      manageSystemACLs: false
      createReplicationToken: false
    #gossipEncryption:
    #  secretName: consul-gossip-encryption-key
    #  secretKey: key
  meshGateway:
    enabled: true
  connectInject:
    enabled: true
  
  

• add in auto-encrypt
• add in manageSystemACLs: true, createreplicationtoken: true
• add in gossipEncryption secret

[ishustava]

Nice work! Super clear comments everywhere and great test coverage! I left some comments as I read through the code, but haven't tested it yet.

[ishustava]

Luke, this looks great to me! Thanks for making the changes. I'm going to approve this because I don't think pulling out the command flags should be a blocker for this PR, but definitely let me know if you want me to re-review in case you get that working.

I have done my testing on the helm branch you've mentioned like so:

• Tested on a single cluster with ACLs, TLS with and without auto-encrypt, and gossip encryption enabled. I have then just compared the values in the federation secret are what I expect. I also did one additional test with custom CAs.
• I did one end-to-end test with two clusters (one on AKS, one on GKE) with ACLs, TLS with auto-encrypt, and gossip encryption all enabled. I've created static-server in dc2 and static-client in dc1. Everything worked beautifully! The user flow was so easy I didn't even realize what I had accomplished in just 20-30 minutes.

🎉 🎉 🎉