K8s retool: Introducing "Cabotage"

[ewdurbin]

This PR Is a WIP until documentation and READMEs show up... but here's the 50,000 foot view!

Disclaimer: Beware sharp bash edges :)

TLS

• Chain of trust for TLS is rooted off of the Kubernetes Internal Certificate Authority. TLS Certs for consul and vault are requested as a Kubernetes CertificateSigningRequest by the certificate-requestor initContainer in the deployments for consul and vault
• Certificate Approval is managed by a controller and a Kubernetes RBAC SubjectAccessReview is used to verify that CertificateSigningRequests are valid for signing.
• Once Vault is up and running, we can generate and sign an Intermediate Certificate Authority which will allow us to distribute TLS certificates to pods based on roles granted by our "enrollment-controller".

Consul Bootstrapping

• Consul Quorum is bootstrapped on a well known and insecure Gossip Encryption Key as well as a well known and insecure Agent Token. See https://www.consul.io/docs/agent/encryption.html for more information on Gossip Encryption.
• Consuls ACLs are then bootstrapped using the agent api. This endpoint is "tamper proof" as it can only be called once.
• Now that Consul's ACLs are in place, we can generate a new gossip key and rotate it in using the keyring operator api.
• Consul Servers and Agents are verified via mutual TLS authentication using the certificates signed by the Kubernetes Certificate Authority.

Vault Bootstrapping

• An operator using a Consul Management Token bootstraps a Consul ACL for vault.
• Once the vault server is up, an operator then bootstraps vault, optionally setting keyshare counts, unseal thresholds, a Keybase user to encrypt the initial root token to, and keybase users to encrypt shares to
• If no encryption was used for vault initialization, the initial pods are immediately unsealed.

Vault Kubernetes Auth Backend

• In order to automate distribution of Vault Tokens to pods, the Vault Kubernetes Auth Backend is mounted and configured.
• Additionally, a ServiceAccount is bound to a Kubernetes Auth Role for our enrollment-controller with the appropriate vault policies.

Cabotage "Enrollment"

• Our cabotage-enrollment-controller manages the creation and deletion of Vault Kubernetes Auth roles, Vault Consul Secret Backend roles, and Vault PKI Secret Backend roles for Kubernetes ServiceAccounts created with a specified label
• When the specified label is added to an existing ServiceAccount or a ServiceAccount is created with the label, this controller does not modify existing policies. This should allow us to use ConfigMaps to specify custom policies as needed.
• When the specified label is removed from an existing ServiceAccount or a ServiceAccount with that label is deleted, all policies are purged.

Cabotage Sidecar

• This behemoth has two entry points, kube_login intended as an initContainer, and maintain which is intended to run as a sidecar to Pods.
• As an example, a splendid Vault UI called Goldfish is deployable.
• kube_login allows for a container to fetch a vault token, storing it for later use either in clear text or as a wrapped response for another container to manage. Optionally, a TLS Certificate signed by the automated Intermediate Certificate Authority can fetched, as well as a Consul Token which has limited access to a prefix in Consul's KV store.
• The kube_login entry point is intended to execute once... meaning that anything obtained by it will expire when it's Lease expires.
• Enter the maintain side car! This sidecar will keep the Vault Token obtained by kube_login renewed, as well optionally the Consul Token and TLS certificate.
• Bonus: Clients may drop files into leases directory of known path containing Vault Lease IDs and the maintain loop will work to renew them

Goldfish

• A Goldfish was used during development as a testbed, with the big benefit of giving easy access to check Vault policy creation along the way. Shoutout to @Caiyeon.
• This deployment leverages a few features of our enrollment and sidecar!
• By applying the label specified to the cabotage-enrollment-controller to our ServiceAccount for Goldfish it is granted access!
• An initContainer fetches a token from vault with the specified Kubernetes Auth Backend role and stores it unwrapped for Goldfish to access at startup.
• Additionally, the Goldfish Pod obtains access to and uses TLS certificates issued by the Intermediate CA.
• The sidecar then runs along side the pod keeping the TLS certificates and vault token renewed.
• With minor improvements, Goldfish should be extendable to obtain all the necessary SubjectAlternativeNames in order to remove the need for the sidecar all-together.
• In an ideal world, the sidecar process is not necessary and applications retrieve, use, and renew their own Leases for secrets from Vault :).
• Even more ideal is for the kube_login initContainer to not be required to ever even have the token, but rather to write a wrapped response which the service itself can unwrap and use. This is supported by the kube_login command by supplying the --no-unwrap flag.

Known Issues:

• Vault Audit Logs must be enabled before deploying to production
• Policies for TLS Certificate Issuance are laughably lax and should be tightened up.
• ??? Please comment <3

Next Steps:

• From here, the next layer up is a service/controller for managing services! We'll use something like envconsul to fetch Environment Variable for configuration from Consul/Vault.
• Some simple UI/cli for authorized users to set but not retrieve secure environment variables via Vault.
• Some simple UI/cli for authorized user to set and retrieve plaintext environment variables via Consul.
• Ingress and LetsEncrypt!
• Automation for deployment of Dockerfiles or container images provided by users into Deployments
• Automation for rollbacks.
• Automation for 🔴/⚫️ and or 💚/💙 style rollouts.
• Automation for Canary 🐦 deployments.
• Automation for versioned configurations, including overlays/overrides for canaries.