Rancher Kubernetes Engine with Hashicorp Vault

Trousseau running on RKE with Hashicorp Vault

Table of contents

• Installation with HashiCorp Vault
• Setup HashiCorp Vault
  • Requirements
  • Shell Environment Variables
  • Enable a Transit Engine
• Kubernetes
  • RKE Specifics
• Setup monitoring

Installation with HashiCorp Vault

Requirements

The following are required:

• a HashiCorp Vault instance (Community or Enterprise)
• a HashiCorp Vault token
• a SSH access to the control plane nodes as an admin
• the necessary user permissions to handle files in etc and restart serivces, root is best, sudo is better ;)
• the vault cli tool
• the kubectl cli tool

Setup HashiCorp Vault

Shell Environment Variables

Export environment variables to reach out the HashiCorp Vault instance:

export VAULT_ADDR="https://addresss:8200"
export VAULT_TOKEN="s.oYpiOmnWL0PFDPS2ImJTdhRf.CxT2N"

NOTE: when using the HashiCorp Vault Enterprise, the concept of namespace is introduced.
This requires an additional environment variables to target the base root namespace:

export VAULT_NAMESPACE=admin

or a sub namespace like admin/gke01

export VAULT_NAMESPACE=admin/gke01

Enable a Transit engine

Make sure to have a Transit engine enable within Vault:

vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

List the secret engines:

vault secrets list
Path          Type            Accessor                 Description
----          ----            --------                 -----------
cubbyhole/    ns_cubbyhole    ns_cubbyhole_491a549d    per-token private secret storage
identity/     ns_identity     ns_identity_01d57d96     identity store
sys/          ns_system       ns_system_d0f157ca       system endpoints used for control, policy and debugging
transit/      transit         transit_3a41addc         n/a

NOTE about missing VAULT_NAMESPACE
Not exporting the VAULT_NAMESPACE will results in a similar error message when enabling the transit engine or even trying to list them:

vault secrets enable transit

Error enabling: Error making API request.

URL: POST https://vault-dev.vault.3c414da7-6890-49b8-b635-e3808a5f4fee.aws.hashicorp.cloud:8200/v1/sys/mounts/transit
Code: 403. Errors:

* 1 error occurred:
        * permission denied

Finally, create a transit key:

vault write -f transit/keys/vault-kms-demo
Success! Data written to: transit/keys/vault-kms-demo

Kubernetes

RKE Specifics

When deploying using rke (not RKE2) and after successfuly deploying a working kubernetes using your cluster.yml with rke up, modify the following sections of your cluster.yml:

the kube-api section:

  kube-api:
    image: ""
    extra_args:
      encryption-provider-config: /opt/vault-kms/encryption_config.yaml
    extra_binds: 
      - "/opt/vault-kms:/opt/vault-kms"

the kubelet section:

  kubelet:
    image: ""
    extra_args: 
      pod-manifest-path: "/etc/kubernetes/manifests"
    extra_binds: 
      - "/opt/vault-kms:/opt/vault-kms"

Once everything in place, perform a rke up to reload the configuration.

Setup monitoring

Trousseau is coming with a Prometheus endpoint for monitoring with basic Grafana dashboard.

An example of configuration for the Prometheus endpoint access is available within the folder scripts/templates/monitoring with the name prometheus.yaml.

An example of configuration for the Grafana dashboard configuration is available within the folder scripts/templates/monitoring with the name grafana-dashboard.yaml.