Enable configure Connect Injector and Controller Webhooks to be manag…

…ed by Vault (#1191) * refactored TestVault * Fixing name of CreateConnectCARootAndIntermediatePIKPolicy to CreateConnectCARootAndIntermediatePKIPolicy * refactored all except WAN Fed has error * fixing vault wan fed test * comment formatting * PR Feedback. Moving vault helper functions to be on the structs they were taking as arguments. * PR Feedback. changing name of Save() onkv2secret to something moredescriptive. adding comment string. * Do not load webhook cert manager when vault is enabled. * do not mount volumes when using vault * configuring vault injector * Fixing linting * Pods are all running * Fixing cert-dir paths for vault and non-vault use * fixing volume mount * fixing volume mount again * adding bats tests for tls cert directory * fixing cert for tls-cert dir * Adding logic to tests for controller tls cert. also adding snapshot agent and vault namespaces tests * it works...with hardocded stuff...need to refactor from here * refactoring out the updating of the webhook config * adding missing file * adding tests for webhook-cert-manager * refining webhook cert manager to remove setting of global.enablePodSecurityPolicies * adding connect-inject bats tests * adding test for controller and use of resource-prefix * adding tests for mwc update code * configure controller to only update mwc with ca bundle when using vault as a secrets backend. * configure connect inject to only update mwc with ca bundle when using vault as a secrets backend. * fixing lint errors for unnahdled errors * embedding tlsCertDir for controller and connectInject under vault in values.yaml * embedding tlsCertDir for controller and connectInject under vault in values.yaml - fixing tests * change vault role for controller to come from global.secretsBackend.vault.consulControllerCARole * fixing vault namespaces and snapshot agent on vault acceptance tests * fixing VAULT_TLSAUtoReload test * rebased fromvault refactor. static server replicaset has error about unknown authority. * Working using separate CAs for connect-injector and controller. only vault test is configured. * adding failure if .Values.global.secretsBackend.vault.consulConnectInjectCARole, .Values.global.secretsBackend.vault.connectInject.tlsCert, .Values.global.secretsBackend.vault.connectInject.caCert are either all not set or all set. * updating chart with caCert configs * enforcing setting both controller and connectInject CA and tls vault settings all at once. * correcting connect inject tests * Update controller tests * fixing condition on which web certmanager is shut off * only rendering vault role when suppled in controller and connect-inject deployments * fixing connect inject deploy for vault role * making global.secretsBackend.vault.consulCARole the fallback in the controller and connect-inject deployments if the web cert replacement roles are not defined. * Updating the doc string for TestVault_WebhookCerts * correct rebasing issues * removing unneccessary format changes. refactoring consul.serverTLSCATemplate to use consul.vaultCATemplate * updating test descriptions for webhook-cert-manager resources * updating connect-inject-clusterrole and controller-clusterrole tests * updated maybeFailValuesIfVaultWebhookCertSettingsAreIncomplete * Adding Changelog * Update maybeFailValuesIfVaultWebhookCertSettingsAreIncomplete to validateVaultWebhookCertConfiguration * Updating wbhook vault test to make sure that webhook-cert-manager is not deployed. * Fixing the validation that webhook-cert-manager is not running in the TestVault_WebhookCerts test * Renamed consulControllerCARole to consulControllerRole and consulContronnectInjectCARole to consulConnectInjectRole * fixing linting * Renamed configureCABundleUpdate() to updateWebhookCABundle() * Make ca.crt a constant * Adding doc strings for webhook certs secretName * updated alt_names for controller and connect inject deployments to be the same as they are under web-cert-manager. updated path of where webhook certs get saved. * Change mutatingwebhookconfigurationswhen to mutatingwebhookconfigurations when * added test cases for vault to controller test * Webhook certs vault test - checking cert rotation. currently failing. * moved vault webhook stuff into main vault test and deleted the webhook cert specific test. * getting rid of lint error * refactoring long conditional in webhook-cert-manager files into a variable for readability * addressing PR feedback * Apply suggestions from code review Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com> * Fixing broken test with retry change * Update charts/consul/templates/_helpers.tpl Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com> * Removing 127.0.0.1 from ip_sans * Removing reference to common_name: Consul Webhook Certificates Service * Removing a dangle reference to Consul Webhook Certificates Service * adding 127.0.0.1 back into server ip_sans * making common name the name of the service for connect-inject and controller * Update the description for enable-webhook-ca-update flag in control-plane/subcommand/controller/command.go Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com> * Dropping the consul prefix from consulConnectInjectRole and consulControllerRole * Updating values.yaml file descriptions for connectInject and controller under vault. * Updating cert expiry in logging in vault test from RPC expiry. Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
hashicorp · Jun 13, 2022 · a16c4ee · a16c4ee
1 parent f685c9c
commit a16c4ee
Show file tree

Hide file tree

Showing 33 changed files with 1,816 additions and 95 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,8 @@
 ## UNRELEASED
 
 IMPROVEMENTS:
+* Control Plane
+  * Enable configuring Connect Injector and Controller Webhooks' certificates to be managed by Vault. [[GH-1191](https://github.com/hashicorp/consul-k8s/pull/1191/)]
 * Helm
   * Enable the configuring of snapshot intervals in the client snapshot agent via `client.snapshotAgent.interval`. [[GH-1235](https://github.com/hashicorp/consul-k8s/pull/1235)]
   * Enable configuring the pod topologySpreadConstraints for mesh, terminating, and ingress gateways. [[GH-1257](https://github.com/hashicorp/consul-k8s/pull/1257)]

diff --git a/acceptance/framework/consul/helm_cluster.go b/acceptance/framework/consul/helm_cluster.go
@@ -274,12 +274,16 @@ func (h *HelmCluster) Upgrade(t *testing.T, helmValues map[string]string) {
 }
 
 func (h *HelmCluster) CreatePortForwardTunnel(t *testing.T, remotePort int) string {
-	localPort := terratestk8s.GetAvailablePort(t)
 	serverPod := fmt.Sprintf("%s-consul-server-0", h.releaseName)
+	return h.CreatePortForwardTunnelToResourcePort(t, serverPod, remotePort)
+}
+
+func (h *HelmCluster) CreatePortForwardTunnelToResourcePort(t *testing.T, resourceName string, remotePort int) string {
+	localPort := terratestk8s.GetAvailablePort(t)
 	tunnel := terratestk8s.NewTunnelWithLogger(
 		h.helmOptions.KubectlOptions,
 		terratestk8s.ResourceTypePod,
-		serverPod,
+		resourceName,
 		localPort,
 		remotePort,
 		h.logger)

diff --git a/acceptance/framework/vault/helpers.go b/acceptance/framework/vault/helpers.go
@@ -33,8 +33,8 @@ func GenerateGossipSecret() (string, error) {
 func ConfigurePKICerts(t *testing.T,
 	vaultClient *vapi.Client, baseUrl, allowedSubdomain, roleName, ns, datacenter,
 	maxTTL string) string {
-	allowedDomains := fmt.Sprintf("%s.consul,%s,%s.%s,%s.%s.svc", datacenter,
-		allowedSubdomain, allowedSubdomain, ns, allowedSubdomain, ns)
+	allowedDomains := fmt.Sprintf("%s.consul,%s,%s.%s,%s.%s.svc,%s.default.svc.cluster.local", datacenter,
+		allowedSubdomain, allowedSubdomain, ns, allowedSubdomain, ns, allowedSubdomain)
 	params := map[string]interface{}{
 		"allowed_domains":    allowedDomains,
 		"allow_bare_domains": "true",

diff --git a/acceptance/tests/controller/controller_test.go b/acceptance/tests/controller/controller_test.go
@@ -6,25 +6,42 @@ import (
 	"testing"
 	"time"
 
+	"github.com/hashicorp/consul-k8s/acceptance/framework/config"
 	"github.com/hashicorp/consul-k8s/acceptance/framework/consul"
+	"github.com/hashicorp/consul-k8s/acceptance/framework/environment"
 	"github.com/hashicorp/consul-k8s/acceptance/framework/helpers"
 	"github.com/hashicorp/consul-k8s/acceptance/framework/k8s"
 	"github.com/hashicorp/consul-k8s/acceptance/framework/logger"
+	"github.com/hashicorp/consul-k8s/acceptance/framework/vault"
 	"github.com/hashicorp/consul/api"
 	"github.com/hashicorp/consul/sdk/testutil/retry"
+	"github.com/hashicorp/go-uuid"
 	"github.com/stretchr/testify/require"
 )
 
+const (
+	KubernetesAuthMethodPath = "kubernetes"
+	ManageSystemACLsRole     = "server-acl-init"
+	ClientRole               = "client"
+	ServerRole               = "server"
+)
+
 func TestController(t *testing.T) {
 	cfg := suite.Config()
 
 	cases := []struct {
 		secure      bool
 		autoEncrypt bool
+		useVault    bool
 	}{
-		{false, false},
-		{true, false},
-		{true, true},
+		{false, false, false},
+		{true, false, false},
+		{true, true, false},
+		{true, true, true},
+		{false, false, true},
+		// Vault with TLS requires autoEncrypt set to true as well, so the below
+		// is not valid
+		// {true, false, true},
 	}
 
 	// The name of a service intention in consul is
@@ -46,11 +63,22 @@ func TestController(t *testing.T) {
 			}
 
 			releaseName := helpers.RandomName()
+
+			var bootstrapToken string
+			var helmConsulValues map[string]string
+			if c.useVault {
+				helmConsulValues, bootstrapToken = configureAndGetVaultHelmValues(t, ctx, cfg, releaseName, c.secure)
+				helpers.MergeMaps(helmConsulValues, helmValues)
+			}
 			consulCluster := consul.NewHelmCluster(t, helmValues, ctx, cfg, releaseName)
 
 			consulCluster.Create(t)
 			consulClient, _ := consulCluster.SetupConsulClient(t, c.secure)
 
+			if c.useVault {
+				consulCluster.ACLToken = bootstrapToken
+			}
+
 			// Test creation.
 			{
 				logger.Log(t, "creating custom resources")
@@ -340,3 +368,198 @@ func TestController(t *testing.T) {
 		})
 	}
 }
+
+func configureAndGetVaultHelmValues(t *testing.T, ctx environment.TestContext,
+	cfg *config.TestConfig, consulReleaseName string, secure bool) (map[string]string, string) {
+	vaultReleaseName := helpers.RandomName()
+	ns := ctx.KubectlOptions(t).Namespace
+
+	vaultCluster := vault.NewVaultCluster(t, ctx, cfg, vaultReleaseName, nil)
+	vaultCluster.Create(t, ctx, "")
+	// Vault is now installed in the cluster.
+
+	// Now fetch the Vault client so we can create the policies and secrets.
+	vaultClient := vaultCluster.VaultClient(t)
+
+	// -------------------------
+	// PKI
+	// -------------------------
+	// Configure Service Mesh CA
+	connectCAPolicy := "connect-ca-dc1"
+	connectCARootPath := "connect_root"
+	connectCAIntermediatePath := "dc1/connect_inter"
+	// Configure Policy for Connect CA
+	vault.CreateConnectCARootAndIntermediatePKIPolicy(t, vaultClient, connectCAPolicy, connectCARootPath, connectCAIntermediatePath)
+
+	// Configure Server PKI
+	serverPKIConfig := &vault.PKIAndAuthRoleConfiguration{
+		BaseURL:             "pki",
+		PolicyName:          "consul-ca-policy",
+		RoleName:            "consul-ca-role",
+		KubernetesNamespace: ns,
+		DataCenter:          "dc1",
+		ServiceAccountName:  fmt.Sprintf("%s-consul-%s", consulReleaseName, "server"),
+		AllowedSubdomain:    fmt.Sprintf("%s-consul-%s", consulReleaseName, "server"),
+		MaxTTL:              "1h",
+		AuthMethodPath:      "kubernetes",
+		CommonName:          "Consul CA",
+	}
+	serverPKIConfig.ConfigurePKIAndAuthRole(t, vaultClient)
+
+	webhookCertTtl := 25 * time.Second
+	// Configure controller webhook PKI
+	controllerWebhookPKIConfig := &vault.PKIAndAuthRoleConfiguration{
+		BaseURL:             "controller",
+		PolicyName:          "controller-ca-policy",
+		RoleName:            "controller-ca-role",
+		KubernetesNamespace: ns,
+		DataCenter:          "dc1",
+		ServiceAccountName:  fmt.Sprintf("%s-consul-%s", consulReleaseName, "controller"),
+		AllowedSubdomain:    fmt.Sprintf("%s-consul-%s", consulReleaseName, "controller-webhook"),
+		MaxTTL:              webhookCertTtl.String(),
+		AuthMethodPath:      "kubernetes",
+	}
+	controllerWebhookPKIConfig.ConfigurePKIAndAuthRole(t, vaultClient)
+
+	// Configure controller webhook PKI
+	connectInjectorWebhookPKIConfig := &vault.PKIAndAuthRoleConfiguration{
+		BaseURL:             "connect",
+		PolicyName:          "connect-ca-policy",
+		RoleName:            "connect-ca-role",
+		KubernetesNamespace: ns,
+		DataCenter:          "dc1",
+		ServiceAccountName:  fmt.Sprintf("%s-consul-%s", consulReleaseName, "connect-injector"),
+		AllowedSubdomain:    fmt.Sprintf("%s-consul-%s", consulReleaseName, "connect-injector"),
+		MaxTTL:              webhookCertTtl.String(),
+		AuthMethodPath:      "kubernetes",
+	}
+	connectInjectorWebhookPKIConfig.ConfigurePKIAndAuthRole(t, vaultClient)
+
+	// -------------------------
+	// KV2 secrets
+	// -------------------------
+	// Gossip key
+	gossipKey, err := vault.GenerateGossipSecret()
+	require.NoError(t, err)
+	gossipSecret := &vault.KV2Secret{
+		Path:       "consul/data/secret/gossip",
+		Key:        "gossip",
+		Value:      gossipKey,
+		PolicyName: "gossip",
+	}
+	gossipSecret.SaveSecretAndAddReadPolicy(t, vaultClient)
+
+	// License
+	licenseSecret := &vault.KV2Secret{
+		Path:       "consul/data/secret/license",
+		Key:        "license",
+		Value:      cfg.EnterpriseLicense,
+		PolicyName: "license",
+	}
+	if cfg.EnableEnterprise {
+		licenseSecret.SaveSecretAndAddReadPolicy(t, vaultClient)
+	}
+
+	// Bootstrap Token
+	bootstrapToken, err := uuid.GenerateUUID()
+	require.NoError(t, err)
+	bootstrapTokenSecret := &vault.KV2Secret{
+		Path:       "consul/data/secret/bootstrap",
+		Key:        "token",
+		Value:      bootstrapToken,
+		PolicyName: "bootstrap",
+	}
+	bootstrapTokenSecret.SaveSecretAndAddReadPolicy(t, vaultClient)
+
+	// -------------------------
+	// Additional Auth Roles
+	// -------------------------
+	serverPolicies := fmt.Sprintf("%s,%s,%s,%s", gossipSecret.PolicyName, connectCAPolicy, serverPKIConfig.PolicyName, bootstrapTokenSecret.PolicyName)
+	if cfg.EnableEnterprise {
+		serverPolicies += fmt.Sprintf(",%s", licenseSecret.PolicyName)
+	}
+
+	// server
+	consulServerRole := ServerRole
+	srvAuthRoleConfig := &vault.KubernetesAuthRoleConfiguration{
+		ServiceAccountName:  serverPKIConfig.ServiceAccountName,
+		KubernetesNamespace: ns,
+		AuthMethodPath:      KubernetesAuthMethodPath,
+		RoleName:            consulServerRole,
+		PolicyNames:         serverPolicies,
+	}
+	srvAuthRoleConfig.ConfigureK8SAuthRole(t, vaultClient)
+
+	// client
+	consulClientRole := ClientRole
+	consulClientServiceAccountName := fmt.Sprintf("%s-consul-%s", consulReleaseName, ClientRole)
+	clientAuthRoleConfig := &vault.KubernetesAuthRoleConfiguration{
+		ServiceAccountName:  consulClientServiceAccountName,
+		KubernetesNamespace: ns,
+		AuthMethodPath:      KubernetesAuthMethodPath,
+		RoleName:            consulClientRole,
+		PolicyNames:         gossipSecret.PolicyName,
+	}
+	clientAuthRoleConfig.ConfigureK8SAuthRole(t, vaultClient)
+
+	// manageSystemACLs
+	manageSystemACLsRole := ManageSystemACLsRole
+	manageSystemACLsServiceAccountName := fmt.Sprintf("%s-consul-%s", consulReleaseName, ManageSystemACLsRole)
+	aclAuthRoleConfig := &vault.KubernetesAuthRoleConfiguration{
+		ServiceAccountName:  manageSystemACLsServiceAccountName,
+		KubernetesNamespace: ns,
+		AuthMethodPath:      KubernetesAuthMethodPath,
+		RoleName:            manageSystemACLsRole,
+		PolicyNames:         bootstrapTokenSecret.PolicyName,
+	}
+	aclAuthRoleConfig.ConfigureK8SAuthRole(t, vaultClient)
+
+	// allow all components to access server ca
+	srvCAAuthRoleConfig := &vault.KubernetesAuthRoleConfiguration{
+		ServiceAccountName:  "*",
+		KubernetesNamespace: ns,
+		AuthMethodPath:      KubernetesAuthMethodPath,
+		RoleName:            serverPKIConfig.RoleName,
+		PolicyNames:         serverPKIConfig.PolicyName,
+	}
+	srvCAAuthRoleConfig.ConfigureK8SAuthRole(t, vaultClient)
+
+	vaultCASecret := vault.CASecretName(vaultReleaseName)
+
+	consulHelmValues := map[string]string{
+		"server.extraVolumes[0].type": "secret",
+		"server.extraVolumes[0].name": vaultCASecret,
+		"server.extraVolumes[0].load": "false",
+
+		"global.secretsBackend.vault.enabled":              "true",
+		"global.secretsBackend.vault.consulServerRole":     consulServerRole,
+		"global.secretsBackend.vault.consulClientRole":     consulClientRole,
+		"global.secretsBackend.vault.consulCARole":         serverPKIConfig.RoleName,
+		"global.secretsBackend.vault.manageSystemACLsRole": manageSystemACLsRole,
+
+		"global.secretsBackend.vault.ca.secretName": vaultCASecret,
+		"global.secretsBackend.vault.ca.secretKey":  "tls.crt",
+	}
+
+	if cfg.EnableEnterprise {
+		consulHelmValues["global.enterpriseLicense.secretName"] = licenseSecret.Path
+		consulHelmValues["global.enterpriseLicense.secretKey"] = licenseSecret.Key
+	}
+
+	if secure {
+		consulHelmValues["server.serverCert.secretName"] = serverPKIConfig.CertPath
+		consulHelmValues["global.tls.caCert.secretName"] = serverPKIConfig.CAPath
+		consulHelmValues["global.secretsBackend.vault.connectInject.tlsCert.secretName"] = connectInjectorWebhookPKIConfig.CertPath
+		consulHelmValues["global.secretsBackend.vault.connectInject.caCert.secretName"] = connectInjectorWebhookPKIConfig.CAPath
+		consulHelmValues["global.secretsBackend.vault.controller.tlsCert.secretName"] = controllerWebhookPKIConfig.CertPath
+		consulHelmValues["global.secretsBackend.vault.controller.caCert.secretName"] = controllerWebhookPKIConfig.CAPath
+		consulHelmValues["global.secretsBackend.vault.connectInjectRole"] = connectInjectorWebhookPKIConfig.RoleName
+		consulHelmValues["global.secretsBackend.vault.controllerRole"] = controllerWebhookPKIConfig.RoleName
+		consulHelmValues["global.acls.bootstrapToken.secretName"] = bootstrapTokenSecret.Path
+		consulHelmValues["global.acls.bootstrapToken.secretKey"] = bootstrapTokenSecret.Key
+		consulHelmValues["global.gossipEncryption.secretName"] = gossipSecret.Path
+		consulHelmValues["global.gossipEncryption.secretKey"] = gossipSecret.Key
+	}
+
+	return consulHelmValues, bootstrapToken
+}