external secrets are not synched in examples/external-secrets example

[MohammadAlavi1986]

Description

The two ClusterSecretStore and SecretStore custom resources use IRSA to access AWS secret manager and SSM parameter store, but service accounts specified in theses two resources are not being created.

> kubectl get ClusterSecretStore
NAME                     AGE     STATUS                  CAPABILITIES   READY
cluster-secretstore-sm   8m47s   InvalidProviderConfig   ReadWrite      False

> kubectl get SecretStore -A
NAMESPACE          NAME             AGE     STATUS                  CAPABILITIES   READY
external-secrets   secretstore-ps   8m53s   InvalidProviderConfig   ReadWrite      False

> kubectl get sa -n external-secrets
NAME                               SECRETS   AGE
default                            0         2m23s
external-secrets-cert-controller   0         2m22s
external-secrets-sa                0         2m22s
external-secrets-webhook           0         2m22s

> kubectl get secret -A
NAMESPACE          NAME                                     TYPE                 DATA   AGE
external-secrets   external-secrets-webhook                 Opaque               4      18m
external-secrets   sh.helm.release.v1.external-secrets.v1   helm.sh/release.v1   1      18m

Unlike the local ../../modules/irsa module, the terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks module does not create Kubernetes service account resources. Since no service accounts are being created, ClusterSecretStore and SecretStore resources will be created with an InvalidProviderConfig status.

K8s service account for the external secret controller is created by the helm chart. However ClusterSecretStore and SecretStore resources are using two separate service accounts (cluster-secretstore-sa and secretstore-sa), which were used to be created by the local irsa module, but after migrating to the iam-role-for-service-accounts-eks module these two service accounts are NOT being created.

Versions

• Module version [Required]: ~> 1.0

• Terraform version: v1.2.2

• Provider version(s):

+ provider registry.terraform.io/gavinbunney/kubectl v1.14.0
+ provider registry.terraform.io/hashicorp/aws v5.4.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.2
+ provider registry.terraform.io/hashicorp/helm v2.10.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.21.1
+ provider registry.terraform.io/hashicorp/time v0.9.1
+ provider registry.terraform.io/hashicorp/tls v4.0.4

• Module version(s):

"aws-ia/eks-blueprints-addons/aws": "~> 1.0"
"terraform-aws-modules/eks/aws": "~> 19.13"
"terraform-aws-modules/vpc/aws": "~> 5.0"
"terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks": "~> 5.20"

Reproduction Code [Required]

Steps to reproduce the behavior:
Run terraform apply in examples/external-secrets directory.

Expected behaviour

> kubectl get externalsecret -A
NAMESPACE          NAME                  STORE                    REFRESH INTERVAL   STATUS         READY
external-secrets   external-secrets-ps   secretstore-ps           1h                 SecretSynced   True
external-secrets   external-secrets-sm   cluster-secretstore-sm   1h                 SecretSynced   True

> kubectl get secret -A
NAMESPACE          NAME                                     TYPE                 DATA   AGE
external-secrets   external-secrets-ps                      Opaque               2      50s
external-secrets   external-secrets-sm                      Opaque               2      50s
external-secrets   external-secrets-webhook                 Opaque               4      85s
external-secrets   sh.helm.release.v1.external-secrets.v1   helm.sh/release.v1   1      85s

Actual behaviour

> kubectl get ClusterSecretStore
NAME                     AGE     STATUS                  CAPABILITIES   READY
cluster-secretstore-sm   8m47s   InvalidProviderConfig   ReadWrite      False

> kubectl get SecretStore -A
NAMESPACE          NAME             AGE     STATUS                  CAPABILITIES   READY
external-secrets   secretstore-ps   8m53s   InvalidProviderConfig   ReadWrite      False

Terminal Output Screenshot(s)

Additional context

[bryantbiggs]

tracking in aws-ia/terraform-aws-eks-blueprints-addons#185

[alanwu4321]

My workaround had to create secretstore_sa myself, specify service_account_name in the addon, and fix secretstore_role

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = var.eks.cluster_name
  cluster_endpoint  = var.eks.cluster_endpoint
  cluster_version   = var.eks.cluster_version
  oidc_provider_arn = var.eks.oidc_provider_arn

  enable_external_secrets = true
  external_secrets = {
    # NEED TO SPECIFY
    service_account_name = local.cluster_secretstore_sa
    chart_version = "0.9.5"
    values = [
      yamlencode({ "webhook" : { "port" = "9443" } })
    ]
  }
}

# have to add service account manually since the helm chart only creates the cluster service account
resource "kubectl_manifest" "secretstore_sa" {
  yaml_body  = <<YAML
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${local.secretstore_sa}
  namespace: ${local.namespace}
  annotations:
    # secretstore_role instead of cluster store
    eks.amazonaws.com/role-arn: ${module.secretstore_role.iam_role_arn}
YAML
  depends_on = [kubectl_manifest.secretstore]
}

module "secretstore_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.20"

  role_name_prefix = "${var.eks.cluster_name}-parameter-store"

  role_policy_arns = {
    policy = aws_iam_policy.secretstore.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = var.eks.oidc_provider_arn
      # NEEDS TO BE secretstore_sa instead of cluster secret store
      namespace_service_accounts = ["${local.namespace}:${local.secretstore_sa}"]
    }
  }

  tags = local.tags
}

[jamiesweeney-t5i]

just a note for newbies ( like me ) that the main issue is not closed; just closed here and relocated.
I got confused as newish to the process and believed from above that it was resolved.
see external-secrets does not sync external secrets #185