can't get cas issuer to work doesn't matter which way I go with number of issues

[mikejoseph-ah]

I write this issue after spending my work day on getting it to work, I am running a gke 1.22 cluster with the combo of external-dns, cert-manager, traefik, cilium, kube-prom-stack, and now want to expose grafana using this feature. (it's currently exposed but via http).

Configurations:

1. my pools and roots setup:

resource "google_privateca_ca_pool" "certs-pool" {
  name = "${var.cluster_name}-certs-pool"
  location = var.region
  tier = "DEVOPS"
  publishing_options {
    publish_ca_cert = true
    publish_crl = false
  }
  labels = {
    environment = var.environment
  }
  depends_on = [
    google_project_service.certificate-authority-service
  ]
}

resource "google_privateca_ca_pool_iam_binding" "binding" {
  ca_pool = google_privateca_ca_pool.certs-pool.id
  role = "roles/privateca.certificateRequester"
  location = var.region
  members = [
    "serviceAccount:${google_service_account.sa-google-cas-issuer.email}",
  ]
}

resource "google_privateca_certificate_authority" "certs-roots" {
  pool = google_privateca_ca_pool.certs-pool.name
  certificate_authority_id = "${var.cluster_name}-certificate-authority"
  location = var.region
  deletion_protection = false

  config  {
    subject_config  {
      subject {
        organization = "org name"
        common_name  = "${var.cluster_name}-certificate-authority"
      }
    }
    x509_config {
      ca_options {
        is_ca = true
        max_issuer_path_length = 2
      }
      key_usage {
        base_key_usage {
          cert_sign = true
          crl_sign = true
        }
        extended_key_usage {
          server_auth = false
        }
      }
    }
  }
  key_spec {
    algorithm = "EC_P384_SHA384"
  }

  depends_on = [
    google_project_service.certificate-authority-service
  ]
}

2. installed the issuer via the kubectl command (helm install method didn't work)

resource "helm_release" "cm-gci" {
  count = 0 # it is 0 because it doesn't work so disabled
  name             = "cert-manager-google-cas-issuer"
  namespace        = kubernetes_namespace.cm-ns.metadata.0.name
  chart            = "cert-manager-google-cas-issuer"
  repository       = "https://charts.jetstack.io"
  version          = "v0.6.0" # tried also without the v

  depends_on = [
    helm_release.cm,
    google_privateca_ca_pool.certs-pool,
    google_privateca_certificate_authority.certs-roots
  ]
}

I downloaded the yaml from the release section, changed resources limit from 20Mi to 90Mi

k apply -f google-cas-issuer-v0.5.3.yaml -n cert-manager

3. my issuer manifest

resource "kubectl_manifest" "cas-cluster-issuer" {
  yaml_body = <<YAML
apiVersion: cas-issuer.jetstack.io/v1beta1
kind: GoogleCASClusterIssuer
metadata:
  name: ${local.cluster_issuer_name}
  namespace: ${kubernetes_namespace.cm-ns.metadata.0.name} 
spec:
  project: ${var.project_id}
  location: ${var.region}
  caPoolId: ${google_privateca_ca_pool.certs-pool.name}
  credentials:
    name: google-cas-sa
    key: gci-credentials.json
YAML

  depends_on = [
    kubectl_manifest.google-cas-manifest
  ]
}

clusterissuer deployed successfully

dev-gke-googlecasclusterissuer, READY: True, REASON: CASClientOK, MESSAGE: Successfully constructed CAS client

4. my Certificate setups:

• method no.1, didn't work with the error from the issuer: secret "grafana-tls-xlaks" wasn't found

resource "kubectl_manifest" "grafana-certificate" {
    yaml_body = <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: grafana-tls
  namespace: ${kubernetes_namespace.monitoring-ns.metadata.0.name}
spec:
  commonName: grafana-tls-cn
  # Duration of the certificate
  duration: "2160h"
  # Renew 8 hours before the certificate expiration
  renewBefore: "360h"
  secretName: grafana-tls
  privateKey:
    algorithm: EDCSA
    size: 256
  subject:
    organizations:
    - org name
  issuerRef:
    group: cas-issuer.jetstack.io
    kind: GoogleCASClusterIssuer
    name: ${local.cluster_issuer_name}
YAML

  depends_on = [
    helm_release.monitoring-stack,
    kubectl_manifest.cas-cluster-issuer,
  ]
}

• method no.2, didn't work because a secret wasn't created and cm didn't recognize the issuer:

    ingress:  
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: ${local.cluster_issuer_name}
        cert-manager.io/issuer-kind: GoogleCASClusterIssuer
        cert-manager.io/issuer-group: cas-issuer.jetstack.io
        cert-manager.io/duration: "2160h"
        cert-manager.io/renew-before: "360h"
        acme.cert-manager.io/http01-ingress-class: traefik
        external-dns.alpha.kubernetes.io/hostname: ${local.full_dns} 
        traefik.ingress.kubernetes.io/router.entrypoints: web
        kubernetes.io/ingress.class: traefik
      tls:
        - secretName: grafana-tls
          hosts:
            - ${local.full_dns}

what am I missing here? can someone that managed to get it to work share his configurations?