Helm chart fails to install with TLS and Auth enabled - failed to sync secret cache: timed out waiting for the condition

[devinbost]

Describe the bug

After enabling TLS and Authentication, the helm chart fails to install. (The pods hang in an invalid state.)
The issue preventing the pods from starting appears to be this:

MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition

It is not clear why the secret cache is timing out.

To Reproduce
Here are the exact steps to reproduce this issue:

$ git clone https://github.com/apache/pulsar-helm-chart
$ cd pulsar-helm-chart
$ cat > ./examples/values-minikube.yaml
volumes:
  persistence: false
affinity:
  anti_affinity: false
components:
  autorecovery: false
zookeeper:
  replicaCount: 1
bookkeeper:
  replicaCount: 1
broker:
  replicaCount: 1
  configData:
    autoSkipNonRecoverableData: "true"
    managedLedgerDefaultEnsembleSize: "1"
    managedLedgerDefaultWriteQuorum: "1"
    managedLedgerDefaultAckQuorum: "1"
proxy:
  replicaCount: 1
tls:
  enabled: true
  bookie:
    enabled: true
  autorecovery:
    enabled: true
  toolset:
    enabled: true
  proxy:
    enabled: true
  broker:
    enabled: true
  zookeeper:
    enabled: true
auth:
  authentication:
    enabled: false
    provider: "jwt"
    jwt:
      usingSecretKey: false
  authorization:
    enabled: true
  superUsers:
    broker: "broker-admin"
    proxy: "proxy-admin"
    client: "client-admin"

(ctrl + c)

$ minikube start --memory=8192 --cpus=4
$ ./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-mini -c --pulsar-superusers superadmin,proxy-admin,broker-admin,client-admin
$ ./scripts/pulsar/upload_tls.sh -k pulsar-mini -d ./.ci/tls
$ helm install --values examples/values-minikube.yaml pulsar-mini apache/pulsar

$ kubectl get pods -n pulsar
shows them hanging in incomplete state

$ kubectl describe pods -n pulsar
shows this issue:

Warning FailedMount 7m11s kubelet MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition

Here is some additional context provided when running describe on the zookeeper pod:

Name: pulsar-mini-zookeeper-0
Namespace: pulsar
Priority: 0
Node: minikube/192.168.49.2
Start Time: Mon, 09 Nov 2020 23:45:21 -0700
Labels: app=pulsar
cluster=pulsar-mini
component=zookeeper
controller-revision-hash=pulsar-mini-zookeeper-59c4465569
release=pulsar-mini
statefulset.kubernetes.io/pod-name=pulsar-mini-zookeeper-0
Annotations: prometheus.io/port: 8000
prometheus.io/scrape: true
Status: Pending
IP:
IPs:
Controlled By: StatefulSet/pulsar-mini-zookeeper
Containers:
pulsar-mini-zookeeper:
Container ID:
Image: apachepulsar/pulsar-all:2.6.0
Image ID:
Ports: 2181/TCP, 2888/TCP, 3888/TCP, 2281/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
sh
-c
Args:
bin/apply-config-from-env.py conf/zookeeper.conf;
/pulsar/keytool/keytool.sh zookeeper ${HOSTNAME}.pulsar-mini-zookeeper.pulsar.svc.cluster.local false; bin/generate-zookeeper-config.sh conf/zookeeper.conf; bin/pulsar zookeeper;

State:          Waiting
  Reason:       ContainerCreating
Ready:          False
Restart Count:  0
Requests:
  cpu:      100m
  memory:   256Mi
Liveness:   exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=1s period=30s #success=1 #failure=10
Readiness:  exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=1s period=30s #success=1 #failure=10
Environment Variables from:
  pulsar-mini-zookeeper  ConfigMap  Optional: false
Environment:
  ZOOKEEPER_SERVERS:  pulsar-mini-zookeeper-0
Mounts:
  /pulsar/certs/ca from ca (ro)
  /pulsar/certs/zookeeper from zookeeper-certs (ro)
  /pulsar/data from pulsar-mini-zookeeper-data (rw)
  /pulsar/keytool/keytool.sh from keytool (rw,path="keytool.sh")
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-vtl5l (ro)

Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pulsar-mini-zookeeper-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
zookeeper-certs:
Type: Secret (a volume populated by a Secret)
SecretName: pulsar-mini-tls-zookeeper
Optional: false
ca:
Type: Secret (a volume populated by a Secret)
SecretName: pulsar-mini-ca-tls
Optional: false
keytool:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pulsar-mini-keytool-configmap
Optional: false
default-token-vtl5l:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-vtl5l
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 7m12s default-scheduler Successfully assigned pulsar/pulsar-mini-zookeeper-0 to minikube
Warning FailedMount 7m11s kubelet MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition
Normal Pulling 7m9s kubelet Pulling image "apachepulsar/pulsar-all:2.6.0"

Expected behavior

Installing the helm chart with the provided values should start the Pulsar cluster in minikube with TLS and authentication enabled.

Environment:

😄 minikube v1.14.2 on Darwin 10.15.7
✨ Using the docker driver based on existing profile
🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 kubectl is configured to use "minikube" by default

[devinbost]

This error is misleading. The problem was actually that the TLS certificate names didn't match. Closing this.