Pods stuck in ContainerCreating state

[abhinavagrawal1995]

Describe the bug

When running the helm chart, pan-ngfw-dep-777d6f847f-gqtqh and pan-ngfw-dep-777d6f847f-mxhtq pods are stuck in ContainerCreating status

❯ kubectl get pods
NAME                            READY   STATUS              RESTARTS   AGE
aws-node-kd2tl                  1/1     Running             0          25h
aws-node-w4dww                  1/1     Running             0          25h
coredns-65bfc5645f-5j6s8        1/1     Running             0          25h
coredns-65bfc5645f-xqtf6        1/1     Running             0          25h
kube-proxy-4pd97                1/1     Running             0          25h
kube-proxy-h2tkv                1/1     Running             0          25h
pan-cni-kvcf4                   1/1     Running             0          107m
pan-cni-p4lsb                   1/1     Running             0          107m
pan-mgmt-sts-0                  0/1     Pending             0          107m
pan-mgmt-sts-1                  0/1     Pending             0          107m
pan-ngfw-dep-777d6f847f-gqtqh   0/1     ContainerCreating   0          107m
pan-ngfw-dep-777d6f847f-mxhtq   0/1     ContainerCreating   0          107m

Name:                 pan-ngfw-dep-777d6f847f-gqtqh
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ip-192-168-85-200.ec2.internal/192.168.85.200
Start Time:           Tue, 19 Apr 2022 16:33:07 -0700
Labels:               app=pan-ngfw
                      pod-template-hash=777d6f847f
Annotations:          k8s.v1.cni.cncf.io/networks: pan-cni
                      kubernetes.io/psp: eks.privileged
                      paloaltonetworks.com/app: pan-fw
                      paloaltonetworks.com/firewall: pan-fw
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/pan-ngfw-dep-777d6f847f
Containers:
  pan-ngfw-container:
    Container ID:
    Image:         709825985650.dkr.ecr.us-east-1.amazonaws.com/palo-alto-networks/panos_cn_ngfw:10.1.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /sbin/pan_start
      newnns
      nspan-fw
      eac8617ee91
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   4Gi
    Liveness:   exec [/sbin/pan_alive_check] delay=600s timeout=1s period=5s #success=1 #failure=2
    Readiness:  exec [/sbin/pan_ready_check] delay=15s timeout=1s period=2s #success=2 #failure=1
    Environment Variables from:
      pan-ngfw-config  ConfigMap  Optional: false
    Environment:
      CPU_REQUEST:             1 (requests.cpu)
      CPU_LIMIT:               1 (limits.cpu)
      MEMORY_REQUEST:          4294967296 (requests.memory)
      MEMORY_LIMIT:            4294967296 (limits.memory)
      MY_POD_UUID:              (v1:metadata.uid)
      MY_NODE_NAME:             (v1:spec.nodeName)
      MY_POD_NAME:             pan-ngfw-dep-777d6f847f-gqtqh (v1:metadata.name)
      MY_POD_NAMESPACE:        kube-system (v1:metadata.namespace)
      MY_POD_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
      MY_POD_IP:                (v1:status.podIP)
    Mounts:
      /dev/net/tun from devnettun (rw)
      /dev/shm from dshm (rw)
      /etc/custom-ca from pancustomca (rw)
      /etc/pan-fw-sw from sw-secret (rw)
      /opt/appinfo from appinfo (rw)
      /opt/pan-cni-ready from pan-cni-ready (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5n6mh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  devnettun:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/net/tun
    HostPathType:
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  appinfo:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/pan-appinfo
    HostPathType:  Directory
  pan-cni-ready:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/pan-appinfo/pan-cni-ready
    HostPathType:  Directory
  sw-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pan-fw-sw
    Optional:    false
  pancustomca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  custom-ca-secret
    Optional:    true
  default-token-5n6mh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5n6mh
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  59m (x2 over 70m)      kubelet  Unable to attach or mount volumes: unmounted volumes=[sw-secret], unattached volumes=[pan-cni-ready devnettun dshm sw-secret pancustomca default-token-5n6mh appinfo]: timed out waiting for the condition
  Warning  FailedMount  43m (x5 over 100m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[sw-secret], unattached volumes=[dshm sw-secret pancustomca default-token-5n6mh appinfo pan-cni-ready devnettun]: timed out waiting for the condition
  Warning  FailedMount  14m (x6 over 91m)      kubelet  Unable to attach or mount volumes: unmounted volumes=[sw-secret], unattached volumes=[devnettun dshm sw-secret pancustomca default-token-5n6mh appinfo pan-cni-ready]: timed out waiting for the condition
  Warning  FailedMount  8m42s (x55 over 104m)  kubelet  MountVolume.SetUp failed for volume "sw-secret" : secret "pan-fw-sw" not found
  Warning  FailedMount  2m48s (x10 over 79m)   kubelet  Unable to attach or mount volumes: unmounted volumes=[sw-secret], unattached volumes=[sw-secret pancustomca default-token-5n6mh appinfo pan-cni-ready devnettun dshm]: timed out waiting for the condition

Expected behavior

Pod should start

Current behavior

Pod doesn't start

Your Environment

• Version used: 709825985650.dkr.ecr.us-east-1.amazonaws.com/palo-alto-networks/panos_cn_helm_charts --version 1.0.2
• Operating System and version (desktop or mobile): Desktop, Deploying on EKS.

[welcome-to-palo-alto-networks]

🎉 Thanks for opening your first issue here! Welcome to the community!

[nbansal0]

Both mgmt pods are shown as "pending" in the provided output. When mgmt pod runs, it creates the secret "pan-fw-sw" that ngfw pod mounts. Ensuring the scheduling of an mgmt pod should fix the reported issue.