istio-sidecar-injector stuck ContainerCreating

[Starefossen]

Describe the bug

istio-sidecar-injector is stick in ContianerCreating for fresh install of Istio v1.5.1.

$ kubectl get pods

istio-citadel-65cf6795c8-tcgp2            0/1     CrashLoopBackOff    249        20h
istio-egressgateway-5b6cb5c5-xwsrn        0/1     Running             0          20h
istio-ingressgateway-755776c4fb-24xcw     0/1     Running             0          20h
istio-operator-0                          2/2     Running             0          20h
istio-policy-f7d8d7d9f-v6n8j              2/2     Running             0          20h
istio-sidecar-injector-57f7f8b4fb-b8tvq   0/1     ContainerCreating   0          20h
istiod-797c6c44f7-7pddk                   1/1     Running             0          20h

$ kubectl describe pod istio-sidecar-injector-57f7f8b4fb-b8tvq

Events:
  Type     Reason       Age                    From                                      Message
  ----     ------       ----                   ----                                      -------
  Warning  FailedMount  8m53s (x547 over 20h)  kubelet, aks-default-75135322-vmss000000  Unable to mount volumes for pod "istio-sidecar-injector-57f7f8b4fb-b8tvq_istio-system(4b9e9367-cb86-4951-94ba-5d57c698350d)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istio-sidecar-injector-57f7f8b4fb-b8tvq". list of unmounted volumes=[certs]. list of unattached volumes=[config-volume inject-config certs istiod-ca-cert istio-sidecar-injector-service-account-token-pgtz4]
  Warning  FailedMount  3m55s (x620 over 20h)  kubelet, aks-default-75135322-vmss000000  MountVolume.SetUp failed for volume "certs" : secret "istio.istio-sidecar-injector-service-account" not found

$ kubectl logs istio-citadel-65cf6795c8-tcgp2

2020-04-29T18:37:46.550890Z     info    The custom-defined DNS name list is [istio-pilot-service-account.istio-system:istio-pilot.istio-system]
2020-04-29T18:37:46.552109Z     info    Use certificate from argument as the CA certificate
2020-04-29T18:37:46.552222Z     error   Failed to create an Citadel (error: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate signed by unknown authority))

Steps to reproduce the issue:
Please describe the steps to reproduce the issue.

apiVersion: istio.banzaicloud.io/v1beta1
kind: Istio
metadata:
  creationTimestamp: "2020-04-28T21:47:51Z"
  finalizers:
  - istio-operator.finializer.banzaicloud.io
  generation: 2
  labels:
    app.kubernetes.io/instance: istio
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: istio-aks
    controller-tools.k8s.io: "1.0"
    helm.sh/chart: istio-aks-1.0.2
  name: istio-istio-aks
  namespace: istio-system
  resourceVersion: "133898"
  selfLink: /apis/istio.banzaicloud.io/v1beta1/namespaces/istio-system/istios/istio-istio-aks
  uid: fb66a0c5-20b1-4bf2-a2a3-29a422a8cf7a
spec:
  autoInjectionNamespaces:
  - default
  autoMtls: true
  citadel:
    caSecretName: istio-ca-secret
    enableNamespacesByDefault: true
    enabled: true
    image: docker.io/istio/citadel:1.5.1
  clusterName: Kubernetes
  defaultPodDisruptionBudget:
    enabled: true
  defaultResources:
    requests:
      cpu: 10m
  galley:
    configValidation: true
    enableAnalysis: false
    enableServiceDiscovery: false
    enabled: false
    image: docker.io/istio/galley:1.5.1
    replicaCount: 1
  gateways:
    egress:
      enabled: true
      maxReplicas: 5
      minReplicas: 1
      ports:
      - name: http2
        port: 80
        targetPort: 80
      - name: https
        port: 443
        targetPort: 443
      - name: tls
        port: 15443
        targetPort: 15443
      replicaCount: 1
      resources:
        limits:
          cpu: "2"
          memory: 256Mi
        requests:
          cpu: 100m
          memory: 128Mi
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.5.1
      serviceType: ClusterIP
    enabled: true
    ingress:
      enabled: true
      loadBalancerIP: 13.81.56.197
      maxReplicas: 5
      minReplicas: 1
      ports:
      - name: status-port
        port: 15020
        targetPort: 15020
      - name: http2
        port: 80
        targetPort: 80
      - name: https
        port: 443
        targetPort: 443
      - name: tls
        port: 15443
        targetPort: 15443
      replicaCount: 1
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: 100m
          memory: 128Mi
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.5.1
      serviceType: LoadBalancer
    k8singress:
      enableHttps: false
      enabled: false
  imagePullPolicy: IfNotPresent
  includeIPRanges: '*'
  istioCoreDNS:
    enabled: false
    image: coredns/coredns:1.6.2
    pluginImage: docker.io/istio/coredns-plugin:0.2-istio-1.1
    replicaCount: 1
  istiod:
    enabled: true
  jwtPolicy: first-party-jwt
  localityLB:
    enabled: false
  logging:
    level: default:info
  meshExpansion: false
  meshPolicy:
    mtlsMode: PERMISSIVE
  mixer:
    enabled: false
    image: docker.io/istio/mixer:1.5.1
    maxReplicas: 5
    minReplicas: 1
    replicaCount: 1
    reportBatchMaxEntries: 100
    reportBatchMaxTime: 1s
    sessionAffinityEnabled: false
    stdioAdapterEnabled: false
  mixerlessTelemetry:
    enabled: true
  mountMtlsCerts: false
  multiMesh: false
  networkName: local-network
  nodeAgent:
    enabled: false
    image: docker.io/istio/node-agent-k8s:1.5.1
  outboundTrafficPolicy:
    mode: ALLOW_ANY
  pilot:
    certProvider: istiod
    enableProtocolSniffingInbound: false
    enableProtocolSniffingOutbound: true
    enabled: true
    image: docker.io/istio/pilot:1.5.1
    maxReplicas: 5
    minReplicas: 1
    replicaCount: 1
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
    sidecar: true
    traceSampling: 1
  policy:
    checksEnabled: false
    enabled: true
    image: docker.io/istio/mixer:1.5.1
    maxReplicas: 5
    minReplicas: 1
    replicaCount: 1
  proxy:
    accessLogEncoding: TEXT
    accessLogFile: /dev/stdout
    accessLogFormat: ""
    clusterDomain: cluster.local
    componentLogLevel: misc:error
    coreDumpImage: busybox
    dnsRefreshRate: 300s
    enableCoreDump: false
    envoyAccessLogService:
      enabled: false
      tcpKeepalive:
        interval: 10s
        probes: 3
        time: 10s
      tlsSettings:
        mode: DISABLE
    envoyMetricsService:
      enabled: false
      tcpKeepalive:
        interval: 10s
        probes: 3
        time: 10s
      tlsSettings:
        mode: DISABLE
    envoyStatsD:
      enabled: false
    image: docker.io/istio/proxyv2:1.5.1
    lifecycle: {}
    logLevel: warning
    protocolDetectionTimeout: 100ms
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    useMetadataExchangeFilter: false
  proxyInit:
    image: docker.io/istio/proxyv2:1.5.1
  proxyWasm:
    enabled: false
  sds:
    enabled: true
    tokenAudience: istio-ca
    udsPath: unix:/var/run/sds/uds_path
  sidecarInjector:
    autoInjectionPolicyEnabled: true
    enabled: true
    image: docker.io/istio/sidecar_injector:1.5.1
    init:
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
    initCNIConfiguration:
      binDir: /opt/cni/bin
      chained: true
      confDir: /etc/cni/net.d
      enabled: false
      excludeNamespaces:
      - istio-system
      image: docker.io/istio/install-cni:1.5.1
      logLevel: info
      repair:
        brokenPodLabelKey: cni.istio.io/uninitialized
        brokenPodLabelValue: "true"
        deletePods: true
        enabled: true
        hub: ""
        initContainerName: istio-validation
        labelPods: true
        tag: ""
    replicaCount: 1
    rewriteAppHTTPProbe: true
  telemetry:
    enabled: false
    image: docker.io/istio/mixer:1.5.1
    maxReplicas: 5
    minReplicas: 1
    replicaCount: 1
    reportBatchMaxEntries: 100
    reportBatchMaxTime: 1s
    sessionAffinityEnabled: false
  tracing:
    datadog:
      address: $(HOST_IP):8126
    enabled: false
    lightstep:
      accessToken: <access-token>
      address: lightstep-satellite.lightstep:9292
      cacertPath: /etc/lightstep/cacert.pem
      secure: true
    stackdriver: {}
    tracer: zipkin
    zipkin:
      address: zipkin.istio-system:9411
  trustDomain: cluster.local
  useMCP: false
  version: 1.5.1
status:
  ErrorMessage: gateway is pending
  Status: ReconcileFailed

Expected behavior

Expected

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem like release numberm version, branch, etc.

Istio Operator Version: v0.5.6
Istio Version: v1.5.1
Kubernetes Provider: AKS
Kubernetes Version: v1.15.10

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"059c666b8d0cce7219d2958e6ecc3198072de9bc", GitTreeState:"clean", BuildDate:"2020-04-03T15:17:29Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[Laci21]

Do you intend to use istiod or not? istiod has roughly the functionalities of Pilot, Galley, Citadel and the Sidecar Injector. So usually you should either enable istiod only and disable all other four components or you should disable istiod and enable the other components.

If you want to use istiod only here's a sample CR: https://github.com/banzaicloud/istio-operator/blob/release-1.5/config/samples/istio_v1beta1_istio.yaml

For the historical multiple component control plane model (Pilot, Galley, Citadel and the Sidecar Injector without istiod) this is an example CR: https://github.com/banzaicloud/istio-operator/blob/release-1.5/config/samples/istio_v1beta1_istio_multicontrolplane.yaml

Let me know which one you need and if the issue still persists with that!

[Starefossen]

@Laci21 thanks for the reply, yes, the intention was to use istiod but I got a little confused if the SidecarInjector was included in istiod or if I needed to enabled it like in v1.4x. It is defined in istio_v1beta1_istio.yaml you linked to but enabled.false. I'll try again without Citadel and SidecarInjecto and see if it is working :D

Thanks for making this great operator.

[audrey-brightloom]

I have the same issue - the first pod that starts in a new deployment will wait 10 min on the init-container, once thats started the other pods will start immediately after.

I have pilot and telemetry enabled because Datadog

[audrey-brightloom]

actually - my issue isn't quite the same - its the istio-init container that waits for 10 min

[Laci21]

@audrey-brightloom, could you please open a separate issue and share more details of your environment and setup similarly as @Starefossen did? In that case we can take a closer look and possibly replicate your issue.

[Laci21]

@Laci21 thanks for the reply, yes, the intention was to use istiod but I got a little confused if the SidecarInjector was included in istiod or if I needed to enabled it like in v1.4x. It is defined in istio_v1beta1_istio.yaml you linked to but enabled.false. I'll try again without Citadel and SidecarInjecto and see if it is working :D

Thanks for making this great operator.

@Starefossen, have you managed to solve the issue with the suggestion above or does the issue still persist?

[Laci21]

Please reopen if the issue still persists with the recommend solution.