Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install Knative serving #2878

Open
PMarci opened this issue Jan 9, 2019 · 6 comments

Comments

Projects
None yet
5 participants
@PMarci
Copy link

commented Jan 9, 2019

Expected Behavior

I can install knative on my local Minikube cluster by following the docs.

Actual Behavior

I cannot install knative, the pods for serving never get to a Running state.

Steps to Reproduce the Problem

  1. Start minikube
minikube start --memory=8192 --cpus=4 \
--kubernetes-version=v1.12.4 \
--vm-driver=virtualbox \
--disk-size=30g \
--extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"
  1. appy istio CRDs yaml (after coming across #2195)
curl -L https://github.com/knative/serving/releases/download/v0.2.2/istio-crds.yaml \
  | sed 's/LoadBalancer/NodePort/' \
  | kubectl apply --filename -
  1. wait some time as advised
  2. apply istio.yaml
curl -L https://github.com/knative/serving/releases/download/v0.2.2/istio.yaml \
 | sed 's/LoadBalancer/NodePort/' \
 | kubectl apply --filename -
  1. after several days of trial and error, wiping minikube multiple times, istio pods are finally running stable
$ kubectl -n istio-system get pod --watch
NAME                                        READY   STATUS      RESTARTS   AGE
istio-citadel-6959fcfb88-wmp5x              1/1     Running     0          17m
istio-cleanup-secrets-hd4x8                 0/1     Completed   0          17m
istio-egressgateway-5b765869bf-fnlrt        1/1     Running     0          17m
istio-galley-7fccb9bbd9-m6jtw               1/1     Running     0          17m
istio-ingressgateway-69b597b6bd-78rnn       1/1     Running     0          17m
istio-pilot-7b594977cf-mkdv6                2/2     Running     0          17m
istio-policy-59b7f4ccd5-9sgb7               2/2     Running     0          17m
istio-sidecar-injector-5c4b6cb6bc-nww7b     1/1     Running     0          17m
istio-statsd-prom-bridge-67bbcc746c-9zbnt   1/1     Running     0          17m
istio-telemetry-7686cd76bd-hsktw            2/2     Running     0          17m
knative-ingressgateway-84d56577db-t9rps     1/1     Running     0          13m
  1. however none of the knative serving pods manage to start up after issuing
 curl -L https://github.com/knative/serving/releases/download/v0.2.2/release-lite.yaml \
  | sed 's/LoadBalancer/NodePort/' \
  | kubectl apply --filename -

(disregard the age, I copied the two outputs with some delay)

$  kubectl -n knative-serving get pod --watch
NAME                          READY   STATUS                  RESTARTS   AGE
activator-5d4b58b86d-4mzzn    0/2     Init:ImagePullBackOff   0          24m
activator-5d4b58b86d-9nxtd    0/2     Init:ErrImagePull       0          24m
activator-5d4b58b86d-rxq78    0/2     Init:ImagePullBackOff   0          24m
autoscaler-59f694cbfc-qdnfm   0/2     Init:ImagePullBackOff   0          24m
controller-c657b6496-8rv4l    0/1     ImagePullBackOff        0          24m
webhook-6f9bd9d9d7-g9xs2      0/1     ImagePullBackOff        0          24m

kubectl -n knative-serving describe pod for controller-c657b6496-8rv4l for example outputs the following:

Name:               controller-c657b6496-8rv4l
Namespace:          knative-serving
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Wed, 09 Jan 2019 15:04:48 +0100
Labels:             app=controller
                    pod-template-hash=c657b6496
Annotations:        sidecar.istio.io/inject: false
Status:             Pending
IP:                 172.17.0.20
Controlled By:      ReplicaSet/controller-c657b6496
Containers:
  controller:
    Container ID:   
    Image:          gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612
    Image ID:       
    Port:           9090/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/config-logging from config-logging (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from controller-token-7tmgk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-logging:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      config-logging
    Optional:  false
  controller-token-7tmgk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-token-7tmgk
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  28m                  default-scheduler  Successfully assigned knative-serving/controller-c657b6496-8rv4l to minikube
  Warning  Failed     25m                  kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:48139->10.0.2.3:53: i/o timeout
  Warning  Failed     19m                  kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:41699->10.0.2.3:53: i/o timeout
  Warning  Failed     14m                  kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:51605->10.0.2.3:53: i/o timeout
  Normal   Pulling    13m (x4 over 28m)    kubelet, minikube  pulling image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612"
  Warning  Failed     8m34s (x4 over 25m)  kubelet, minikube  Error: ErrImagePull
  Warning  Failed     8m34s                kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:38344->10.0.2.3:53: i/o timeout
  Normal   BackOff    7m56s (x8 over 25m)  kubelet, minikube  Back-off pulling image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612"
  Warning  Failed     7m41s (x9 over 25m)  kubelet, minikube  Error: ImagePullBackOff
  Warning  Failed     2m14s                kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/controller@sha256:bcfe127285a64f54a80ddbf00d62123a8795bbde6f7d360e4ffd86833ddc7612": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:45232->10.0.2.3:53: i/o timeout

and the same command for the webhook- pod outputs this:

Name:               webhook-6f9bd9d9d7-g9xs2
Namespace:          knative-serving
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Wed, 09 Jan 2019 15:04:48 +0100
Labels:             app=webhook
                    pod-template-hash=6f9bd9d9d7
                    role=webhook
Annotations:        sidecar.istio.io/inject: false
Status:             Pending
IP:                 172.17.0.21
Controlled By:      ReplicaSet/webhook-6f9bd9d9d7
Containers:
  webhook:
    Container ID:   
    Image:          gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/config-logging from config-logging (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from controller-token-7tmgk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-logging:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      config-logging
    Optional:  false
  controller-token-7tmgk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-token-7tmgk
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  29m                   default-scheduler  Successfully assigned knative-serving/webhook-6f9bd9d9d7-g9xs2 to minikube
  Warning  Failed     27m                   kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:39631->10.0.2.3:53: i/o timeout
  Warning  Failed     21m                   kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:49628->10.0.2.3:53: i/o timeout
  Warning  Failed     16m                   kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:40098->10.0.2.3:53: i/o timeout
  Normal   Pulling    15m (x4 over 29m)     kubelet, minikube  pulling image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525"
  Warning  Failed     10m (x4 over 27m)     kubelet, minikube  Error: ErrImagePull
  Warning  Failed     10m                   kubelet, minikube  Failed to pull image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 10.0.2.3:53: read udp 10.0.2.15:47560->10.0.2.3:53: i/o timeout
  Warning  Failed     9m47s (x8 over 27m)   kubelet, minikube  Error: ImagePullBackOff
  Normal   BackOff    4m17s (x13 over 27m)  kubelet, minikube  Back-off pulling image "gcr.io/knative-releases/github.com/knative/serving/cmd/webhook@sha256:179b194db647b976c1e9b35c26013b52a016ef1889c6466c772ef60b40723525"

Additional context
My experience with installing istio was that it's very inconsistent whether the pods are able to start up or not, running the same set of commands 3 times would often yield 3 different error states. It seems that going tabula rasa helped, now they start up more often than not. Even in such cases Knative fails to start.
EDIT : I forgot to mention that I've only managed to get istio to start properly while working from home, but not from our office (whose network consists of a single router on a residential broadband line, much like home) however I suspect this might be a red herring. I don't think that I have any special network configuration on my machine, the only thing which might be configured some way is docker, however I'm able to pull all istio and knative images which report ErrImagePull using docker pull...

Install information:

  • Platform (GKE, IKS, AKS, etc.): Minikube
$ minikube version
minikube version: v0.32.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T07:10:00Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T06:59:37Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
$ uname -a
Linux posamash 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Knative Version: v0.2.2
@vagababov

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2019

Looks like DNS problem of sorts?
Can you pull the image directly from your machine?
Can you ping gcr.io if you ssh to the pod?

@PMarci

This comment has been minimized.

Copy link
Author

commented Jan 9, 2019

Since I posted the issue I stopped the minikube machine and after starting it again with the same args it seems istio can no longer start successfully. This is the current state of the pods:

 kubectl -n istio-system get pod
NAME                                        READY   STATUS              RESTARTS   AGE
istio-citadel-6959fcfb88-td2vs              1/1     Running             0          17m
istio-cleanup-secrets-8sbp7                 0/1     Completed           0          17m
istio-egressgateway-5b765869bf-lst28        1/1     Running             2          17m
istio-galley-7fccb9bbd9-l4ldf               0/1     ContainerCreating   0          17m
istio-ingressgateway-69b597b6bd-wk4ws       1/1     Running             2          17m
istio-pilot-7b594977cf-b56cb                1/2     CrashLoopBackOff    7          17m
istio-policy-59b7f4ccd5-shnh8               2/2     Running             4          17m
istio-sidecar-injector-5c4b6cb6bc-wkzgl     0/1     ContainerCreating   0          17m
istio-statsd-prom-bridge-67bbcc746c-p65tg   1/1     Running             1          17m
istio-telemetry-7686cd76bd-xhbr4            2/2     Running             6          17m

Can you ping gcr.io if you ssh to the pod?

I am quite new to Kubernetes in general, so at first I wasn't sure what you're referring to, so I might be be doing something wrong. I didn't know which istio pod might have a shell binary, so I tried around and ended up being able to connect to the container istio-proxy

kubectl -n istio-system exec -it istio-pilot-7b594977cf-b56cb -c istio-proxy /bin/bash

However executing ping gcr.io returned

unknown host gcr.io

At this point executing

kubectl -n istio-system describe pod istio-pilot-7b594977cf-b56cb | clipcopy

yields

Name:               istio-pilot-7b594977cf-b56cb
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Wed, 09 Jan 2019 17:35:03 +0100
Labels:             app=pilot
                    istio=pilot
                    pod-template-hash=7b594977cf
Annotations:        scheduler.alpha.kubernetes.io/critical-pod: 
                    sidecar.istio.io/inject: false
Status:             Running
IP:                 172.17.0.12
Controlled By:      ReplicaSet/istio-pilot-7b594977cf
Containers:
  discovery:
    Container ID:  docker://dc7453d71571e904d733ff18bc307fe59cb5afcdfdb541bc6648ac6c5e6dbdb8
    Image:         docker.io/istio/pilot:1.0.2
    Image ID:      docker-pullable://istio/pilot@sha256:766482b916b8a3fb80aba05dc38c59c70c5e56bcc1750b582ad29cf05aee1cca
    Ports:         8080/TCP, 15010/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      discovery
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Wed, 09 Jan 2019 17:59:30 +0100
      Finished:     Wed, 09 Jan 2019 18:00:03 +0100
    Ready:          False
    Restart Count:  6
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:8080/ready delay=5s timeout=5s period=30s #success=1 #failure=3
    Environment:
      POD_NAME:                   istio-pilot-7b594977cf-b56cb (v1:metadata.name)
      POD_NAMESPACE:              istio-system (v1:metadata.namespace)
      PILOT_CACHE_SQUASH:         5
      GODEBUG:                    gctrace=2
      PILOT_PUSH_THROTTLE_COUNT:  100
      PILOT_TRACE_SAMPLING:       100
    Mounts:
      /etc/certs from istio-certs (ro)
      /etc/istio/config from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-pilot-service-account-token-vfm8w (ro)
  istio-proxy:
    Container ID:  docker://ade1ffac6cbfe5ca1eb04bbbb31941fb76e8e440b434198e1ea7a1799e4eb1f5
    Image:         docker.io/istio/proxyv2:1.0.2
    Image ID:      docker-pullable://istio/proxyv2@sha256:54e206530ba6ca9b3820254454e01b7592e9f986d27a5640b6c03704b3b68332
    Ports:         15003/TCP, 15005/TCP, 15007/TCP, 15011/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      --serviceCluster
      istio-pilot
      --templateFile
      /etc/istio/proxy/envoy_pilot.yaml.tmpl
      --controlPlaneAuthPolicy
      NONE
    State:          Running
      Started:      Wed, 09 Jan 2019 18:00:28 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Wed, 09 Jan 2019 17:56:15 +0100
      Finished:     Wed, 09 Jan 2019 17:59:40 +0100
    Ready:          True
    Restart Count:  6
    Requests:
      cpu:  10m
    Environment:
      POD_NAME:       istio-pilot-7b594977cf-b56cb (v1:metadata.name)
      POD_NAMESPACE:  istio-system (v1:metadata.namespace)
      INSTANCE_IP:     (v1:status.podIP)
    Mounts:
      /etc/certs from istio-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-pilot-service-account-token-vfm8w (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  false
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-pilot-service-account
    Optional:    true
  istio-pilot-service-account-token-vfm8w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-pilot-service-account-token-vfm8w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       27m                   default-scheduler  Successfully assigned istio-system/istio-pilot-7b594977cf-b56cb to minikube
  Warning  FailedMount     27m                   kubelet, minikube  MountVolume.SetUp failed for volume "istio-pilot-service-account-token-vfm8w" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount     27m                   kubelet, minikube  MountVolume.SetUp failed for volume "config-volume" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount     27m                   kubelet, minikube  MountVolume.SetUp failed for volume "istio-certs" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Created         25m                   kubelet, minikube  Created container
  Warning  Failed          25m                   kubelet, minikube  Failed to pull image "docker.io/istio/pilot:1.0.2": rpc error: code = Unknown desc = error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/3b/3be7ec27d893a76ccb7c63fbd52babbaba5afc9a1eb787d5779ba2bfcf6d8582/data?verify=1547054811-D3%2BzHK7LCfiob4C6%2BEYwlOlBrc8%3D: dial tcp: lookup production.cloudflare.docker.com on 10.0.2.3:53: read udp 10.0.2.15:59865->10.0.2.3:53: i/o timeout
  Normal   Started         25m                   kubelet, minikube  Started container
  Normal   Pulled          25m                   kubelet, minikube  Container image "docker.io/istio/proxyv2:1.0.2" already present on machine
  Warning  Failed          24m                   kubelet, minikube  Failed to pull image "docker.io/istio/pilot:1.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:60461->10.0.2.3:53: i/o timeout
  Normal   Pulling         24m (x3 over 27m)     kubelet, minikube  pulling image "docker.io/istio/pilot:1.0.2"
  Warning  Failed          24m (x3 over 25m)     kubelet, minikube  Error: ErrImagePull
  Warning  Failed          24m                   kubelet, minikube  Failed to pull image "docker.io/istio/pilot:1.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:34112->10.0.2.3:53: i/o timeout
  Warning  Failed          23m (x5 over 25m)     kubelet, minikube  Error: ImagePullBackOff
  Normal   BackOff         17m (x34 over 25m)    kubelet, minikube  Back-off pulling image "docker.io/istio/pilot:1.0.2"
  Normal   SandboxChanged  13m                   kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling         13m                   kubelet, minikube  pulling image "docker.io/istio/pilot:1.0.2"
  Normal   Pulled          13m                   kubelet, minikube  Successfully pulled image "docker.io/istio/pilot:1.0.2"
  Normal   Pulled          13m                   kubelet, minikube  Container image "docker.io/istio/proxyv2:1.0.2" already present on machine
  Normal   Created         13m                   kubelet, minikube  Created container
  Normal   Started         13m                   kubelet, minikube  Started container
  Normal   Pulled          10m (x3 over 12m)     kubelet, minikube  Container image "docker.io/istio/pilot:1.0.2" already present on machine
  Normal   Created         10m (x4 over 13m)     kubelet, minikube  Created container
  Normal   Started         10m (x4 over 13m)     kubelet, minikube  Started container
  Warning  Unhealthy       8m13s (x8 over 13m)   kubelet, minikube  Readiness probe failed: Get http://172.17.0.12:8080/ready: dial tcp 172.17.0.12:8080: connect: connection refused
  Warning  BackOff         3m37s (x23 over 11m)  kubelet, minikube  Back-off restarting failed container

which I find strange because I haven't changed anything between the 2 restarts I did since my post. It seems to me that either gcr.io is resolved only intermittently, or the docker images on the host/cached in the machine are used only intermittently (layman's conjecture).

As I mentioned this is the second restart since posting, and in the previous run a different set of istio pods failed to start with different messages.

Can you pull the image directly from your machine?

I guess this is a silly question, but how would I go about that? Up to this point I've only been using the STDOUT from curling the manifests.
It also occurred to me that up till now I forgot to run eval $(minikube docker-env). After running the former I'd need to re-pull failing images on my host to try this solution, right?
Or does this also involve editing the manifests' imagePullPolicy?
Thanks for your help in advance.

@PMarci

This comment has been minimized.

Copy link
Author

commented Jan 9, 2019

Running docker pull istio/pilot:1.0.2 after the eval command returns

error during connect: Post https://192.168.99.137:2376/v1.35/images/create?fromImage=istio%2Fpilot&tag=1.0.2: dial tcp 192.168.99.137:2376: connect: no route to host

Which clears up some of my questions at the bottom. I guess this is the result of my shell using docker on the minikube machine.

@PMarci

This comment has been minimized.

Copy link
Author

commented Jan 10, 2019

Today I put my setup commands into a script for convenience and ran it once on a fresh minikube machine, after having deleted ~/.minikube as well. All istio pods started up within 5m. After having confirmed that knative still can't be installed, on the next round of minikube stop && minikube delete && ./startup.sh this was the state of the istio pods:

$  kubectl -n istio-system get pod
NAME                                        READY   STATUS             RESTARTS   AGE
istio-citadel-6959fcfb88-trv7n              1/1     Running            0          5m58s
istio-cleanup-secrets-gf762                 0/1     Completed          0          5m59s
istio-egressgateway-5b765869bf-n8f2q        1/1     Running            0          5m58s
istio-galley-7fccb9bbd9-bnwsp               0/1     ImagePullBackOff   0          5m58s
istio-ingressgateway-69b597b6bd-lj8vb       1/1     Running            0          5m58s
istio-pilot-7b594977cf-ks45x                2/2     Running            0          5m58s
istio-policy-59b7f4ccd5-p7g5c               2/2     Running            0          5m58s
istio-sidecar-injector-5c4b6cb6bc-hprzn     0/1     ErrImagePull       0          5m57s
istio-statsd-prom-bridge-67bbcc746c-f8zmm   1/1     Running            0          5m58s
istio-telemetry-7686cd76bd-25nhj            2/2     Running            0          5m58s

running kubectl -n istio-system describe pod istio-sidecar-injector-5c4b6cb6bc-hprzn
gave me

Name:               istio-sidecar-injector-5c4b6cb6bc-hprzn
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Thu, 10 Jan 2019 14:19:27 +0100
Labels:             istio=sidecar-injector
                    pod-template-hash=5c4b6cb6bc
Annotations:        scheduler.alpha.kubernetes.io/critical-pod:
                    sidecar.istio.io/inject: false
Status:             Pending
IP:                 172.17.0.13
Controlled By:      ReplicaSet/istio-sidecar-injector-5c4b6cb6bc
Containers:
  sidecar-injector-webhook:
    Container ID:
    Image:         docker.io/istio/sidecar_injector:1.0.2
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      --caCertFile=/etc/istio/certs/root-cert.pem
      --tlsCertFile=/etc/istio/certs/cert-chain.pem
      --tlsKeyFile=/etc/istio/certs/key.pem
      --injectConfig=/etc/istio/inject/config
      --meshConfig=/etc/istio/config/mesh
      --healthCheckInterval=2s
      --healthCheckFile=/health
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
    Liveness:     exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=4s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Readiness:    exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=4s] delay=4s timeout=1s period=4s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/istio/certs from certs (ro)
      /etc/istio/config from config-volume (ro)
      /etc/istio/inject from inject-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-sidecar-injector-service-account-token-vb6n6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  false
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-sidecar-injector-service-account
    Optional:    false
  inject-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-sidecar-injector
    Optional:  false
  istio-sidecar-injector-service-account-token-vb6n6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-sidecar-injector-service-account-token-vb6n6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    6m29s                  default-scheduler  Successfully assigned istio-system/istio-sidecar-injector-5c4b6cb6bc-hprzn to minikube
  Warning  FailedMount  6m28s                  kubelet, minikube  MountVolume.SetUp failed for volume "certs" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  6m28s                  kubelet, minikube  MountVolume.SetUp failed for volume "istio-sidecar-injector-service-account-token-vb6n6" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  6m28s                  kubelet, minikube  MountVolume.SetUp failed for volume "inject-config" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  6m28s                  kubelet, minikube  MountVolume.SetUp failed for volume "config-volume" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  4m26s                  kubelet, minikube  Unable to mount volumes for pod "istio-sidecar-injector-5c4b6cb6bc-hprzn_istio-system(5acdbed6-14da-11e9-bd99-080027b55f8c)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"istio-sidecar-injector-5c4b6cb6bc-hprzn". list of unmounted volumes=[certs]. list of unattached volumes=[config-volume certs inject-config istio-sidecar-injector-service-account-token-vb6n6]
  Warning  FailedMount  4m20s (x8 over 6m27s)  kubelet, minikube  MountVolume.SetUp failed for volume "certs" : secret "istio.istio-sidecar-injector-service-account" not found
  Warning  Failed       119s                   kubelet, minikube  Failed to pull image "docker.io/istio/sidecar_injector:1.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:57013->10.0.2.3:53: i/o timeout
  Warning  Failed       93s                    kubelet, minikube  Failed to pull image "docker.io/istio/sidecar_injector:1.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:40348->10.0.2.3:53: i/o timeout
  Warning  Failed       81s (x2 over 119s)     kubelet, minikube  Error: ImagePullBackOff
  Normal   BackOff      81s (x2 over 119s)     kubelet, minikube  Back-off pulling image "docker.io/istio/sidecar_injector:1.0.2"
  Normal   Pulling      68s (x3 over 2m16s)    kubelet, minikube  pulling image "docker.io/istio/sidecar_injector:1.0.2"
  Warning  Failed       54s (x3 over 119s)     kubelet, minikube  Error: ErrImagePull
  Warning  Failed       54s                    kubelet, minikube  Failed to pull image "docker.io/istio/sidecar_injector:1.0.2": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:57241->10.0.2.3:53: i/o timeout

Which is especially strange, because in the same setup run issuing minikube ssh and then docker images in the VM showed

REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kubernetes-dashboard-amd64     v1.10.1             f9aed6605b81        3 weeks ago         122MB
k8s.gcr.io/kube-proxy                     v1.12.4             6d393e89739f        3 weeks ago         96.5MB
k8s.gcr.io/kube-controller-manager        v1.12.4             51b2a8e5ff78        3 weeks ago         164MB
k8s.gcr.io/kube-apiserver                 v1.12.4             c04b373449d3        3 weeks ago         194MB
k8s.gcr.io/kube-scheduler                 v1.12.4             c1b5e63c0b56        3 weeks ago         58.4MB
k8s.gcr.io/etcd                           3.2.24              3cab8e1b9802        3 months ago        220MB
istio/citadel                             1.0.2               ca4050c9fed3        4 months ago        50.7MB
istio/mixer                               1.0.2               d559bdcd7a88        4 months ago        64.5MB
istio/proxyv2                             1.0.2               50d4ec2a16fd        4 months ago        371MB
istio/pilot                               1.0.2               3be7ec27d893        4 months ago        308MB
k8s.gcr.io/coredns                        1.2.2               367cdc8433a4        4 months ago        39.2MB
k8s.gcr.io/kube-addon-manager             v8.6                9c16409588eb        10 months ago       78.4MB
prom/statsd-exporter                      v0.6.0              304735eab4e4        11 months ago       14.1MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        12 months ago       742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        14 months ago       80.8MB
quay.io/coreos/hyperkube                  v1.7.6_coreos.0     2faf6f7a322f        16 months ago       699MB

which meant that some istio images were retrieved, others not. Unsuprisingly, issuing docker pull istio/sidecar_injector:1.0.2 in the VM returned

Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:55390->10.0.2.3:53: i/o timeout`

I don't know what's going on anymore, to be honest.

@tcnghia

This comment has been minimized.

Copy link
Contributor

commented Jan 10, 2019

This looks similar to docker/for-mac#1317

@mattmoor

This comment has been minimized.

Copy link
Member

commented Jan 12, 2019

Looks like there are clearly outbound networking problems from minikube. @dlorenc who's the best PoC these days?

@mattmoor mattmoor added this to the Needs Triage milestone Jan 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.