Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time out error when downloading images #16545

Closed
4 of 17 tasks
fragpic opened this issue Apr 4, 2020 · 13 comments
Closed
4 of 17 tasks

Time out error when downloading images #16545

fragpic opened this issue Apr 4, 2020 · 13 comments
Labels
area/chectl Issues related to chectl, the CLI of Che kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.

Comments

@fragpic
Copy link

fragpic commented Apr 4, 2020

Describe the bug

Times out when scheduling/downloading images.
Logs indicate timeout waiting for postgres volumes to be bound

Che version

  • latest

Steps to reproduce

ran
chectl server:start --platform minikube --multiuser

Expected behavior

Eclipse che is deployed and a url is generated

Runtime

  • kubernetes (include output of kubectl version)
  • Openshift (include output of oc version)
  • minikube (include output of minikube version and kubectl version)
  • minishift (include output of minishift version and oc version)
  • docker-desktop + K8S (include output of docker version and kubectl version)
  • other: (please specify)

minikube version: v1.9.1
commit: d8747aec7ebf8332ddae276d5f8fb42d3152b5a1

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Screenshots

✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: Found v1.18.0.
✔ Verify if minikube ingress addon is enabled
✔ Enable minikube ingress addon
✔ Retrieving minikube IP and domain for ingress URLs...172.17.0.2.nip.io.
Eclipse Che logs will be available in '/tmp/chectl-logs/1586030573786'
✔ Start following logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che logs...done
✔ Start following Postgres logs...done
✔ Start following Keycloak logs...done
✔ Start following Plugin registry logs...done
✔ Start following Devfile registry logs...done
✔ Start following events
✔ Start following namespace events...done
✔ 🏃‍ Running the Eclipse Che operator
✔ Copying operator resources...done.
✔ Create Namespace (che)...It already exists.
✔ Create ServiceAccount che-operator in namespace che...It already exists.
✔ Create Role che-operator in namespace che...It already exists.
✔ Create ClusterRole che-operator...It already exists.
✔ Create RoleBinding che-operator in namespace che...It already exists.
✔ Create ClusterRoleBinding che-operator...It already exists.
✔ Create CRD checlusters.org.eclipse.che...It already exists.
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
✔ Create deployment che-operator in namespace che...It already exists.
✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
❯ ✅ Post installation checklist
❯ Eclipse Che pod bootstrap
✖ scheduling
→ ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
downloading images
starting
Retrieving Eclipse Che server URL
Eclipse Che status check
› Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
› Installation failed, check logs in '/tmp/chectl-logs/1586030573786'

Installation method

  • chectl <-->chectl server:start --platform minikube --multiuser

Environment

  • [x ] my computer
    • Windows
    • Linux
    • macOS
  • Cloud
    • Amazon
    • Azure
    • GCE
    • other (please specify)
  • other: please specify

Eclipse Che Logs

time="2020-04-04T20:02:45Z" level=info msg="Default 'info' log level is applied"
time="2020-04-04T20:02:45Z" level=info msg="Go Version: go1.12.12"
time="2020-04-04T20:02:45Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-04-04T20:02:45Z" level=info msg="operator-sdk Version: v0.5.0"
time="2020-04-04T20:02:45Z" level=info msg="Operator is running on Kubernetes"
time="2020-04-04T20:02:45Z" level=info msg="Registering Che Components Types"
time="2020-04-04T20:02:45Z" level=info msg="Starting the Cmd"
time="2020-04-04T20:02:45Z" level=info msg="Waiting for PVC postgres-data to be bound. Default timeout: 10 seconds"
time="2020-04-04T20:02:55Z" level=warning msg="Timeout waiting for a PVC postgres-data to be bound. Current phase is Pending"
time="2020-04-04T20:02:55Z" level=warning msg="Sometimes PVC can be bound only when the first consumer is created"
time="2020-04-04T20:02:56Z" level=info msg="Waiting for deployment postgres. Default timeout: 420 seconds"

LAST SEEN TYPE REASON OBJECT MESSAGE
22m Normal Scheduled pod/che-operator-7b9fd956cb-fwbt8 Successfully assigned che/che-operator-7b9fd956cb-fwbt8 to minikube
22m Normal Pulling pod/che-operator-7b9fd956cb-fwbt8 Pulling image "quay.io/eclipse/che-operator:7.10.0"
21m Normal Pulled pod/che-operator-7b9fd956cb-fwbt8 Successfully pulled image "quay.io/eclipse/che-operator:7.10.0"
21m Normal Created pod/che-operator-7b9fd956cb-fwbt8 Created container che-operator
21m Normal Started pod/che-operator-7b9fd956cb-fwbt8 Started container che-operator
18s Normal SandboxChanged pod/che-operator-7b9fd956cb-fwbt8 Pod sandbox changed, it will be killed and re-created.
16s Normal Pulling pod/che-operator-7b9fd956cb-fwbt8 Pulling image "quay.io/eclipse/che-operator:7.10.0"
13s Normal Pulled pod/che-operator-7b9fd956cb-fwbt8 Successfully pulled image "quay.io/eclipse/che-operator:7.10.0"
13s Normal Created pod/che-operator-7b9fd956cb-fwbt8 Created container che-operator
13s Normal Started pod/che-operator-7b9fd956cb-fwbt8 Started container che-operator
22m Normal SuccessfulCreate replicaset/che-operator-7b9fd956cb Created pod: che-operator-7b9fd956cb-fwbt8
22m Normal ScalingReplicaSet deployment/che-operator Scaled up replica set che-operator-7b9fd956cb to 1
6m46s Warning FailedScheduling pod/postgres-6448d66f7f-2hn8w running "VolumeBinding" filter plugin for pod "postgres-6448d66f7f-2hn8w"$
9s Warning FailedScheduling pod/postgres-6448d66f7f-2hn8w running "VolumeBinding" filter plugin for pod "postgres-6448d66f7f-2hn8w"$
21m Normal SuccessfulCreate replicaset/postgres-6448d66f7f Created pod: postgres-6448d66f7f-2hn8w
6m30s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
21m Normal ScalingReplicaSet deployment/postgres Scaled up replica set postgres-6448d66f7f to 1
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Warning FailedScheduling pod/postgres-6448d66f7f-2hn8w running "VolumeBinding" filter plugin for pod "postgres-6448d66f7f-2hn8w"$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Warning FailedScheduling pod/postgres-6448d66f7f-2hn8w running "VolumeBinding" filter plugin for pod "postgres-6448d66f7f-2hn8w"$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner "k8s.i$
0s Normal ExternalProvisioning persistentvolumeclaim/postgres-data waiting for a volume to be created, either by external provisioner
@fragpic fragpic added the kind/bug Outline of a bug - must adhere to the bug report template. label Apr 4, 2020
@che-bot che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Apr 4, 2020
@tolusha
Copy link
Contributor

tolusha commented Apr 6, 2020

@cbyreddy
That's might be the cause. kubernetes/minikube#7218
pls. downgrade minikube to v1.8 and try again.

@ericwill ericwill added area/chectl Issues related to chectl, the CLI of Che severity/P1 Has a major impact to usage or development of the system. and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Apr 6, 2020
@gattytto
Copy link

gattytto commented Apr 9, 2020

storage provision error, yes. Workaround is to use the storageClassName in crd:

minikube creates a VM for setting up the cluster so /data and /data/wksp have to be created and chmod 777 in the vm for this to work. Sames goes to whatever path you choose if you modify this values.

SIDE NOTE: this could also require to disable default tls option in yaml too:
tlsSupport: false

SIDE NOTE2: also the domain should be forced in yaml:
ingressDomain: 'minikube-lan-ip.nip.io'

#file: /usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml
postgresPVCStorageClassName: eclipseche
workspacePVCStorageClassName: eclipsechewksp
ingressDomain: 'minikube-lan-ip.nip.io' #CHANGE TO A REAL minikube-lan-ip
tlsSupport: false 

create storage classes and volumes accordingly:

#file: storageclass_and_volumes.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: eclipsechewksp
  labels:
    type: local
spec:
  storageClassName: eclipsechewksp
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/wksp"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: eclipseche
  labels:
    type: local
spec:
  storageClassName: eclipseche
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: eclipsechewksp
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: eclipseche
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain

after this use the additional argument in chectl server:start:

chectl server:start --platform minikube --multiuser --che-operator-cr-yaml=/usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

upon attempts to start chectl (using chectl server:delete and server:start again) the postgres folder (called userdata) has to be removed and the volumes in the minikube cluster have to ve removed and created again (using kubectl delete -f and apply -f with the provided yaml).

so to recap:
to remove the unsuccessfull che start garbage files and volumes.

chectl server:delete 
kubectl delete -f <storageclass_and_volumes.yaml> 
rm -rf /data/userdata

to try again:

kubectl apply -f <storageclass_and_volumes.yaml> 
chectl server:start --platform minikube --multiuser --che-operator-cr-yaml=/usr/local/lib/chectl/templates/che-operator/crds/org_v1_che_cr.yaml

@gattytto
Copy link

gattytto commented Apr 9, 2020

che-incubator/chectl#461

@fragpic
Copy link
Author

fragpic commented Apr 12, 2020

@cbyreddy
That's might be the cause. kubernetes/minikube#7218
pls. downgrade minikube to v1.8 and try again.

I was having issues with v1.8 too so I tried again using k3s. I got to a much later stage of the install process before it errored out again.

sudo chectl server:start --platform k8s  --multiuser --domain 192.168.1.137.nip.io
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Kubernetes preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify remote kubernetes status...done.
    ✔ Check Kubernetes version: Found v1.17.4+k3s1.
    ✔ Verify domain is set...set to 192.168.1.137.nip.io.
    ✔ Check if cluster accessible... ok
Eclipse Che logs will be available in '/tmp/chectl-logs/1586681575409'
  ✔ Start following logs
    ↓ Start following Operator logs [skipped]
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
   ✔ 🏃‍  Running Helm to install Eclipse Che
    ✔ Verify if helm is installed
    ✔ Check Helm Version: Found v3.1.2+gd878d4d
    ✔ Create Namespace (che)...done.
    ✔ Check Eclipse Che TLS certificate...going to generate self-signed one
      ✔ Check Cert Manager deployment...not deployed
      ✔ Deploy cert-manager...done
      ✔ Wait for cert-manager...ready
      ✔ Check Cert Manager CA certificate...generating new one
      ✔ Set up Eclipse Che certificates issuer...done
      ✔ Request self-signed certificate...done
      ✔ Wait for self-signed certificate...ready
      ✔ ❗[MANUAL ACTION REQUIRED] Please add local Eclipse Che CA certificate into your browser: /home/admin/cheCA.crt
    ✔ Check Cluster Role Binding...does not exists.
    ✔ Preparing Eclipse Che Helm Chart...done.
    ✔ Updating Helm Chart dependencies...done.
    ✔ Deploying Eclipse Che Helm Chart...done.
  ❯ ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Devfile registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Plugin registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ❯ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✖ starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
    Show important messages
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 ›   Installation failed, check logs in '/tmp/chectl-logs/1586681575409'

Any idea what could have gone wrong now? I don't think it is a storage provisioning issue but I'm not sure.
Here is the log output for the che pod
https://pastebin.com/9NatW21j

@zarinfam
Copy link

I have had this problem with both microk8s and k3s!

@tolusha
Copy link
Contributor

tolusha commented Apr 12, 2020

@cbyreddy
chectl version ?

Pls provide logs for the second installation.
https://pastebin.com/9NatW21j isn't available

@fragpic
Copy link
Author

fragpic commented Apr 12, 2020

@cbyreddy
chectl version ?

Pls provide logs for the second installation.
https://pastebin.com/9NatW21j isn't available

Sorry, here you go.
https://pastebin.com/aTY9zLRn

This seems to be the issue but I'm not sure

Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching keycloak-che.192.168.1.137.nip.io found

@tolusha
Copy link
Contributor

tolusha commented Apr 13, 2020

related one
#16429

@tolusha tolusha added this to the Backlog - Deploy milestone Apr 13, 2020
@fragpic
Copy link
Author

fragpic commented Apr 13, 2020

related one
#16429

Is there anything I can try to fix the error or is it a bug?

@tolusha
Copy link
Contributor

tolusha commented Apr 13, 2020

It is a bug. I will take a look on it later.

What I can suggest for now:

  • use a lower version of minikube (I personally use 1.5.2)
  • deploy che with helm installer:
chectl server:start --platform minikube --installer helm --multiuser

@tolusha tolusha mentioned this issue Apr 16, 2020
50 tasks
@tolusha
Copy link
Contributor

tolusha commented Apr 24, 2020

@cbyreddy
Could you specify the chectl version you used?

@tolusha
Copy link
Contributor

tolusha commented Apr 28, 2020

@zarinfam
Could you specify the chectl version you used?

@tolusha
Copy link
Contributor

tolusha commented Apr 29, 2020

@cbyreddy
I am closing this one.
Feel free to open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/chectl Issues related to chectl, the CLI of Che kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Development

No branches or pull requests

6 participants