Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Che keeps failing on minishift #13995

Closed
5 of 23 tasks
tsmaeder opened this issue Jul 24, 2019 · 15 comments
Closed
5 of 23 tasks

Che keeps failing on minishift #13995

tsmaeder opened this issue Jul 24, 2019 · 15 comments
Assignees
@tsmaeder @mmorhun
Labels
severity/P1 Has a major impact to usage or development of the system.
Milestone
7.1.0

Comments

@tsmaeder
Copy link
Contributor

tsmaeder commented Jul 24, 2019 •
edited
Loading

Describe the bug

The che pod keeps failing for me: I'm following the process described in https://github.com/eclipse/che-theia/blob/master/CONTRIBUTING.md

What it looks like is like somehow, a lot of che pods are started (see screenshots). After 5 mins or so, they all fail.

Che version

  • latest
  • nightly
  • other: please specify

Steps to reproduce

minishift delete
minishift start --disk-space 40G --memory 16G
oc login -u system:admin
chectl server:start -p minishift -a operator
follow the "contributing" process for che-theia.

Expected behavior

Runtime

  • kubernetes (include output of kubectl version)
  • Openshift (include output of oc version)
  • minikube (include output of minikube version and kubectl version)
  • minishift (include output of minishift version and oc version)
  • docker-desktop + K8S (include output of docker version and kubectl version)
  • other: (please specify)

Screenshots

PodsEvicted
manypods

Installation method

  • chectl
  • che-operator
  • minishift-addon
  • I don't know

Environment

  • my computer
    • Windows
    • Linux
    • macOS
  • Cloud
    • Amazon
    • Azure
    • GCE
    • other (please specify)
  • other: please specify

Additional context
@tsmaeder tsmaeder added the severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. label Jul 24, 2019
@tsmaeder
Copy link
Contributor Author

This is a blocker for me: I'm unable to progress on my task for 7.0 (Java LS "Classpath is incomplete" warning when loading petclinic #13427)

@benoitf
Copy link
Contributor

benoitf commented Jul 24, 2019

Could you add logs of che-server pod ? out-of-memory ?

and could you list all kubernetes/openshift deployments objects

@benoitf
Copy link
Contributor

benoitf commented Jul 24, 2019

and the log of the operator (as you're using operator deployment)

  • checluster objects

@tsmaeder
Copy link
Contributor Author

operator log:

time="2019-07-24T11:32:39Z" level=info msg="Go Version: go1.11.5" 
time="2019-07-24T11:32:39Z" level=info msg="Go OS/Arch: linux/amd64" 
time="2019-07-24T11:32:39Z" level=info msg="operator-sdk Version: v0.5.0" 
time="2019-07-24T11:32:39Z" level=info msg="Operator is running on OpenShift v3.x" 
time="2019-07-24T11:32:49Z" level=info msg="Registering Components" 
time="2019-07-24T11:32:49Z" level=info msg="Starting the Cmd" 
time="2019-07-24T11:33:02Z" level=info msg="Waiting for deployment keycloak. Default timeout: 420 seconds" 
time="2019-07-24T11:35:05Z" level=info msg="Deployment 'che' successfully scaled to 1" 
time="2019-07-24T11:35:05Z" level=info msg="Updating eclipse-che CR with Keycloak URL status: http://keycloak-che.192.168.99.114.nip.io " 
time="2019-07-24T11:35:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:35:05Z" level=info msg="Updating eclipse-che CR with status: Eclipse Che server: Available" 
time="2019-07-24T11:35:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:35:05Z" level=info msg="Updating eclipse-che CR with Eclipse Che server URL: http://che-che.192.168.99.114.nip.io " 
time="2019-07-24T11:35:05Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:35:05Z" level=info msg="Eclipse Che is now available at: http://che-che.192.168.99.114.nip.io " 
time="2019-07-24T11:42:26Z" level=info msg="Updating eclipse-che CR with status: Che API: Unavailable" 
time="2019-07-24T11:42:26Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:42:26Z" level=info msg="Waiting for deployment che. Default timeout: 420 seconds" 
time="2019-07-24T11:48:13Z" level=info msg="Deployment 'che' successfully scaled to 1" 
time="2019-07-24T11:48:13Z" level=info msg="Updating eclipse-che CR with Keycloak URL status: http://keycloak-che.192.168.99.114.nip.io " 
time="2019-07-24T11:48:13Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:48:13Z" level=info msg="Updating eclipse-che CR with status: Eclipse Che server: Available" 
time="2019-07-24T11:48:13Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:48:13Z" level=info msg="Updating eclipse-che CR with Eclipse Che server URL: http://che-che.192.168.99.114.nip.io " 
time="2019-07-24T11:48:13Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:48:13Z" level=info msg="Eclipse Che is now available at: http://che-che.192.168.99.114.nip.io " 
time="2019-07-24T11:56:51Z" level=info msg="Updating eclipse-che CR with status: Che API: Unavailable" 
time="2019-07-24T11:56:51Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T11:56:51Z" level=info msg="Waiting for deployment che. Default timeout: 420 seconds" 
time="2019-07-24T12:03:24Z" level=info msg="Deployment 'che' successfully scaled to 1" 
time="2019-07-24T12:03:24Z" level=info msg="Updating eclipse-che CR with Keycloak URL status: http://keycloak-che.192.168.99.114.nip.io " 
time="2019-07-24T12:03:24Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T12:03:24Z" level=info msg="Updating eclipse-che CR with status: Eclipse Che server: Available" 
time="2019-07-24T12:03:24Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T12:03:24Z" level=info msg="Updating eclipse-che CR with Eclipse Che server URL: http://che-che.192.168.99.114.nip.io " 
time="2019-07-24T12:03:24Z" level=info msg="Custom resource eclipse-che updated" 
time="2019-07-24T12:03:24Z" level=info msg="Eclipse Che is now available at: http://che-che.192.168.99.114.nip.io "

@tsmaeder
Copy link
Contributor Author

Could you add logs of che-server pod ? out-of-memory ?

and could you list all kubernetes/openshift deployments objects

What do you mean by "list"? Contents?

@amisevsk
Copy link
Contributor

Looks like your minishift cluster is evicting the Che pods, how much memory/disk are you giving minishift? What's the output of minishift status?

I don't recall where events related to evictions would be found.

@sunix
Copy link
Contributor

sunix commented Jul 24, 2019

is the better without using the operator ( to "unblock" that issue)

@tsmaeder
Copy link
Contributor Author

40Gig disk, 16G RAM

@amisevsk
Copy link
Contributor

Only other thing I can think of is oc describe node as system:admin and checking the conditions section.

@benoitf
Copy link
Contributor

benoitf commented Jul 24, 2019

@tsmaeder did you did a minishift start with fewer memory and disk and then start again with higher options or you deleted/started a new one

because If you just do stop/start with different settings AFAIK it is not applied on other start commands (but maybe now it's not as it was)

could you check

$ minishift config view

@nickboldt
Copy link
Contributor

nickboldt commented Jul 24, 2019 •
edited
Loading

FWIW I don't encounter this when I deploy with the minishift addon instead of the operator:

# Grant enough memory to run CRW 1.2 / Che 6 and workspaces
# 6G min for one workspace, plus 2G for each additional workspace
minishift config set memory 12G

# Grant enough disk for downloading container images
minishift config set disk-size 25G

# if not using default virtualization engine, eg., want Virtual Box, use
minishift config set vm-driver virtualbox

# enable cluster admin user
minishift addons install --defaults
minishift addons enable admin-user

minishift start

eval $(minishift docker-env)

oc adm policy  --as system:admin add-cluster-role-to-user cluster-admin developer
oc login -u developer -p 123

oc create secret generic registryredhatio --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=${HOME}/.docker/config.json && oc secrets link default registryredhatio --for=pull && oc secrets link builder registryredhatio && oc get secret registryredhatio

then

chectl server:start -a minishift-addon -p minishift

# required if you're on minishift 1.34.0 or earlier
oc set env dc/che CHE_WORKSPACE_PLUGIN__REGISTRY__URL=https://che-plugin-registry.openshift.io/v3

@slemeur
Copy link
Contributor

slemeur commented Jul 24, 2019

While this is a blocker to you @tsmaeder it seems that the solution is working on minikube. I would not make it a blocker as there is an existing path. But that is something to workout. Also please check the proposed solution from @nickboldt .

@slemeur slemeur added severity/P1 Has a major impact to usage or development of the system. team/ide2 and removed severity/blocker Causes system to crash and be non-recoverable or prevents Che developers from working on Che code. labels Jul 24, 2019
@slemeur slemeur added this to the 7.1.0 milestone Jul 24, 2019
@mmorhun
Copy link
Contributor

mmorhun commented Jul 25, 2019

Tried to reproduce the issue but it worked for me.
What I did:

  1. minishift delete
  2. Download latest minishift (minishift v1.34.1+c2ff9cb).
  3. Download latest chectl (chectl/0.0.2-a74ad81 linux-x64 node-v10.4.1)
  4. minishift start --vm-driver=virtualbox --cpus=4 --memory=16192 --disk-size=50GB --docker-opt userland-proxy=false
  5. oc login -u system:admin
  6. ./chectl-linux server:start -p minishift -a operator (after this I had Che multiuser deployed)
  7. ./chectl-linux workspace:start -f https://raw.githubusercontent.com/eclipse/che-theia/master/devfiles/che-theia-all.devfile.yaml --access-token=<the-token> (I retreived it from a dashboard query in browser)
  8. In the workspace from a terminal in che-dev (imo, name is wrong, correct one should be che-theia-dev). Then from the terminal: cd /projects/theia, che:theia init, yarn. The build finished successfully.

On host I have Fedora 30 and VirtualBox 6.0.8

Screenshot from minishift dashboard:
minishift-dashboard

P.S. From my experience with Minishift + Che I may say it is not that stable. It may work just fine today and later I may end up in spending a half of day trying to resolve some issues with it. So, I will not wonder if it doesn't work on your side right now.
If it is ok for you, give a try to Minikube + single user Che.

@slemeur
Copy link
Contributor

slemeur commented Jul 26, 2019

Hopefuly things are getting better on your side @tsmaeder ?

@evidolob evidolob mentioned this issue Aug 7, 2019
Closed
26 tasks
@sunix
Copy link
Contributor

sunix commented Sep 4, 2019

@tsmaeder I am closing this one as some of us are not able to reproduce. Feel free to reopen if you have more info.

@sunix sunix closed this as completed Sep 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsmaeder tsmaeder

@mmorhun mmorhun

Labels
severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Milestone
7.1.0
Development

No branches or pull requests
7 participants
@nickboldt @benoitf @sunix @slemeur @tsmaeder @mmorhun @amisevsk