Impossible to get a terminal in a workspace

[slemeur]

Description

It is impossible to me to get a terminal in my workspace, created from the default stacks
Either by going to:

• « Che Workspaces » panel
• Menu « Terminal »
• Command Palette

Reproduction Steps

Create a workspace from the default stack java-maven. Add project springboot-http-booster

OS and version:
Che 7 RC 2 on OCP 4.1
Che-theia:nightly

Diagnostics:

[akurinnoy]

Hi @slemeur
I cannot reproduce this issue neither on minikube, minishift or che.openshift.io .
Could you confirm that the issue is still there?

[slemeur]

I cannot reproduce it either on the latest version.
Closing, and if anybody find this issue, we will reopen

[tsmaeder]

I just saw this: with theia:latest and Chrome: browser console shows errors when opening terminal "che-machine-exececu" from the containers view:

Uncaught (in promise) Error: Request 'create' failed
    at Proxy.<anonymous> (theia.fd3a7dbd89c7575a73b9.js:1)
    at n.<anonymous> (che.8e307cc851d8e8a3132b.js:1)
    at che.8e307cc851d8e8a3132b.js:1
    at Object.next (che.8e307cc851d8e8a3132b.js:1)
    at che.8e307cc851d8e8a3132b.js:1
    at new Promise (<anonymous>)
    at c (che.8e307cc851d8e8a3132b.js:1)
    at n.createTerminal (che.8e307cc851d8e8a3132b.js:1)
    at n.<anonymous> (che.8e307cc851d8e8a3132b.js:1)
    at che.8e307cc851d8e8a3132b.js:1

and

Uncaught (in promise) Error: Request 'resize' failed
    at Proxy.<anonymous> (theia.fd3a7dbd89c7575a73b9.js:1)
    at n.resizeTerminalProcess (che.8e307cc851d8e8a3132b.js:1)
    at n.t.onUpdateRequest (theia.fd3a7dbd89c7575a73b9.js:1)
    at n.e.processMessage (vendors.795ac5958e7158cc3edd.js:63)
    at n.t.processMessage (theia.fd3a7dbd89c7575a73b9.js:1)
    at h (vendors.795ac5958e7158cc3edd.js:19)
    at n (vendors.795ac5958e7158cc3edd.js:19)
    at g (vendors.795ac5958e7158cc3edd.js:19)

[l0rd]

Note that 7.0.0 is with che-theia:7.0.0, not latest.

[akurinnoy]

@tsmaeder I've just tested che-theia:7.0.0 in Google Chrome.

Uncaught (in promise) Error: Request 'resize' failed happens when first terminal is opened and I seem it doesn't affects how it works afterward. All terminals except che-machine-exec work well.

Trying to open terminal for che-machine-exec I see it opens for a while, shows short output and closes and browser dev console shows this error:

terminal ERROR Terminal error: command terminated with exit code 126

che-machine-exec terminal output:

AFAIK che-machine-exec is not supposing to open a terminal atm. I guess @AndrienkoAleksandr is the right person to ask about it but he is now on PTO.
Maybe, @evidolob can comment more about it.

[tsmaeder]

AFAIK che-machine-exec is not supposing to open a terminal atm.

the behaviour is the same for every other container as well for me.

[akurinnoy]

could you add more details then?
Where and how do you run a workspace?

[tsmaeder]

Are you saying this works for you with other containers? You can simply create a workspace with the java-maven devfile on minishift. @akurinnoy

[akurinnoy]

@tsmaeder yes,
I've done it and all terminals except che-machine-exec work well

[tsmaeder]

Which version of che-server are you running and which version of che-theia are you using in the workspace?

[akurinnoy]

@tsmaeder hello,
che-server: 7.0.0-rc-4.0-SNAPSHOT
theia: Che-Theia@5dfc5c8 using Theia@5dfc5c8 Built 7/11/2019, 8:32:33 AM

[evidolob]

It can be related to #13454, we have adopted images to openshift with this PR: eclipse-che/che-devfile-registry#24
@tsmaeder Are you using that adopted images for your devfile?

[tsmaeder]

I don't use a devfile, I use what's in the dashboard of che-server:nightly. Anyway: I can open a terminal into those images no problem inside the minishift console, so I don't think this is related to images.

[evidolob]

I test java-maven stack, and I can open terminal in maven container, and any other containers except che-machine-exec. So issue in that exceptions? But they do not block the work, I think for them we need to have another issue.

[evidolob]

I create issue for that resize error - #13830

[tsmaeder]

Strangely...when I create the java-maven stack, it pulls che-theia:next. I'm using che-server:nightly and just redeployed che on my minishift.

[tsmaeder]

Also, the java-maven stack does not check out any projects for me.

[tsmaeder]

Just did a video chat with @evidolob and we could confirm that it happens for me, but not for him. Possible differences:

1. He has a Mac, I run Fedora
2. His way of installing Che seems to be different: he reassigned the key for the che registry url, I don't even have that setting neither in the deployment env nor in the config map.

I switched to using the minishift add-on to install becuase the operator was broken with respect to the registry url.

[nickboldt]

Using minishift 1.34.0 (and its included mini-che addon), you will still have the wrong registry URL, but the workaround once deployed is easy:

oc set env dc/che CHE_WORKSPACE_PLUGIN__REGISTRY__URL=https://che-plugin-registry.openshift.io/v3

More details / steps: https://divby0.blogspot.com/2019/06/deploying-che-7-to-minishift-with-chectl.html

I've also posted on that blog how to get the list of images that Che pulls as part of deployment so you can verify if it's pulling latest or nightly:

https://divby0.blogspot.com/2019/06/discovering-containers-loaded-by-che-7.html

[tsmaeder]

@nickboldt strangely, my plugins are working just fine, it's just that I can't open a terminal into them.

[rhopp]

I've tried this today with latest che-theia image - Terminals are being opened with the exception of che-machine-exec terminal.
I'm running Fedora 30, Chrome 75.0.3770.100

[evidolob]

@rhopp That on purpose, che-machine-exec is protected, we don't want to let user create terminal to it. User can kill machine-exec process and can loose all opened terminals, and he cannot create new.

[tsmaeder]

@rhopp so it's not a fedora vs. Mac issue. Still happens on my machine 100% of the time, though. Spot the difference!

[vitaliy-guliy]

Terminal does not work on minishift
To reproduce I don't use some special stuff, just start minishift and then deploy Che using che-ctrl.

minishift start --memory=10GB --cpus=4 --disk-size=30GB
./chectl-linux server:start --platform=minishift

Having running Che I created a workspace from NodeJS Express Web Application stack, started it and tried to open a terminal.

Tested on Linux Mint 18.3 Sylvia

[l0rd]

If you are using an old version of minishift you may be using an old version of the minishift-addon and that may explain why you cannot do exec commands.

And anyway, even in latest minishift, the addon is not the latest one. Can you test deploying Che using the operator (with latest chectl release)?

[vitaliy-guliy]

@l0rd I cannot deploy Che with the latest version of chectl 20190716105045.
I get different errors when trying to deploy with operator

vitaliy@vitaliy-ThinkPad-P50 ~/bin $ ./chectl-linux server:start --platform=minishift --installer=operator
  ✔ ✈️  Minishift preflight checklist
    ✔ Verify if oc is installed...done.
    ✔ Verify if minishift is installed...done.
    ✔ Verify if minishift is running...done.
    ✔ Retrieving minishift IP and domain for routes URLs...192.168.99.100.nip.io.
  ❯ 🏃‍  Running the Che Operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...done.
    ✔ Create ServiceAccount che-operator in namespace che...done.
    ✖ Create Role che-operator in namespace che
      → ERROR: It looks like you don't have enough privileges. You need to grant more privileges to current user or use a different user. If you are using minishift you can "oc login -u system:admin"
      Create RoleBinding che-operator in namespace che
      Create CRD checlusters.org.eclipse.che
      Waiting 5 seconds for the new Kubernetes resources to get flushed
      Create deployment che-operator in namespace che
      Create Che Cluster eclipse-che in namespace che
 ›   Error: ERROR: It looks like you don't have enough privileges. You need to grant more privileges to current user or use a different user. If you are using minishift you can "oc login -u system:admin"
vitaliy@vitaliy-ThinkPad-P50 ~/bin $ 

and when deploy with helm

vitaliy@vitaliy-ThinkPad-P50 ~/bin $ ./chectl-linux server:start --platform=minishift
  ✔ ✈️  Minishift preflight checklist
    ✔ Verify if oc is installed...done.
    ✔ Verify if minishift is installed...done.
    ✔ Verify if minishift is running...done.
    ✔ Retrieving minishift IP and domain for routes URLs...192.168.99.100.nip.io.
  ✔ 🏃‍  Running the Che minishift-addon
    ✔ Check minishift version...done.
    ✔ Check logged...done.
    ✔ Check che addon is available...done.
    ✔ Apply Che addon...done.
  ❯ ✅  Post installation checklist
    ❯ Che pod bootstrap
      ✖ scheduling
        → ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Che Server URL
      Che status check
Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
    at KubeHelper.<anonymous> (/snapshot/chectl/lib/api/kube.js:0:0)
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/chectl/node_modules/tslib/tslib.js:107:62)
vitaliy@vitaliy-ThinkPad-P50 ~/bin $ 

Yesterday I played a lot and got the same problem. I was able to deploy minishift only with chectl 20190712093149

[vitaliy-guliy]

Link with #13881

[vitaliy-guliy]

Cannot reproduce after updating minishift from 1.34.0 to 1.34.1 with updating add-ons.

[l0rd]

@vitaliy-guliy with the operator the message say that you don't have enough privileges and propose a workaround:

If you are using minishift you can "oc login -u system:admin"

[tsmaeder]

After deploying with '-p operator' and from the latest chectl (as of today), I can open terminals in Chrome. I would, however not close this issue but would propose that we improve feedback to the user when something like this goes wrong. I'm pretty sure some command fails when we lack the role or priviledges. Maybe even a preflight-check page in the Che UI?

[evidolob]

I agree with @tsmaeder about check, but it sounds like improvement and not fit to this blocker issue.
So @tsmaeder could you create separate issue for terminal availability check?

[sunix]

I could reproduce on openshift v4 (OCP and CRC)
It happens for containers that don't have an entry in /etc/passwd for the arbitrary user. Openshift v4 is adding this kind of line by default:

1001360000:x:1001360000:0:1001360000 user:/myworkdir:/sbin/nologin

and the terminal can't open /sbin/nologin

A fix would be to use /bin/sh if we have /sbin/nologin.

[amisevsk]

Should be fixed by recent devfile-registry changes (at least for patched images).

[l0rd]

I beleive the original issue described here has been addressed.

From the discussion 2 new issues (probably not for 7.0.0) have emerged:

• Provide a message when a terminal fails to open
• Open terminal with non-patched images on OCP 4 fails

Closing this issue

[isbur]

Running fresh cpp workspace on che.openshift.io. Same problem.

[isbur]

However terminal runs smoothly for NodeJS Angular stack. Maybe some images are not patched?

[RomanNikitenko]

@isbur
unfortunately yes, we have the issue for that.
you can try the workaround #15969 (comment)