Issues running chectl on windows

[bfitzpat]

Trying to run che locally using chectl on windows and hitting a funky error.

My kubernetes instance is running:

minikube config set memory 8000
minikube config set disk-size 50g
minikube config set cpus 4
minikube start

And when I do
chectl server:start -a operator --k8spodwaittimeout=900000
I get...

$ chectl server:start -a operator  --k8spodwaittimeout=900000
[10:01:06] ✈️  Minikube preflight checklist [started]
[10:01:06] Verify if kubectl is installed [started]
[10:01:07] Verify if kubectl is installed [completed]
[10:01:07] Verify if minikube is installed [started]
[10:01:07] Verify if minikube is installed [completed]
[10:01:07] Verify if minikube is running [started]
[10:01:18] Verify if minikube is running [completed]
[10:01:18] Start minikube [started]
[10:03:02] Start minikube [completed]
[10:03:02] Verify if minikube ingress addon is enabled [started]
[10:03:04] Verify if minikube ingress addon is enabled [completed]
[10:03:04] Enable minikube ingress addon [started]
[10:03:04] Enable minikube ingress addon [skipped]
[10:03:04] → Ingress addon is already enabled.
[10:03:04] Retrieving minikube IP and domain for ingress URLs [started]
[10:03:06] Retrieving minikube IP and domain for ingress URLs...192.168.0.142.nip.io. [title changed]
[10:03:06] Retrieving minikube IP and domain for ingress URLs...192.168.0.142.nip.io. [completed]
[10:03:06] ✈️  Minikube preflight checklist [completed]
[10:03:06] 🏃‍  Running the Che Operator [started]
[10:03:06] Copying operator resources [started]
[10:03:06] Copying operator resources...done. [title changed]
[10:03:06] Copying operator resources...done. [completed]
[10:03:06] Create Namespace (che) [started]

undefined:0

Error: ENOENT: no such file or directory, lstat 'C:\Users\brianf\templates\che-operator'

Minikube version 1.1.1
Chectl version chectl-v0.0.2-b508feb-win32-x64.tar.gz https://github.com/che-incubator/chectl/releases/tag/20190715063232

Describe the bug

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
• che-operator
• minishift-addon
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

[bfitzpat]

I got a little further after deleting the .minikube folder. But it's still not finishing.

$ chectl server:start -a operator --k8spodwaittimeout=200000
  √ ✈️  Minikube preflight checklist
    √ Verify if kubectl is installed
    √ Verify if minikube is installed
    √ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    √ Verify if minikube ingress addon is enabled
    ↓ Enable minikube ingress addon [skipped]
      → Ingress addon is already enabled.
    √ Retrieving minikube IP and domain for ingress URLs...192.168.0.143.nip.io.
  √ �🏃‍  Running the Che Operator
    √ Copying operator resources...done.
    √ Create Namespace (che)...It already exist.
    √ Create ServiceAccount che-operator in namespace che...It already exist.
    √ Create Role che-operator in namespace che...It already exist.
    √ Create RoleBinding che-operator in namespace che...It already exist.
    √ Create CRD checlusters.org.eclipse.che...It already exist.
    √ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    √ Create deployment che-operator in namespace che...It already exist.
    √ Create Che Cluster eclipse-che in namespace che...It already exist.
  > ✅  Post installation checklist
    √ PostgreSQL pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      √ starting...done.
    > Keycloak pod bootstrap
      √ scheduling...done.
      √ downloading images...done.
      × starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Che pod bootstrap
      Retrieving Che Server URL
      Che status check
Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
    at KubeHelper.<anonymous> (C:/snapshot/chectl/lib/api/kube.js:0:0)
    at Generator.next (<anonymous>)
    at fulfilled (C:/snapshot/chectl/node_modules/tslib/tslib.js:107:62)

[bfitzpat]

Is there a way for me to extend the pod timeout for keycloak?

[l0rd]

@bfitzpat how have you been able to fix the first problem? (Error: ENOENT: no such file or directory, lstat 'C:\Users\brianf\templates\che-operator')

Your second problem looks something we have already tracked #13870. To verify it you should retrieve the log of wait-for-postgres init container.

[bfitzpat]

@l0rd I did two things. First, I deleted my /user/[me]/.minikube directory and I backed off from version 1.2.0 of minikube back to 1.1.1, then I ran it again and got the results you saw above.

I'll take another look at the wait-for-postgres container when I get back to this tomorrow.

[bfitzpat]

I also created this script:

KUBERNETES_VERSION=v1.12.0
MEMORY=8000
CPUS=4
DISK_SIZE=50g
# blow away everything in the che profile for a clean install
minikube delete --profile camelk
minikube delete --profile che

# set up minikube for hyper-v
# Note that Virtual Switch "MiniKube" must be created ahead of time in the Hyper-V Manager
minikube config set vm-driver hyperv
minikube config set hyperv-virtual-switch MiniKube

# configure camelk profile
minikube profile che
minikube config set kubernetes-version ${KUBERNETES_VERSION}
minikube config set memory ${MEMORY}
minikube config set cpus ${CPUS}
minikube config set disk-size ${DISK_SIZE}

# Start minikube
minikube start -p che

#Start che
chectl server:start -a operator --k8spodwaittimeout=200000

I'm wondering about the KUBERNETES_VERSION at the top and if I should bump it to 1.15.0 as is the default now for minikube?

[l0rd]

@bfitzpat It should work with both v1.15 and v1.12 so whatever you prefer. And it should work with minikube 1.2.0 as well.

What is weird is that

• After you have deleted folder .minikube, you get Create Namespace (che)...It already exist.. But you were starting from scratch so the namespace should not exist
• A chectl problem (cannot find a chectl template) is solved by deleting the .minikube folder. Maybe you have updated chectl as well in the meantime?

[bfitzpat]

Here's the general flow of how this went...

• Originally I used the full https://github.com/che-incubator/chectl/releases/download/20190716105045/chectl-v0.0.2-68f8872-win32-x64.tar.gz but I had issues when I unzipped it where the symlinks weren't created correctly by 7zip.
• @apupier suggested I drop to https://github.com/che-incubator/chectl/releases/download/20190716105045/chectl-win.exe and that got me past the initial set of issues I had and ran me into the weird issue I reported first.
• From there, I deleted the .minikube folder and that got me to the timeout issue.

I think that's where I'm at.

[bfitzpat]

I also probably rebooted in there somewhere, which may have played a part. I was trying everything I could think of to get it working a bit more...

[bfitzpat]

@apupier found this option in the documentation - https://github.com/che-incubator/chectl/blame/master/README.md#L210 - so I will give that a shot maybe with...

chectl server:start -a operator --k8spodwaittimeout=100000 --k8podreadytimeout=200000

Will let you know

[bfitzpat]

That did not work.

* Creating hyperv VM (CPUs=4, Memory=8000MB, Disk=50000MB) ...
E0717 07:40:26.656720    6572 start.go:529] StartHost: create: creating: exit status 1

X Unable to start VM: create: creating: exit status 1

* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new
 »   Error: Unexpected argument: --k8podreadytimeout=200000
 »   See more help with --help

[apupier]

there is a typo, missing the 's':

--k8podreadytimeout

--k8spodreadytimeout

[bfitzpat]

even with --k8spodreadytimeout I see

$ chectl server:start -a operator --k8spodwaittimeout=100000 --k8spodreadytimeout=200000
  > ✈️  Minikube preflight checklist
    √ Verify if kubectl is installed
    √ Verify if minikube is installed
    √ Verify if minikube is running
    × Start minikube
      → * Restarting existing hyperv VM for "che" ...
      Verify if minikube ingress addon is enabled
      Enable minikube ingress addon
      Retrieving minikube IP and domain for ingress URLs
Error: Command failed: minikube start --memory=4096 --cpus=4 --disk-size=50g
E0717 07:45:40.673993    2060 start.go:529] StartHost: start: exit status 1
X Unable to start VM: start: exit status 1
* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new
* minikube v1.1.1 on windows (amd64)
* Restarting existing hyperv VM for "che" ...
    at makeError (C:/snapshot/chectl/node_modules/execa/index.js:174:9)
    at module.exports.Promise.all.then.arr (C:/snapshot/chectl/node_modules/execa/index.js:278:16)

[apupier]

seems you rminikube instance, or at least the che namespace inminikube is in a bad state.
You can try to delete the che namespace

kubectl delete namespaces che

then relaunch

chectl server:start -a operator --k8spodwaittimeout=100000 --k8spodreadytimeout=200000

or even minishift delete if it is still not working

[bfitzpat]

Attempted deleting namespaces. Also delete is part of the script I run. No difference.

[l0rd]

minikube failing to start, not much we can do about that. As @apupier suggested minikube delete && rm -rf ~/.minikube && rm -rf ~/.kube

[bfitzpat]

even trying on Fedora, I'm running into issues...

[bfitzpat@localhost scripts]$ chectl server:start -a operator --k8spodwaittimeout=200000 --k8spodreadytimeout=200000
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Verify if minikube ingress addon is enabled
    ↓ Enable minikube ingress addon [skipped]
      → Ingress addon is already enabled.
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.39.141.nip.io
.
  ✔ 🏃‍  Running the Che Operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exist.
    ✔ Create ServiceAccount che-operator in namespace che...It already exist.
    ✔ Create Role che-operator in namespace che...It already exist.
    ✔ Create RoleBinding che-operator in namespace che...It already exist.
    ✔ Create CRD checlusters.org.eclipse.che...It already exist.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace che...It already exist.
    ✔ Create Che Cluster eclipse-che in namespace che...It already exist.
  ❯ ✅  Post installation checklist
    ❯ PostgreSQL pod bootstrap
      ✖ scheduling
        → ERR_TIMEOUT: Timeout set to pod wait timeout 200000. podExist: false, 
…
        downloading images
        starting
      Keycloak pod bootstrap
      Che pod bootstrap
      Retrieving Che Server URL
      Che status check
Error: ERR_TIMEOUT: Timeout set to pod wait timeout 200000. podExist: false, currentPhase: undefined
    at KubeHelper.<anonymous> (/snapshot/chectl/lib/api/kube.js:0:0)
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/chectl/node_modules/tslib/tslib.js:107:62)

[bfitzpat]

It's very consistent on Fedora, failing at PostgreSQL pod startup.

[apupier]

Can you check the logs of the nodes/pods for Che and Postgres?

for instance by using VS Code with Kubernetes extension:

[bfitzpat]

My keycloak log looks like this:

Certificate was added to keystore
keytool error: java.io.FileNotFoundException: /scripts/openshift.jks (Permission denied)

[bfitzpat]

Postgres pod actually fired up and ran ok this time.

Wondering if it's a memory issue. I'm waiting on some additional memory to arrive to expand my Fedora box now from 8 gigs to 16 gigs

[JeanMGirard]

You can go in your user folder and delete the "Template" folder, that will fix your issues. Also it is system protected and hidden so you will have to go in your explorer options and uncheck "hide operating system files" to see it

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.