Could not start workspace <wokespace-id> Reason: Timeout reached

[dtoebe]

I loaded Che with the default docker start up explained in your docs:
https://eclipse-che.readme.io/docs/usage-docker#usage

The exact command I used:

docker run -t -v /var/run/docker.sock:/var/run/docker.sock \                                       [09:54:18]
           codenvy/che-launcher:nightly start

Everything downloaded and booted fine. I went to http:localhost:8080 in Google Chrome (Not Chromium) created a Java Console Hello world workspace. I didn't even change the name. When it IDE i get a popup error message:

"
Could not start workspace wksp-9b0k. Reason: Timeout reached. The Che server has been unable to verify that your workspace's agent has successfully booted. Either the workspace is unreachable, the agent had an error during startup, or your workspace is starting slowly. You can configure machine.ws_agent.max_start_time_ms in Che properties to increase the timeout.
"

I created a gist with the logs docker logs -f che-server:

https://gist.github.com/dtoebe/746ef514824078044636f271aecc94d3

Here is a screenshot:

Che version: docker:latest (4.7.0-SNAPSHOT)
OS and version: Fedora 24 x86_64
Docker version: 1.12.0 / Go version 1.6.3
Che install: Docker

EDIT:
I deleted all the docker images and rm the che dir in /home/users. Then I ran the docker command again as stated above. I created this gist of the CLI output:
https://gist.github.com/dtoebe/7fd1478b967ad5637b994f855cf3d724

After that I tried again and got the same results.

The only Issue I saw in the cli output was line 19 in the gist ip: can't find device 'eth0'

[TylerJewell]

@l0rd @eivantsov @ddementieva - given that we are just now releasing the new launcher, let's dive into this issue. It's Fedora, so I am unfamiliar with that setup. But this looks like a networking issue.

[TylerJewell]

@dtoebe - please take a look at this page where we talk about how the different subsystems of Che bridge networks. It looks like there is an issue with the Che server being able to verify that the workspace agent is connecting. So take a look at the chart as to how the agent communicates to the server and the server communicates to the workspace.

https://eclipse-che.readme.io/v4.6/docs/networking#connectivity

[dtoebe]

Ok, I will start by disabling firewalld (Fedora uses that in place of UFW), and work my way up from there

[TylerJewell]

We had some challenges with firewalld blocking communications from workspaces to Che server in testing of the new launcher. However, those issues were around discovery to verify that the server was alive. We were able to get the communications to work at the subsystem level - so maybe this is about overriding one of the IP addresses to be one that is reachable between workspace and server, which are each in their own container, communicating through a docker daemon.

[dtoebe]

Upon looking at my debugging info I get the missing device eth0 and no docker host ip.

docker run -t -v /var/run/docker.sock:/var/run/docker.sock \                                       [12:49:46]
           codenvy/che-launcher:nightly info
ip: can't find device 'eth0'
DEBUG: ---------------------------------------
DEBUG: ---------  CHE DEBUG INFO  ------------
DEBUG: ---------------------------------------
DEBUG:
DEBUG: ---------  PLATFORM INFO  -------------
DEBUG: DOCKER_INSTALL_TYPE       = native
DEBUG: DOCKER_HOST_OS            = Fedora 24 (Workstation Edition)
ip: can't find device 'eth0'
DEBUG: DOCKER_HOST_IP            =
DEBUG: DOCKER_DAEMON_VERSION     = 1.12.0
DEBUG:
DEBUG:
DEBUG: --------- CHE INSTANCE INFO  ----------
DEBUG: CHE CONTAINER EXISTS      = NO
DEBUG: CHE CONTAINER STATUS      = stopped
DEBUG:
DEBUG:
DEBUG: ----  CURRENT COMMAND LINE OPTIONS  ---
DEBUG: CHE_PORT                  = 8080
DEBUG: CHE_VERSION               = nightly
DEBUG: CHE_RESTART_POLICY        = no
DEBUG: CHE_USER                  = root
DEBUG: CHE_HOST_IP               =
DEBUG: CHE_LOG_LEVEL             = info
DEBUG: CHE_HOSTNAME              = localhost
DEBUG: CHE_DATA_FOLDER           = /home/user/che
DEBUG: CHE_CONF_FOLDER           = not set
DEBUG: CHE_LOCAL_BINARY          = not set
DEBUG: CHE_SERVER_CONTAINER_NAME = che-server
DEBUG: CHE_SERVER_IMAGE_NAME     = codenvy/che-server
DEBUG:
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------

[TylerJewell]

Ok. This must be a bug in the way that we get the docker host IP for your system.

get_docker_host_ip() {
  NETWORK_IF="eth0"
  if is_boot2docker; then
    NETWORK_IF="eth1"
  fi

  docker run --rm --net host \
            alpine sh -c \
            "ip a show ${NETWORK_IF}" | \
            grep 'inet ' | \
            cut -d/ -f1 | \
            awk '{ print $2}'
}

Can you provide the output of docker run --rm --net host alpine sh -c "ip a show"?

[dtoebe]

docker run --rm --net host alpine sh -c "ip a show"                                                [12:58:51]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether 44:8a:5b:f1:48:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.115/24 brd 192.168.1.255 scope global dynamic enp4s0
       valid_lft 69467sec preferred_lft 69467sec
    inet6 fe80::f7e7:ccdb:8e91:9d70/64 scope link
       valid_lft forever preferred_lft forever
3: wlp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 40:e2:30:d2:2e:14 brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN qlen 1000
    link/ether 52:54:00:3c:4a:08 brd ff:ff:ff:ff:ff:ff
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:91:44:a1:1b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:91ff:fe44:a11b/64 scope link
       valid_lft forever preferred_lft forever

[TylerJewell]

Ok, let's try using the docker0 interface. Can you set CHE_HOST_IP=172.17.0.1 and give the program a go? The che info will still give an error, but this overrides the IP address provided by the default get_docker_host_ip() method.

[dtoebe]

Ok I was able to create the Java empty workspace and create a project in the workspace, and open a Java class. Edit and run it. I will create other workspaces with other languages and test those later today. Then Nuke everything and make sure I am able to do it again.

The command:

docker run -t -v /var/run/docker.sock:/var/run/docker.sock \
           -e CHE_HOST_IP=172.17.0.1 codenvy/che-launcher:nightly start

Should I mark this as closed?

[TylerJewell]

@dtoebe - please leave the issue open. It's a bug that we could not auto-discover the right interface when doing the $(get_docker_host_ip) function. Could you tell us about your Fedora installation? Was it a clean VM - or anything funky modified to it after setup?

[dtoebe]

I installed it last night. Using Gnome desktop, and x86_64 Workstation edition.

I am using the docker repo in place of the default fedora install:
https://docs.docker.com/engine/installation/linux/fedora/

I also used fedy to install many other things including the Oracle JDK, and Google Chrome (Version 52.0.2743.82 (64-bit))
http://folkswithhats.org/

And am using libvirt qemu-kvm (as you saw in my network list) from the default repos

Other than Gnome themes and the codecs from fedy... Nothing else.

Also FYI I had the default firewalld (no added ports or anything) when running the final test

[TylerJewell]

Thanks for helping me diagnose the issue. We need to take a longer look at how the docker_host_ip function works. It may be always safer to just query the docker0 interface.

What do you think of Che so far?

[dtoebe]

Ya no matter what distro I use (Ubuntu/Debian, Fedora, Arch) they all default to docker0.

I am typically a vim user for everything (JS/Web Go Rust C/C++ python) but Java for that I use Eclipse, because I I like the organization behind it. So far I have only used the interface with che and that seems great. I will prob use an electron wrapper (https://github.com/TylerJewell/che-electron). Ha thats yours. and set it up to start and shut down the server from the app. I will hopefully have time to do that this weekend.

But I really like that the whole thing and each workspace is containerized

[TylerJewell]

If you are able to auto-start and stop the server from that app, we'd love that contribution. We also have a new capability coming soon, called che-files, which will give you a syntax where you git clone a repo, che up, and then we will convert that directory into a workspace into a che server. If no che server is running, we'll use Docker to launch one quietly for you.

[dtoebe]

I'll be more than happy to contribute.. That sounds great! when I get time I will tear into that electron app and see if I can get that feature to work. And other feature I will try to add is connecting to a remote che with that feature. now I just have to get used to not :w all the time.

The che up sounds amazing. that was something I was wondering about. I look forward to it.

[l0rd]

Nice catch! Great job @TylerJewell and @dtoebe

[ghost]

Closing it then. Yes, docker0 makes much more sense here.