Che server is unable to ping your workspace

[mmalyj]

I am a newbie, did a fresh install of Eclipse Che 5.0.1 on CentOS 7.2, located at 10.1.2.75. I used Chrome to get to Eclipse Che at http://10.1.2.75:8000, then did Create Workspace, with 2GB ram, I used Ready-to-go stack for Node, and chose web-nodejs-simple, then clicked Create. I got

"Error when starting agent. Unable to start workspace agent. Error when trying to start the workspace agent: Start of environment nodejs_app5 failed. Error: Timeout. The Che server is unable to ping your workspace. This implies a network configuration issue, workspace boot failure, or an unusually slow workspace boot."

Issue #3299 suggested "vm resources are too low for che to start up a workspace", I was already at 8GB ram for the CentOS VM, but I increased to 2vcpu, and then I repeated Create Workspace with 4GB ram. I got the same error. I tried Create new workspace from stack referencing a container I am working on "FROM mmalyj/nodejs_app:21", and got the same error.

I used this command to install Eclipse Che:
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /usr/etc/che/data:/data eclipse/che start

Che version: 5.0.1
OS and version: CentOS 7.2
Docker version: 1.12.6
Che cli.log output:
Sat Jan 14 19:24:00 UTC 2017
INFO: (che cli): Checking network... (hint: '--fast' skips version, network, an
d nightly checks)
/usr/bin/curl "-I -k dockerhub.com -s -o /dev/null --write-out %{http_code}"
INFO: (che cli): Pulling image alpine:3.4
docker pull alpine:3.4 >> "/data/cli.log" 2>&1
INFO: (che cli): Pulling image eclipse/che-ip:nightly
docker pull eclipse/che-ip:nightly >> "/data/cli.log" 2>&1
INFO: (che cli): Loading cli...
DEBUG: docker_run
DEBUG: is_boot2docker
DEBUG: is_initialized
DEBUG: cli_parse
DEBUG: cmd_start
DEBUG: is_docker_for_mac
DEBUG: is_initialized
DEBUG: is_initialized
INFO: (che cli): Checking registry for version '5.0.1' images
eval IMAGE_PUPPET=puppet/puppet-agent-alpine:4.6.2
eval IMAGE_REGISTRY=registry:2.5.0
eval IMAGE_INIT=eclipse/che-init:5.0.1
eval IMAGE_CHE=eclipse/che-server:5.0.1
DEBUG: update_image_if_not_found
INFO: (che download): Checking for image 'puppet/puppet-agent-alpine:4
.6.2'...
not found
DEBUG: update_image
INFO: (che download): Pulling image puppet/puppet-agent-alpine:4.6.2

docker pull puppet/puppet-agent-alpine:4.6.2 >> "/data/cli.log" 2>&1

DEBUG: update_image_if_not_found
INFO: (che download): Checking for image 'registry:2.5.0'...
not found
DEBUG: update_image
INFO: (che download): Pulling image registry:2.5.0

docker pull registry:2.5.0 >> "/data/cli.log" 2>&1

DEBUG: update_image_if_not_found
INFO: (che download): Checking for image 'eclipse/che-init:5.0.1'...
not found
DEBUG: update_image
INFO: (che download): Pulling image eclipse/che-init:5.0.1

docker pull eclipse/che-init:5.0.1 >> "/data/cli.log" 2>&1

DEBUG: update_image_if_not_found
INFO: (che download): Checking for image 'eclipse/che-server:5.0.1'...
not found
DEBUG: update_image
INFO: (che download): Pulling image eclipse/che-server:5.0.1

docker pull eclipse/che-server:5.0.1 >> "/data/cli.log" 2>&1

INFO: (che init): Installing configuration and bootstrap variables:
mkdir -p "/data"
mkdir -p "/data/instance"
DEBUG: docker_run
INFO: (che init): CHE_HOST=172.17.0.1
INFO: (che init): CHE_VERSION=5.0.1
INFO: (che init): CHE_CONFIG=/usr/etc/che/data
INFO: (che init): CHE_INSTANCE=/usr/etc/che/data/instance
INFO: (che config): Generating che configuration...
DEBUG: generate_configuration_with_puppet
DEBUG: is_docker_for_windows
docker_run --env-file="/data/che.env" --env-
file=/version/5.0.1/images -v "/usr/etc/che/data/instance":/o
pt/che:rw -e "CHE_ENV_FILE=/usr/etc/che/da
ta/instance/config/che.env" -e "CHE_CONTAINER_ROOT=/data"
-e "CHE_ENVIRONMENT=production" -e "CHE_CONFIG
=/usr/etc/che/data/instance" -e "CHE_INSTANCE=/usr/etc/che/da
ta/instance" -e "CHE_REPO=off" --entrypoint
=/usr/bin/puppet eclipse/che-init:5.0.1
apply --modulepath /etc/puppet/modules
/ /etc/puppet/manifests/che.pp --show_diff >> "
/data/cli.log"
DEBUG: docker_run
Notice: Compiled catalog for 0074da0d9d9f.hlab.hits.local in environment prod
uction in 0.25 seconds
Notice: /Stage[main]/Base/File[/opt/che/data]/ensure: created
Notice: /Stage[main]/Base/File[/opt/che/config]/ensure: created
Notice: /Stage[main]/Base/File[/opt/che/logs]/ensure: created
Notice: /Stage[main]/Base/File[/opt/che/templates]/ensure: created
Notice: /Stage[main]/Base/File[/opt/che/stacks]/ensure: created
Notice: /Stage[main]/Che/File[/opt/che/config/che.env]/ensure: defined conten
t as '{md5}a8424d02c91d48070ed4c317d504f427'
Notice: /Stage[main]/Compose/Compose::Generate_compose_file[docker-compose-co
ntainer.yml]/File[/opt/che/docker-compose-container.yml]/ensure: defined conten
t as '{md5}77c55ec656289a6152183ea2ba0f45d8'
Notice: /Stage[main]/Compose/Compose::Generate_compose_file[docker-compose.ym
l]/File[/opt/che/docker-compose.yml]/ensure: defined content as '{md5}26d39b5c2
540166963f64bc1a4ae4a11'
Notice: Applied catalog in 0.09 seconds
INFO: (che config): Customizing docker-compose for running in a container
INFO: (che start): Preflight checks
DEBUG: port_open
port 8080 (http): [AVAILABLE]
DEBUG: port_open

INFO: (che start): Starting containers...
docker_compose --file="/data/instance/docker-compose-container.yml" -p="che" up
-d >> "/data/cli.log" 2>&1
DEBUG: docker_compose
DEBUG: docker_run
Unable to find image 'docker/compose:1.8.1' locally
1.8.1: Pulling from docker/compose
e110a4a17941: Already exists
92120570534d: Pulling fs layer
cdbf7c00098c: Pulling fs layer
0be58b1540e4: Pulling fs layer
c75e78215727: Pulling fs layer
dd11baf6e7b0: Pulling fs layer
c75e78215727: Waiting
dd11baf6e7b0: Waiting
cdbf7c00098c: Verifying Checksum
cdbf7c00098c: Download complete
c75e78215727: Verifying Checksum
c75e78215727: Download complete
dd11baf6e7b0: Verifying Checksum
dd11baf6e7b0: Download complete
0be58b1540e4: Verifying Checksum
0be58b1540e4: Download complete
92120570534d: Verifying Checksum
92120570534d: Download complete
92120570534d: Pull complete
cdbf7c00098c: Pull complete
0be58b1540e4: Pull complete
c75e78215727: Pull complete
dd11baf6e7b0: Pull complete
Digest: sha256:028833587aba5ce443693a7c75415fb05e9f374fa309f8b89e1efd13d1cd22db
Status: Downloaded newer image for docker/compose:1.8.1
Creating che
docker inspect -f '{{.Id}}' che
INFO: (che start): Services booting...
INFO: (che start): Server logs at "docker logs -f che"
/usr/bin/curl "-I -k 172.17.0.1:8080/api/ -s -o /dev/null --write-out %{http_c
ode}"
000
sleep 2
/usr/bin/curl "-I -k 172.17.0.1:8080/api/ -s -o /dev/null --write-out %{http_c
ode}"
200
/usr/bin/curl "-I -k 172.17.0.1:8080/api/ -s -o /dev/null --write-out %{http_c
ode}"
200
DEBUG: is_docker_for_mac
INFO: (che start): Booted and reachable
DEBUG: is_initialized
INFO: (che start): Ver: 5.0.1
INFO: (che start): Use: http://172.17.0.1:8080
INFO: (che start): API: http://172.17.0.1:8080/swagger

docker info output:
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 17
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-253:0-51855641-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.991 GB
Data Space Total: 107.4 GB
Data Space Available: 2.907 GB
Metadata Space Used: 4.293 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.143 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.64 GiB
Name: hdkj-host04.hlab.hits.local
ID: UYPS:CFUS:FM7C:KBNP:5BGT:FWO7:6GR4:W5HE:TG6F:MKC2:W7MX:72LR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: mmalyj
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8

[TylerJewell]

Thank you for a thorough summary of the issue. I am typing this on my phone so apologies for brevity. My first reaction is that you may have firewall blocking necessary ports between Che server and your ws. Can you take a look at the firewall section in our docs under installation and configuration?

[mmalyj]

I found the Ports and Firewall sections in the docs https://eclipse-che.readme.io/docs/networking. I don't understand what action to take from the Ports section, so I stopped my CentOS 7 firewall completely using #service firewalld stop. Then I did another Create Workspace, this time I got

Error when starting agent. Unable to start workspace agent. Error when trying to start the workspace agent: Start of environment nodejs_app7 failed. Error: Error response from docker API, status: 500, message: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i br-85cec8738ea8 -j RETURN: iptables: No chain/target/match by that name. (exit status 1))

[TylerJewell]

I am not a strong iptables guy. I just want to help while I am online today. I suspect that your firewall is still on somehow.

[mmalyj]

I made sure the CentOS 7 firewall was stopped and disabled with # systemctl stop firewalld, # systemctl disable firewalld, and # shutdown -r now. Then for good measure, I added EXPOSE 8000 to my stack definition FROM mmalyj/nodejs_app:21 (since my container is accessed over 8000). I then repeated accessing Eclipse Che from my Windows computer's Chrome going to http://10.1.2.75:8080. When I did another Create Workspace I got further than before, but ended up this error:

It seems that your workspace is running, but we cannot connect your browser to it. This commonly happens when Che was not configured properly. If your browser is connecting to workspaces running remotely, then you must start Che with the --remote: flag where the is the IP address of the node that is running your Docker workspaces.Please restart Che with this flag. You can read about what this flag does and why it is essential at: https://eclipse-che.readme.io/docs/configuration#envrionment-variables

So then I went to my CentOS 7.2 console, and aimed Firefox at http://10.1.2.75:8080. This time the workspace got created with no error! Thanks for the help, Tyler.

[TylerJewell]

Whew - not sure how much help I was there, but really glad this got resolved for you @mmalyj. What are you planning to use Che for?

[mmalyj]

I am trying to introduce microservices development into my company, a U.S. Federal IT contractor. I'm a DevOps guy, mostly Ops, and am working part-time on a pipeline demo to show to my management of Eclipse Che > Github > Jenkins > Docker Hub > Mesos/Marathon (to illustrate scaling to production). I have the last four parts done, now I am adding Eclipse Che to the front end. Now that I have a workspace running in Eclipse Che using my production container ( @bmicklea explained that part to me), I have to hook up Che to Github to get my hello world code, modify it, and send back to Github. Then I have to slide deck for management to follow.

I know this is old hat for you, but the government and its contractors are years behind. I'm doing just a single microservice for now, I am excited that Eclipse Che 5.0 deals with multi-machines much better.

[TylerJewell]

@mmalyj - they are not that far behind, believe it or not. Most of Codenvy's paying customers are in highly regulated environments, similar to the government. The sort of automation and control that a hosted workspace offers is appealing for that use case.

When you get these basics working and you have a good project working within your repository, we'll teach you about Chedir and Chefiles, a tactic to describe the repo in such a way so that the end user only has to git clone <repo>; docker run <STUFF> eclipse/che dir up, and then we will launch Che with the stack, with the workspace, with the project from the clone, with the right project type, all ready to compile and run.