Che server should connect to ws-agent on internal URL #2030

ghost · 2016-08-03T06:40:55Z

Cannot run Che in Dockert

Expected behavior:
Che starts

Observed behavior:

eugene@eugene:~/projects/artik-ide (master)$ docker run -t -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher start
Unable to find image 'codenvy/che-launcher:latest' locally
latest: Pulling from codenvy/che-launcher

4fe15f8d0ae6: Already exists 
4d8efc227322: Already exists 
5f70bf18a086: Pull complete 
6fb31488f897: Already exists 
5b30f66f120f: Pull complete 
4c092e74c599: Pull complete 
66227230d784: Pull complete 
Digest: sha256:ab282d4f1239610fd2a7f80ac9cb6e7e6edd2d97dabef64608a4083c939de5ad
Status: Downloaded newer image for codenvy/che-launcher:latest
INFO: ECLIPSE CHE: PULLING IMAGE codenvy/che-server:codenvy/che-launcher

FATA[0000] Invalid repository name (che-server:codenvy/che-launcher), only [a-z0-9-_.] are allowed

Che version: [Enter Che version here]
OS and version: Ubuntu 14.04
Docker version: 1.11.2

The text was updated successfully, but these errors were encountered:

ghost · 2016-08-03T06:41:13Z

@l0rd can you reproduce it?

l0rd · 2016-08-03T06:54:37Z

This has been fixed in #1983 but codenvy/che-launcher hasn't been updated.

l0rd · 2016-08-03T06:55:31Z

@riuvshin can you please check?

ghost · 2016-08-04T06:38:08Z

@l0rd I tried it with nightly

Fail pinging ws agent. Workspace ID:workspace94c7eunv699lec0u. Url:http://172.19.20.16:32773/wsagent/ext/. Timestamp:{}

Server cannot reach agent, disabling ufw solves the problem.

Are we OK with this?

l0rd · 2016-08-04T08:16:16Z

We had an issue with the firewall on centos: the launcher could not verify if the server was up or not. So that's a different issue.

Can you please copy/paste the output of:
docker run -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly info

ghost · 2016-08-04T13:17:31Z

eugene@eugene:~$ docker run -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly info
DEBUG: ---------------------------------------
DEBUG: ---------  CHE DEBUG INFO   -----------
DEBUG: ---------------------------------------
DEBUG: 
DEBUG: DOCKER_INSTALL_TYPE       = native
DEBUG: 
DEBUG: CHE_SERVER_CONTAINER_NAME = che-server
DEBUG: CHE_SERVER_IMAGE_NAME     = codenvy/che-server
DEBUG: 
DEBUG: CHE CONTAINER EXISTS?     YES
DEBUG: CHE CONTAINER IS RUNNING? NO
DEBUG: CHE CONTAINER IS STOPPED? YES
DEBUG: CHE SERVER IS BOOTED?     YES
DEBUG: 
DEBUG: CHE_PORT                  = 8080
DEBUG: CHE_VERSION               = nightly
DEBUG: CHE_RESTART_POLICY        = no
DEBUG: CHE_USER                  = root
DEBUG: CHE_HOST_IP               = 172.19.20.16
DEBUG: CHE_LOG_LEVEL             = info
DEBUG: CHE_HOSTNAME              = localhost
DEBUG: CHE_DATA_FOLDER           = /home/user/che
DEBUG: CHE_CONF_FOLDER           = not set
DEBUG: CHE_LOCAL_BINARY          = not set
DEBUG: 
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------

l0rd · 2016-08-04T13:31:33Z

@eivantsov you don't have the latest nighlty of the launcher. Can you try to:

docker pull codenvy/che-launcher:nightly && docker pull codenvy/che-server:nightly

and test to see if you can reproduce the bug?

TylerJewell · 2016-08-04T13:45:00Z

That likely will fail as our ci systems have been down for a couple days due to weather. We will need to help Eugene to build the nightly locally for testing.

ghost · 2016-08-04T13:51:55Z

I updated all nightly images. The same result when ufw is on.

l0rd · 2016-08-04T13:56:34Z

@TylerJewell is right you may not have latest fix (nighlty are 3 days old). You can create the images from the root of the repo:

docker build -t codenvy/che-launcher:nightly dockerfiles/che-launcher/
docker build -t codenvy/che-server:nightly -f dockerfiles/che-server/Dockerfile .

l0rd · 2016-08-04T15:49:46Z

If this fails can you confirm that:

host => wsagent SUCCEED:
curl http://<yourip>:<yourport>/wsagent/ext/
wsmaster => wsagent (external address) FAILS:
docker exec -ti che-server curl http://<yourip>:<yourport>/wsagent/ext/
wsmaster => wsagent (internal address) SUCCEED:
docker exec -ti che-server curl http://<wsagent-container-ip>:4401/wsagent/ext/

This looks really similar to #1924 where che-launcher failed to ping wsmaster. Here is wsmaster fails to connect to wsagent but the reason is the same: the firewall is blocking containers to host connections.

TylerJewell · 2016-08-04T18:38:07Z

@l0rd - these are some great tips. I am going to add them into a diagnostics section for people who are setting up Che the first time.

When you run the last one, I get the IP address from doing a docker inspect on the workspace container that was created. I get the internal IP address. HOwever, the output doesn't look suitable. Is this what you would expect? I have a valid and running workspace.

C:\codenvy\che>docker exec -ti che-server curl http://172.17.0.2:4401/wsagent/ext
<!DOCTYPE html><html><head><title>Apache Tomcat/8.0.32 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style> </head><body><h1>HTTP Status 404 - /wsagent/ext</h1><div class="line"></div><p><b>type</b> Status report</p><p><b>message</b> <u>/wsagent/ext</u></p><p><b>description</b> <u>The requested resource is not available.</u></p><hr class="line"><h3>Apache Tomcat/8.0.32</h3></body></html>

l0rd · 2016-08-04T20:25:29Z

@TylerJewell you are missing the trailing slash in your path
http://172.17.0.2:4401/wsagent/ext/

On Thursday, 4 August 2016, Tyler Jewell notifications@github.com wrote:

@l0rd https://github.com/l0rd - these are some great tips. I am going
to add them into a diagnostics section for people who are setting up Che
the first time.

When you run the last one, I get the IP address from doing a docker
inspect on the workspace container that was created. I get the internal IP
address. HOwever, the output doesn't look suitable. Is this what you would
expect? I have a valid and running workspace.

C:\codenvy\che>docker exec -ti che-server curl http://172.17.0.2:4401/wsagent/ext
<title>Apache Tomcat/8.0.32 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style>
HTTP Status 404 - /wsagent/ext
type Status report
message /wsagent/ext
description The requested resource is not available.
Apache Tomcat/8.0.32

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2030 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAlC70Z9xLCDROaLyDDH0kBRsjYFdiNjks5qcjGQgaJpZM4JbWfe
.

TylerJewell · 2016-08-05T18:06:07Z

@l0rd - I have added in all of these tests into our documentation and also into the Che CLI. You can now run "che debug --networking" to get a run through of four tests, the three tests you provide and a web socket connectivity test. See the PR here. #2057

JamesDrummond · 2016-08-08T00:32:38Z

@eivantsov Please provide status label.

l0rd · 2016-08-08T09:33:59Z

A couple of things about this issue:

The title should be changed to something like: Che server cannot connect to the wsagent if the firewall is activated on Ubuntu. Current title is misleading: that's not about running Che in Docker. ~~This connectivity issue exists even if Che server is started as a java application outside a container~~.
Even if we add the tests in che.sh this issue should still be fixed. Che server should use wsagent interal IP Address when both are running in containers.

TylerJewell · 2016-08-08T19:16:39Z

@eivantsov - I believe this issue is no longer active and can be resolved due to recent changes made with the launcher. Can you verify and then we will close issue permanently?

l0rd · 2016-08-08T23:27:24Z

@TylerJewell I think that the issue is still there. Even if we added some checks and provided a workaround (disable the firewall) we could still fix it and let users use both Che and their firewall. What do you think?

TylerJewell · 2016-08-09T03:34:43Z

@l0rd - not quite following your suggestion here. I was unable to reproduce it this evening - and we have the connectivity tests which can help diagnose this. Can you be more specific on what you are thinking?

ghost · 2016-08-09T05:34:03Z

@TylerJewell i have updated che-launcher and che-server images. The problem is still there. If firewall is on, Che server cannot reach workspace agent.

TylerJewell · 2016-08-09T12:04:21Z

Can you paste the output of 'che.sh --networking'. Please the script from master.

l0rd · 2016-08-09T13:41:12Z

@TylerJewell currently Che server uses the external IP to contact wsagent (http://198.75..57.22:32773/ in the diagram). Every HTTP request does the roundrip to and from eth0 (1 and 2 in the diagram).

As a result, if a firewall blocks requests to eth0 coming from containers, the Che server won't be able to connect to the wsagent.

Che server could instead connect to wsagent using the internal URL (http://172.17.0.3:4401/). This would be a security improvement too and would help when Che will run in swarm cluster. But needs some coding, configuration is not enough. That's why I suggest to keep this issue open.

TylerJewell · 2016-08-09T21:51:49Z

The summary and graphic makes it really clear. Thanks, @l0rd.

l0rd · 2016-09-22T14:33:27Z

@amisevsk from RedHat is going working on that

JasonB42 · 2016-10-12T01:38:39Z

In case others come here with the same issue, I was finally able to run Eclipse Che on Ubuntu 16.04 with an active ufw firewall blocking external access. Can't claim this is the best or correct way, just glad to get it working.

Running che with the firewall active would fail to at "Initializing workspace" with the "Could not start workspace ... Timeout reached" error. Running che info --networking command showed:

$ bin/che info --networking
INFO: ---------------------------------------
INFO: -------- CONNECTIVITY TEST --------
INFO: ---------------------------------------
INFO: Browser => Workspace Agent (Hostname) : Connection succeeded
INFO: Browser => Workspace Agent (External IP): Connection succeeded
INFO: Che Server => Workspace Agent (External IP): Connection failed
INFO: Che Server => Workspace Agent (Internal IP): Connection succeeded

The solution was to allow intra-docker communication while blocking docker inserted rules to open ports to the public internet.

This command allows intra-docker container communication
$ sudo iptables -I INPUT -i docker0 -j ACCEPT
This stops external access by intercepting docker public access -> internal port forward rules
$ sudo iptables -t nat -I DOCKER ! -i docker0 -j RETURN

To access the server remotely, I am using a ssh SOCKS proxy with a session of chrome specifically launched to use that proxy:
$ ssh -Dlocalhost:5000 user@remote-che-machine
$ google-chrome --proxy-server=socks://localhost:5000 --user-data-dir=./che-chrome-data/

Hope this helps someone!

TylerJewell · 2016-10-12T01:53:59Z

That is amazing, @JasonB42 - thank you for the great contribution.

garagatyi · 2016-10-12T12:59:24Z

What about new functioanlity contributed by @l0rd that allows usage of different IPs for internal vs external communication?
@l0rd don't you think it can hlp in that case instead of playing with IPTables?

JasonB42 · 2016-10-12T13:19:37Z

@garagatyi That would be the preferred solution going forward. Is @lord's code available now? Did I overlook how to set that up?

l0rd · 2016-10-12T13:42:51Z

@JasonB42 @garagatyi I'm not sure that #2402 will help here. We can try to set two different IPs using env variables or che.properties:

machine.docker.local_node_host.external=<eth0 IP>
machine.docker.local_node_host=<docker0 IP>

but I'm afraid that some IP tables rule will reject packets coming from the che-server directed to docker0.

bmicklea · 2016-11-17T16:34:42Z

What's the status of this issue? Is it being worked on by anyone?

TylerJewell · 2016-11-17T16:41:29Z

Yes, there is a PR on this - ongoing development with Red Hat and Codenvy - multiple people on this. It will be solved (hopefully) in time for GA.

Add classes to support refactoring of DockerInstanceRuntimeInfo. Adds interface ServerEvaluationStrategyProvider, which provides objects implementing ServerEvaluationStrategy, an interface which supports getting servers from ContainerInfo json via port protocol. Additionally adds default ServerEvaluationStrategy which represents the current getServers() functionality. Additionally, DefaultServerEvaluationStrategy supports functionality required by issue eclipse-che#2030 (Che server should connect to ws-agent on internal URL) via property che.docker.ip.use_internal_address Signed-off-by: Angel Misevski <amisevsk@redhat.com>

Refactor DockerInstanceRuntimeInfo#getServers() (#2030)

Refactor DockerInstanceRuntimeInfo#getServers() (eclipse-che#2030)

ghost added the kind/bug Outline of a bug - must adhere to the bug report template. label Aug 3, 2016

TylerJewell changed the title ~~Cannot run Che in Docker~~ Che server cannot connect to the wsagent if the firewall is activated on Ubuntu Aug 8, 2016

bmicklea modified the milestone: 4.7.1 Aug 29, 2016

bmicklea added kind/enhancement A feature request - must adhere to the feature request template. and removed kind/bug Outline of a bug - must adhere to the bug report template. labels Aug 29, 2016

bmicklea changed the title ~~Che server cannot connect to the wsagent if the firewall is activated on Ubuntu~~ Che server should connect to ws-agent on internal URL Aug 29, 2016

TylerJewell added the status/open-for-dev An issue has had its specification reviewed and confirmed. Waiting for an engineer to take it. label Sep 22, 2016

bmicklea unassigned garagatyi Sep 22, 2016

amisevsk mentioned this issue Oct 19, 2016

Che server should connect to ws-agent on internal URL (#2030) #2837

Closed

6 tasks

vkuznyetsov added the sprint/next label Oct 21, 2016

vkuznyetsov assigned garagatyi and unassigned vkuznyetsov Oct 21, 2016

vkuznyetsov added sprint/current and removed sprint/next labels Oct 27, 2016

garagatyi added team/platform and removed sprint/current team/production labels Nov 9, 2016

amisevsk mentioned this issue Dec 5, 2016

Refactor DockerInstanceRuntimeInfo#getServers() (#2030) #3282

Merged

2 tasks

JamesDrummond modified the milestones: 5.0.0-M1, 5.1.0 Dec 20, 2016

l0rd mentioned this issue Jan 9, 2017

Che on a cloud server - Not able to create a workspace #3576

Closed

garagatyi closed this as completed in #3282 Jan 10, 2017

garagatyi pushed a commit that referenced this issue Jan 10, 2017

Merge pull request #3282 from amisevsk/CHE-2030-refactor

33a4d07

Refactor DockerInstanceRuntimeInfo#getServers() (#2030)

JamesDrummond mentioned this issue Jan 31, 2017

5.1.0 Milestone Overview #3972

Closed

9 tasks

JPinkney pushed a commit to JPinkney/che that referenced this issue Aug 17, 2017

Merge pull request eclipse-che#3282 from amisevsk/CHE-2030-refactor

25d87ef

Refactor DockerInstanceRuntimeInfo#getServers() (eclipse-che#2030)

bucs-fan813 mentioned this issue Feb 2, 2018

Che server is unable to ping your workspace, RHEL7, Browser => Workspace Agent (localhost): Connection failed #8595

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Che server should connect to ws-agent on internal URL #2030

Che server should connect to ws-agent on internal URL #2030

ghost commented Aug 3, 2016

ghost commented Aug 3, 2016

l0rd commented Aug 3, 2016

l0rd commented Aug 3, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016 •

edited

Loading

TylerJewell commented Aug 4, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016

l0rd commented Aug 4, 2016

TylerJewell commented Aug 4, 2016

l0rd commented Aug 4, 2016

HTTP Status 404 - /wsagent/ext

Apache Tomcat/8.0.32

TylerJewell commented Aug 5, 2016

JamesDrummond commented Aug 8, 2016

l0rd commented Aug 8, 2016 •

edited

Loading

TylerJewell commented Aug 8, 2016

l0rd commented Aug 8, 2016

TylerJewell commented Aug 9, 2016

ghost commented Aug 9, 2016

TylerJewell commented Aug 9, 2016

l0rd commented Aug 9, 2016

TylerJewell commented Aug 9, 2016

l0rd commented Sep 22, 2016

JasonB42 commented Oct 12, 2016

TylerJewell commented Oct 12, 2016

garagatyi commented Oct 12, 2016

JasonB42 commented Oct 12, 2016

l0rd commented Oct 12, 2016

bmicklea commented Nov 17, 2016

TylerJewell commented Nov 17, 2016

Che server should connect to ws-agent on internal URL #2030

Che server should connect to ws-agent on internal URL #2030

Comments

ghost commented Aug 3, 2016

ghost commented Aug 3, 2016

l0rd commented Aug 3, 2016

l0rd commented Aug 3, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016 • edited Loading

TylerJewell commented Aug 4, 2016

ghost commented Aug 4, 2016

l0rd commented Aug 4, 2016

l0rd commented Aug 4, 2016

TylerJewell commented Aug 4, 2016

l0rd commented Aug 4, 2016

HTTP Status 404 - /wsagent/ext

Apache Tomcat/8.0.32

TylerJewell commented Aug 5, 2016

JamesDrummond commented Aug 8, 2016

l0rd commented Aug 8, 2016 • edited Loading

TylerJewell commented Aug 8, 2016

l0rd commented Aug 8, 2016

TylerJewell commented Aug 9, 2016

ghost commented Aug 9, 2016

TylerJewell commented Aug 9, 2016

l0rd commented Aug 9, 2016

TylerJewell commented Aug 9, 2016

l0rd commented Sep 22, 2016

JasonB42 commented Oct 12, 2016

The solution was to allow intra-docker communication while blocking docker inserted rules to open ports to the public internet.

TylerJewell commented Oct 12, 2016

garagatyi commented Oct 12, 2016

JasonB42 commented Oct 12, 2016

l0rd commented Oct 12, 2016

bmicklea commented Nov 17, 2016

TylerJewell commented Nov 17, 2016

l0rd commented Aug 4, 2016 •

edited

Loading

l0rd commented Aug 8, 2016 •

edited

Loading