Docker: Connection refused: /127.0.0.1:12900

[anaion]

Trying to use the docker image on a CentOS Linux release 7.0.1406, but the web app hangs at startup with a connection refused error.

==> /var/log/graylog/web/current <==
2015-03-04_13:56:15.04150 [error] o.g.r.l.ApiClient - API call failed to execute.
2015-03-04_13:56:15.04154 java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: /127.0.0.1:12900 to http://127.0.0.1:12900/system/cluster/node
2015-03-04_13:56:15.04154   at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:342) ~[com.ning.async-http-client-1.8.14.jar:na]
2015-03-04_13:56:15.04155   at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:108) ~[com.ning.async-http-client-1.8.14.jar:na]
2015-03-04_13:56:15.04155   at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:431) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04155   at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:422) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04155   at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:384) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04156 Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:12900 to http://127.0.0.1:12900/system/cluster/node
2015-03-04_13:56:15.04158   at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104) ~[com.ning.async-http-client-1.8.14.jar:na]
2015-03-04_13:56:15.04158   at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:431) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04158   at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:422) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04158   at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:384) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04159   at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04159 Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:12900
2015-03-04_13:56:15.04159   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_71]
2015-03-04_13:56:15.04159   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_71]
2015-03-04_13:56:15.04159   at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04160   at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[io.netty.netty-3.9.3.Final.jar:na]
2015-03-04_13:56:15.04161   at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[io.netty.netty-3.9.3.Final.jar:na]

I checked the logs inside the container and elasticsearch and mongodb are up. This happens both when using the Docker Hub Registry, as well as for the locally built image.

[mariussturm]

Could you please provide a longer dump of the server logs? When ES and MongoDB are running, Graylog server should also start.

[anaion]

Hi,

Here are the container logs:
https://onedrive.live.com/redir?resid=A4F0B06FF24DE882%21118

Thanks for looking into this!

[mariussturm]

Really odd! The server logs look fine but nothing is listening on the API port. The only thing that irritates me is the beginning of the server log. It looks like the server was started two times. Could you please check how many graylog-server processes are running? Something like ps ax| grep graylog.jar would give informations.

[anaion]

Checked and there's only one graylog process running.

/opt/graylog/embedded/jre/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m -XX:MaxPermSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -jar /opt/graylog/server/graylog.jar server -f /opt/graylog/conf/graylog.conf

[mariussturm]

Sorry can't reproduce the issue. Just checked here a couple of times :/

[chuckleb]

Ah, I'm having the same problem in my Cent7 system, I'm not sure if it's related to firewalld or not. I'm more used to running this under Cent6 with iptables.

I've deleted all images and pulled fresh copies of the docker container just now and am getting a new error of Chef dying. Will make a new ticket

[chuckleb]

Ok just to update, it seems to cause this if I point to my existing data. I destroyed everything and just ran it from inside the container completely and things worked fine, no problems that anaion was having.

@anaion Can you post your startup string? Do you crash if you start like this?

docker run -t -p 9000:9000 -p 12201:12201 graylog2/allinone

I tested with firewalld up and down.

[mariussturm]

Thanks for the update!
It is possible that parts of your data got corrupted during a container stop/remove. I will take a look how to improve this. Is the container also not starting when you move the journal data away from the persistent folder?

[mariussturm]

Latest image is shutting down all services gracefully when the container gets a docker stop. This should prevent your problem.

[curzonj]

I got this issue because I'm trying to run graylog on OS X with boot2docker. https://gist.github.com/sevastos/2fa4c86f01541351efe8

Due to this mongodb never started and therefore the api server never started.