Graylog v2.0-alpha can't get to web interface

[123dev]

Problem description

Testing out Graylog v2.0-alpha

Steps to reproduce the problem

Installed on Ubuntu 14.04
Accessing http://graylog:9000
I get

Server currently unavailable
We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:12900//. Please verify that the server is healthy and working correctly.

You will be automatically redirected to the previous page once we can connect to the server.

Do you need a hand? We can help you.

Less details
This is the last response we received from the server:

Error message
Bad request
Original Request
GET http://127.0.0.1:12900/system/cluster/node
Status code
undefined
Full error message
Error: Request has been terminated Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

1. ...

Environment

• Graylog Version:
  Latest 2.0 alpha.5
• Elasticsearch Version:
  2.2.0
• MongoDB Version:
  3.2.3
• Operating System:
  Ubuntu 14.04 LTS

In graylog logs I see the following warning

2016-03-05 13:26:01,497 WARN : org.elasticsearch.discovery - [graylog-8721cfc4-0816-4be1-ba19-18ddbcb9cdae] waited for 3s and no initial state was set by the discovery
2016-03-05 13:26:01,498 INFO : org.elasticsearch.node - [graylog-8721cfc4-0816-4be1-ba19-18ddbcb9cdae] started
2016-03-05 13:26:04,926 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [10.1.1.94:9000]
2016-03-05 13:26:04,930 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2016-03-05 13:26:04,932 INFO : org.graylog2.initializers.WebInterfaceService - Started Web Interface at <http://10.1.1.94:9000/>
2016-03-05 13:26:06,505 WARN : org.graylog2.initializers.IndexerSetupService - Could not connect to Elasticsearch

I even changed the bind address to accept connections on all network interfaces.

tcp        0      0 0.0.0.0:12900           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9350            0.0.0.0:*               LISTEN
tcp        0      0 10.1.1.94:9000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN
tcp6       0      0 :::9200                 :::*                    LISTEN
tcp6       0      0 :::9300                 :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 ::1:6010                :::*                    LISTEN
tcp6       0      0 ::1:6011                :::*                    LISTEN

Based on the error reported on the web interface, Graylog is having problems connecting to
We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:12900//

Yet I'm able to access it from localhost

{"cluster_id":"855f6744-5881-4555-b11a-273793f9a538","node_id":"8721cfc4-0816-4be1-ba19-18ddbcb9cdae","type":"server","transport_address":"http://10.1.1.94:12900","last_seen":"2016-03-05T18:27:45.000Z","short_node_id":"8721cfc4","hostname":"Graylog.CELLS","is_master":true}

both
wget http://127.0.0.1:12900/system/cluster/node
and
wget http://10.1.1.94:12900/system/cluster/node
Work

ElasticSearch is also accesible
curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'

{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Even the client running the browser is able to access the REST_API and ES

What's wrong?
Am I missing something?

Thanks in advance

[123dev]

Not sure if this could have anything to do with the problem. I'm also seeing this in the logs.

2016-03-05 13:31:28,114 ERROR: org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical - Uncaught exception in periodical
org.elasticsearch.discovery.MasterNotDiscoveredException
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:205) ~[graylog.jar:?]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239) ~[graylog.jar:?]
        at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:794) ~[graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

[123dev]

This issue can be closed.

It turns out the reported error was misleading, and the root problem was related to the configuration of the discovery.
It might still be a bug, as I had left it at default (multicast) based on the docs here
http://docs.graylog.org/en/2.0/pages/configuring_es.html#configuration
My test setup is a single box environment, so was wrongly assuming that the default would have worked, to top it off the reported message sent me hunting in the wrong direction.

One other thing I noticed while trying to narrow down the problem.
When I noticed timeouts in the log, I increased this setting
elasticsearch_cluster_discovery_timeout = 55000

What I ended up getting was the server shutdown with the following message.

2016-03-05 17:49:50,339 ERROR: org.graylog2.bootstrap.ServerBootstrap - Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[IndexerSetupService [FAILED]]}
    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:713) ~[graylog.jar:?]
    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:542) ~[graylog.jar:?]
    at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:299) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:122) [graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196) [graylog.jar:?]
    at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2016-03-05 17:49:50,344 INFO : org.graylog2.commands.Server - SIGNAL received. Shutting down.

Not sure why that was happening, thought I'd report it.
In any case, the server is now operational.
Thanks for a great product.

[numair]

When running the graylog2/server image from the Docker repo, I'm encountering this problem when loading the web interface. The interface login screen loads, but eventually it goes to the error page. Since this happens with a fresh install of the image, I am assuming that this is a repeatable problem... Any ideas on a fix?

[joschi]

@numair Please refer to http://docs.graylog.org/en/2.0/pages/configuring_webif.html for information how to configure the Graylog web interface.

[numair]

Sorry, but this doesn't solve the problem of a Docker image installation
that is broken out-of-box. If anyone has an idea on how to get this to
work, let me know...

On Thursday, May 19, 2016, Jochen Schalanda notifications@github.com
wrote:

@numair https://github.com/numair Please refer to
http://docs.graylog.org/en/2.0/pages/configuring_webif.html for
information how to configure the Graylog web interface.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1908 (comment)

[joschi]

@numair We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!

[numair]

@joschi I noticed you guys have a lot of ongoing issues opened about this particular problem; the solution is pretty simple -- just edit your example commands on https://hub.docker.com/r/graylog2/server/ to run the Docker image to include

-e GRAYLOG_REST_TRANSPORT_URI='http://your_public_ip:12900'

... and then there will be far fewer confused people! There is mention of this in the comments on that page, but it is at the very bottom of the comment list, and thus simply following the instructions given makes it seem as though the image itself is broken. Thanks for all of your work on this great project!

[rfdrake]

The docker page I was following, http://docs.graylog.org/en/2.0/pages/installation/docker.html doesn't even have comments so the only way I was able to figure this out was to search for the error message and find this issue.

I did find in the configuration file documentation saying that it would use the first non-loopback ipv4 address if rest_listen_uri was set to a wildcard, but it still seemed like an odd thing to be changing if there wasn't a comment in the docker docs.

The purpose of GRAYLOG_PASSWORD_SECRET is also under-documented and there should be a suggestion to use "pwgen -N 1 -s 96" in the docker documentation (or the whole thing should be initialized on first boot with a random seed without user intervention)

If I can type something coherent I'll see about submitting a patch for the docs.. Although it might be better if it was done by someone more familiar with the project, or someone better at writing.

[kwerle]

It's disappointing the docker-compose fails out of the box. And the workaround of using the public IP address doesn't work real well for me because I'm deploying to a cluster and can't know (without more work) what the public IP address will be before deployment.

[bernd]

@mariussturm Is there anything we can do to make the Docker experience better?

[kroepke]

This should get a little easier by #2515

[esseti]

same problem here. it's a pitty that i've to put the public ip.. is there anything that can be automated or solved by using docker capability such as links?

[bbaassssiiee]

The error page says "Please verify that the server is healthy and working correctly." but there is no error in the log, nor a troubleshooting page on the website. Document getting the product to run, and troubleshooting when it doesn't. List the checks for all the components. Time does not permit that we are going through all the noise of a mailing list or an IRC channel.

[joschi]

@bbaassssiiee We also offer professional support services if you cannot or don't want to use the community support: https://www.graylog.org/professional-support

This being said, I think this issue isn't really actionable (and the questions are really answered in the documentation), so I'll close this issue.

[Pseudonomy]

Same issue, unable to make the docker compose work in a vagrant box. Setting GRAYLOG_REST_TRANSPORT_URI with a local public ip in the docker compose didn't work for me. How can I proceed with troubleshooting this?

[Pseudonomy]

Ah, made it work with also setting the GRAYLOG_WEB_ENDPOINT_URI to the local public ip address.