New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very slow startup without connection to MongoDB #1249

Closed
bernd opened this Issue Jun 15, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@bernd
Member

bernd commented Jun 15, 2015

Starting the server without having a MongoDB running results in a very slow startup (look at the timestamps in the log below) and an error message eventually.

The timeout seems to be 10 seconds, no idea why it takes so long until the MongoDB error shows up.

It makes it really hard to see what's going on, why the server starts so slow. Is there a way to make the error show up earlier?

If I start MongoDB during server startup, the error shows up immediately.

2015-06-15 13:47:19,915 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugins: []
2015-06-15 13:47:20,078 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:41638,suspend=y,server=n -Djava.library.path=lib/sigar-1.6.4 -Xmx1g -XX:NewRatio=1 -XX:PermSize=512m -XX:MaxPermSize=1024m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dfile.encoding=UTF-8
2015-06-15 13:47:23,014 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2015-06-15 13:47:23,232 INFO : kafka.log.LogManager - Loading log 'messagejournal-0'
2015-06-15 13:47:23,296 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal
2015-06-15 13:47:23,306 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2015-06-15 13:47:33,490 INFO : org.graylog2.plugin.system.NodeId - Node ID: f9ddc5b7-3fdc-487e-989c-90414fc11280
2015-06-15 13:47:33,808 INFO : org.elasticsearch.node - [graylog2-server] version[1.5.2], pid[24940], build[62ff986/2015-04-27T09:21:06Z]
2015-06-15 13:47:33,808 INFO : org.elasticsearch.node - [graylog2-server] initializing ...
2015-06-15 13:47:33,816 INFO : org.elasticsearch.plugins - [graylog2-server] loaded [graylog2-monitor], sites []
2015-06-15 13:47:36,186 INFO : org.elasticsearch.node - [graylog2-server] initialized
2015-06-15 13:47:36,193 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:47:37,937 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2015-06-15 13:47:47,973 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:47:48,043 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:08,073 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:18,090 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:28,109 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:38,127 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:48,161 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:48:58,208 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:08,223 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:18,313 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:28,329 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:38,341 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:48,380 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
Disconnected from the target VM, address: '127.0.0.1:41638', transport: 'socket'
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,393 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]
2015-06-15 13:49:58,394 ERROR: org.graylog2.bootstrap.commands.Server - 

################################################################################

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused}}]

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

################################################################################


Process finished with exit code 255

@bernd bernd added the bug label Jun 15, 2015

@bernd

This comment has been minimized.

Member

bernd commented Jun 15, 2015

Tried to lower mongodb_threads_allowed_to_block_multiplier and mongodb_max_connections, doesn't change anything.

bernd added a commit that referenced this issue Jun 15, 2015

@joschi joschi closed this in #1252 Jun 17, 2015

@joschi joschi added this to the 1.2.0 milestone Jun 17, 2015

bernd added a commit that referenced this issue Jun 17, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment