Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graylog not starting after upgrade to 3.1 #6322

Closed
JTabel opened this issue Aug 19, 2019 · 3 comments · Fixed by #6333

Comments

@JTabel
Copy link

commented Aug 19, 2019

After upgrading from 3.0 to 3.1 graylog did not start anymore. Graylog server started it's startup process but failed when connecting to the mongodb.

Expected Behavior

Graylog server should start

Current Behavior

Graylog service stops shortly after starting with the following log message:

2019-08-19T11:45:07.637+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 67: 'namespace name generated from index name "graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1" is too long (127 byte max)' on server mongodb.example.com:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "ok" : 0.0, "errmsg" : "namespace name generated from index name \"graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1\" is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "96i/ZHdqiWilzORS27VWweXhwEg=", "$type" : "00" }, "keyId" : { "$numberLong" : "6701582727742750722" } } } }
2019-08-19T11:45:07.637+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 67: 'namespace name generated from index name "graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1" is too long (127 byte max)' on server mongodb.example.com:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "ok" : 0.0, "errmsg" : "namespace name generated from index name \"graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1\" is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "96i/ZHdqiWilzORS27VWweXhwEg=", "$type" : "00" }, "keyId" : { "$numberLong" : "6701582727742750722" } } } }

I have obfuscated the real clustername and mongodb host, though the actual cluster would have 20 characters. It seems to hit a hard limit in mongodb versions before 4.2: https://docs.mongodb.com/manual/reference/limits/#Index-Name-Length

Possible Solution

Upgrade to mongodb 4.2 or higher (untested) or downgrade to graylog 3.0 (tested). Also maybe a shorter cluster name might work, though that is not really a solution in our case.

Steps to Reproduce (for bugs)

  1. Install graylog 3.0
  2. connect it to mongodb 3.6 (our current running version)
  3. upgrade to graylog 3.1
  4. graylog will fail to start (if the cluster name is at least 20 characters)

Context

I tried to test the upgrade to graylog 3.1 on a test cluster. No production impact.

Your Environment

  • Graylog Version: 3.1
  • Elasticsearch Version: 6.7.2
  • MongoDB Version: 3.6.12 (percona)
  • Operating System: ubuntu 18.04
@kmerz

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

@JTabel Thank you for the report!

With cluster name you mean the database name? The complete index name in the error message seems to be less than 127 characters.

We will change the index name in the next bug fix release so it's not an issue anymore in the future.

@bernd bernd self-assigned this Aug 21, 2019

bernd added a commit that referenced this issue Aug 21, 2019
Use custom name for the compound index in DBProcessingStatusService
Using the auto generated one can become a problem with the 127 character
limit in MongoDB if the database name is long.

Fixes #6322
@bernd

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

We are implementing a fix for this in #6333.

@JTabel As a workaround you can manually create the index in your MongoDB instance:

db.processing_status.createIndex(
  { "updated_at": 1, "input_journal.uncommitted_entries": 1, "input_journal.written_messages_1m_rate": 1 } ,
  { name: "compound_0" }
)

@bernd bernd added the bug label Aug 21, 2019

@JTabel

This comment has been minimized.

Copy link
Author

commented Aug 21, 2019

yes, database and cluster actually have the same name for use, so I might have confused it. Thanks for the quick fix, since we just tested the upgrade path on a preview cluster, we can wait for the bugfix. thanks for the help.

@JTabel JTabel closed this Aug 21, 2019

mpfz0r added a commit that referenced this issue Aug 22, 2019
Use custom name for the compound index in DBProcessingStatusService (#…
…6333)

Using the auto generated one can become a problem with the 127 character
limit in MongoDB if the database name is long.

Fixes #6322
mpfz0r added a commit that referenced this issue Aug 22, 2019
Use custom name for the compound index in DBProcessingStatusService (#…
…6333)

Using the auto generated one can become a problem with the 127 character
limit in MongoDB if the database name is long.

Fixes #6322

(cherry picked from commit eec1460)

@bernd bernd added this to the 3.1.1 milestone Aug 23, 2019

bernd added a commit that referenced this issue Aug 26, 2019
Use custom name for the compound index in DBProcessingStatusService (#…
…6333) (#6344)

Using the auto generated one can become a problem with the 127 character
limit in MongoDB if the database name is long.

Fixes #6322

(cherry picked from commit eec1460)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.