Graylog not starting after upgrade to 3.1

[JTabel]

After upgrading from 3.0 to 3.1 graylog did not start anymore. Graylog server started it's startup process but failed when connecting to the mongodb.

Expected Behavior

Graylog server should start

Current Behavior

Graylog service stops shortly after starting with the following log message:

2019-08-19T11:45:07.637+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 67: 'namespace name generated from index name "graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1" is too long (127 byte max)' on server mongodb.example.com:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "ok" : 0.0, "errmsg" : "namespace name generated from index name \"graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1\" is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "96i/ZHdqiWilzORS27VWweXhwEg=", "$type" : "00" }, "keyId" : { "$numberLong" : "6701582727742750722" } } } }
2019-08-19T11:45:07.637+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 67: 'namespace name generated from index name "graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1" is too long (127 byte max)' on server mongodb.example.com:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "ok" : 0.0, "errmsg" : "namespace name generated from index name \"graylog.processing_status.$updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1\" is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1566207903, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "96i/ZHdqiWilzORS27VWweXhwEg=", "$type" : "00" }, "keyId" : { "$numberLong" : "6701582727742750722" } } } }

I have obfuscated the real clustername and mongodb host, though the actual cluster would have 20 characters. It seems to hit a hard limit in mongodb versions before 4.2: https://docs.mongodb.com/manual/reference/limits/#Index-Name-Length

Possible Solution

Upgrade to mongodb 4.2 or higher (untested) or downgrade to graylog 3.0 (tested). Also maybe a shorter cluster name might work, though that is not really a solution in our case.

Steps to Reproduce (for bugs)

1. Install graylog 3.0
2. connect it to mongodb 3.6 (our current running version)
3. upgrade to graylog 3.1
4. graylog will fail to start (if the cluster name is at least 20 characters)

Context

I tried to test the upgrade to graylog 3.1 on a test cluster. No production impact.

Your Environment

• Graylog Version: 3.1
• Elasticsearch Version: 6.7.2
• MongoDB Version: 3.6.12 (percona)
• Operating System: ubuntu 18.04

[kmerz]

@JTabel Thank you for the report!

With cluster name you mean the database name? The complete index name in the error message seems to be less than 127 characters.

We will change the index name in the next bug fix release so it's not an issue anymore in the future.

[bernd]

We are implementing a fix for this in #6333.

@JTabel As a workaround you can manually create the index in your MongoDB instance:

db.processing_status.createIndex(
  { "updated_at": 1, "input_journal.uncommitted_entries": 1, "input_journal.written_messages_1m_rate": 1 } ,
  { name: "compound_0" }
)

[JTabel]

yes, database and cluster actually have the same name for use, so I might have confused it. Thanks for the quick fix, since we just tested the upgrade path on a preview cluster, we can wait for the bugfix. thanks for the help.