New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch index failures not log into log file #4166

Closed
jalogisch opened this Issue Sep 20, 2017 · 0 comments

Comments

Projects
None yet
3 participants
@jalogisch
Member

jalogisch commented Sep 20, 2017

Expected Behavior

If Graylog is not able to ingest the messages to elasticsearch it should write a meaningful log message that contains something the admin can work with. That should contain what and why with as much information as possible.

The information / log should be found in the server.log as this would be the place where ppl check if anything is not working.

Current Behavior

In the MongoDB we have currently the following entry:

{
    "timestamp": "2017-09-17T13:32:19.000Z",
    "letter_id": "a11dbb91-9bac-11e7-9aa5-782bcb5376e0",
    "message": "{\"type\":\"unavailable_shards_exception\",\"reason\":\"[graylog_5728][0] Not enough active copies to meet write consistency of [QUORUM] (have 1, needed 2). Timeout: [1m], request: [BulkShardRequest to [graylog_5728] containing [113] requests]\"}",
    "index": "graylog_5728",
    "type": "message"
  },

But in the Logfile during that time is only visible that a deflector cycle had happened.

2017-09-17T13:32:18.964Z INFO  [MongoIndexSet] Pointing index alias <graylog_deflector> to new index <graylog_5728>.
2017-09-17T13:32:19.107Z INFO  [SystemJobManager] Submitted SystemJob <a0b62020-9bac-11e7-9aa5-782bcb5376e0> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2017-09-17T13:32:19.107Z INFO  [MongoIndexSet] Successfully pointed index alias <graylog_deflector> to index <graylog_5728>.
2017-09-17T13:32:49.296Z INFO  [SetIndexReadOnlyJob] Flushing old index <graylog_5727>.
2017-09-17T13:32:49.945Z INFO  [SetIndexReadOnlyJob] Setting old index <graylog_5727> to read-only.
2017-09-17T13:32:51.115Z INFO  [SystemJobManager] Submitted SystemJob <b3ca4fb6-9bac-11e7-9aa5-782bcb5376e0> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]

Context

In case something unexpected happen we could not shift between different location of possible log information. In fact the server.log should be the single source of information. To make investigation easier.

Additional the given Information should be with out a doubt useful and say what happened.

Your Environment

  • Graylog Version: 2.3.1

joschi added a commit that referenced this issue Sep 29, 2017

Log indexing failures on WARN level instead of TRACE
Additionally to the "Indexer failures" in the web interface (System / Overview),
indexing failures are now also logged on WARN level.

Closes #4166

@joschi joschi self-assigned this Sep 29, 2017

joschi added a commit that referenced this issue Sep 29, 2017

Log indexing failures on WARN level instead of TRACE
Additionally to the "Indexer failures" in the web interface (System / Overview),
indexing failures are now also logged on WARN level.

Closes #4166

@wafflebot wafflebot bot added the in progress label Sep 29, 2017

@bernd bernd closed this in #4195 Sep 29, 2017

@wafflebot wafflebot bot removed the in progress label Sep 29, 2017

bernd added a commit that referenced this issue Sep 29, 2017

Log indexing failures on WARN level instead of TRACE (#4195)
Additionally to the "Indexer failures" in the web interface (System / Overview),
indexing failures are now also logged on WARN level.

Closes #4166

bernd added a commit that referenced this issue Sep 29, 2017

Log indexing failures on WARN level instead of TRACE (#4195)
Additionally to the "Indexer failures" in the web interface (System / Overview),
indexing failures are now also logged on WARN level.

Closes #4166

(cherry picked from commit 5e112bd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment