New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graylog improperly handling timestamps #1320

Closed
bluegeist opened this Issue Jul 22, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@bluegeist

bluegeist commented Jul 22, 2015

I have a Cisco FireSIGHT that is sending logs to Graylog. Graylog is parsing the messages improperly and seting adding 7 hours to the timestamp. Even if it is assuming UTC because it cannot determine the time zone how is it that its adding 7 hours when I live in PDT which is minus 7 hours from UTC?

See an example message below

<118>Jul 22 15:53:51 btsdefense SFIMS: [119:15:1] http_inspect: OVERSIZE REQUEST-URI DIRECTORY [Impact: Potentially Vulnerable] From "btsips1" at Wed Jul 22 15:53:50 2015 UTC [Classification: Pornography was Detected] [Priority: 2] {tcp} 10.107.5.116:60603->10.106.240.176:9004

The message clearly has the UTC timezone in it. I have other devices that call out timezone the same way and they get tagged properly.

I have tried playing around with extractors to see if they will work but so far nothing has been successful.

All of my timestamps are set to PDT. Including the server that Graylog is installed on as well as Graylog config files.

capture

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jul 23, 2015

Hi,

If I understood it correctly, you are referring to an extracted timestamp from the log message, not to the default timestamp value in the syslog message. Could you please share the configuration of the extractor you use to get the date? It's hard to debug this issue otherwise.

@bluegeist

This comment has been minimized.

bluegeist commented Jul 23, 2015

Yes correct the timestamp that is being sent by the device contains the correct timestamp. Graylog extracts the timestamp then adds 7 hours to it. The timestamps being sent by my device are set to UTC. I live in PDT which is UTC -7 so I do not get alerts until 7 hours AFTER the fact, and they do not show up in my dashboard. I can find the message if I set the time to include messages that are at least 7 hours in the future though.

The only thing that makes sense to me is that Graylog is assuming that the messages are set to PDC timezone, I don't understand why though because the original messages do not contain anything that says PDC or UTC-7 or anything that could be misconstrued as PDC or PTC. What confuses me is that if Graylog assumes UTC if it cannot determine the timezone then why is it extracting as PDC when there is nothing in the original message that can be interpreted as PDC or PTC.

My original message shows a very typical example of what comes into Graylog and the screenshot has what is extracted. It definitely adds 7 hours to extracted timestamp.

Here is the exported extractor

{
  "extractors": [
    {
      "condition_type": "regex",
      "condition_value": "(btsdefense\\sSFIMS)",
      "converters": [
        {
          "config": {
            "date_format": "eee  MMM dd HH:mm:ss YYYY zzz"
          },
          "type": "date"
        }
      ],
      "cursor_strategy": "copy",
      "extractor_config": {
        "regex_value": "([a-zA-Z]{3}\\s[a-zA-Z]{3}\\s[0-9]{2}\\s[0-9]{2}:[0-9]{2}:[0-9]{2}\\s[0-9]{4}\\sUTC)"
      },
      "extractor_type": "regex",
      "order": 0,
      "source_field": "message",
      "target_field": "timestamp",
      "title": "Timestamp"
    }
  ],
  "version": "1.1.4 (59783f6)"
}
@joschi

This comment has been minimized.

Contributor

joschi commented Jul 24, 2015

The DateConverter is indeed using the locally configured timezone as timezone for the converted dates.

We've changed that in Graylog 1.1.x for the flexible date converter (https://github.com/Graylog2/graylog2-server/blob/1.1.4/graylog2-server/src/main/java/org/graylog2/inputs/converters/FlexibleDateConverter.java), so you might use that as a workaround for now.

@joschi joschi added the bug label Jul 24, 2015

joschi added a commit that referenced this issue Jul 24, 2015

joschi added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jul 24, 2015

joschi added a commit that referenced this issue Jul 24, 2015

@bluegeist

This comment has been minimized.

bluegeist commented Jul 24, 2015

Using the flexible date converter seems to have fixed my issue. Thanks!

@joschi joschi added this to the 1.2.0 milestone Jul 27, 2015

@joschi joschi self-assigned this Jul 27, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment