Skip to content

/ graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not allow arbitrary value for timestamp field #2064

Closed
kroepke opened this issue Apr 12, 2016 · 0 comments
Closed

Do not allow arbitrary value for timestamp field #2064

kroepke opened this issue Apr 12, 2016 · 0 comments
Assignees
@kroepke
Labels
improvement
Milestone
2.0.0

Comments

@kroepke
Copy link
Member

@kroepke kroepke commented Apr 12, 2016

Problem description

Message inputs that use part of an incoming message as timestamp value or extractors that overwrite timestamp can lead to malformed timestamp fields. This leads to losing messages, because the mapping for timestamp in elasticsearch is very strict.

Steps to reproduce the problem

  1. Send a minimal gelf message such as {"version":"1.1", "short_message":"foo","host":"hostname"}.
  2. Create an extractor copy_input on field message, overwriting the field timestamp.
  3. The message is discarded during indexing.

Environment

  • Graylog Version: all
  • Elasticsearch Version: all
  • MongoDB Version: -
  • Operating System: -
  • Browser version: -
@kroepke kroepke self-assigned this Apr 12, 2016
@kroepke kroepke added the improvement label Apr 12, 2016
@kroepke kroepke added this to the 2.0.0 milestone Apr 12, 2016
kroepke added a commit that referenced this issue Apr 12, 2016
@kroepke
override timestamp to now for bad timestamp values
Loading status checks…
fc9d004 
If the timestamp field of a message is incompatible to the ES format we are using, or an unknown data type (non Date, DateTime or String), force the timestamp to be "now".

This prevents bad extractors or inputs from causing these malformed messages to be discarded.

fix #2064
@kroepke kroepke mentioned this issue Apr 12, 2016
Merged
@bernd bernd closed this in #2065 Apr 12, 2016
bernd added a commit that referenced this issue Apr 12, 2016
@kroepke @bernd
override timestamp to now for bad timestamp values (#2065)
Loading status checks…
fc94e57 
If the timestamp field of a message is incompatible to the ES format we are using, or an unknown data type (non Date, DateTime or String), force the timestamp to be "now".

This prevents bad extractors or inputs from causing these malformed messages to be discarded.

fix #2064
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kroepke kroepke

Labels
improvement
Projects
None yet
Milestone
2.0.0
Linked pull requests

Successfully merging a pull request may close this issue.

override timestamp to now for bad timestamp values
1 participant
@kroepke