Capture unmatched lines #1421

Merged
merged 4 commits into from Jan 25, 2017

Projects

None yet

2 participants

@jitran
Contributor
jitran commented Jan 17, 2017

Essentially all unmatched lines will be placed inside the parse_fail attribute, i.e.

{  
   "parse_fail":"unmatched line....rest of the line",
   "server":"myserver",
   "stack":"my-app-dev-01",
   "application":"my-app",
   "log_type":"apache.access",
   "time":"2017-01-16T23:06:49Z"
}

This will enable our analytics tools to pick up any pattern matching regexp issues.

jitran added some commits Jan 16, 2017
@jitran jitran Add a new option for in_tail called 'enable_catch_all'. When set to t…
…rue, all singular and multiline log lines that don't match the parser regex will be stored in an attribute called parse_fail, and eventually saved in the output destination.
49ff636
@jitran jitran Use Time.now.to_i as its backwards compatible with the earlier versio…
…ns of fluentd
aba4b73
@repeatedly
Member

Additional comments

  • Need test for this parameter
  • How about adding unmatched_tag for emitting unmatched lines to other tag, not including same EventStream.
lib/fluent/plugin/in_tail.rb
@@ -65,6 +65,8 @@ def initialize
config_param :read_lines_limit, :integer, default: 1000
desc 'The interval of flushing the buffer for multiline format'
config_param :multiline_flush_interval, :time, default: nil
+ desc 'Enable the option to capture unmatched lines.'
+ config_param :enable_catch_all, :bool, default: false
@repeatedly
repeatedly Jan 19, 2017 Member

emit_unmatched_lines or emit_parse_failed_lines is more better.
enable_catch_all is unclear name for me.

lib/fluent/plugin/in_tail.rb
+ if @enable_catch_all
+ record = {'parse_fail' => line}
+ record[@path_key] ||= tail_watcher.path unless @path_key.nil?
+ es.add(Time.now.to_i, record)
@repeatedly
repeatedly Jan 19, 2017 Member

Use EventTime.now. Don't use Time directly for event time.

@jitran
jitran Jan 19, 2017 Contributor

We're using the Redhat version of td-agent which comes with Fluentd v0.12. Do you know when the 0.14 branch will be stable? If it's still quite long away, should I merge my branch into the 0.12 branch instead?

@repeatedly
repeatedly Jan 19, 2017 Member

If you want this feature in v0.12, you also need to send a patch to v0.12 branch.

@jitran jitran referenced this pull request Jan 20, 2017
Closed

#1416

@jitran
Contributor
jitran commented Jan 23, 2017

Hi @repeatedly, I have implemented the tests for the new parameter, but the existing unit tests seem to fail randomly between the different ruby releases, and I've noticed when the CI runs again, those tests pass again.

lib/fluent/plugin/in_tail.rb
+ if @emit_unmatched_lines
+ record = {'unmatched_line' => line}
+ record[@path_key] ||= tail_watcher.path unless @path_key.nil?
+ es.add(Time.now.to_i, record)
@repeatedly
repeatedly Jan 24, 2017 Member

Still Time.now.to_i

test/plugin/test_in_tail.rb
@@ -149,6 +149,29 @@ def test_emit(data)
assert_equal(1, d.emit_count)
end
+ data(flat: config_element("", "", { "format" => "/^(?<message>test.*)/", "emit_unmatched_lines" => true }),
@repeatedly
repeatedly Jan 24, 2017 Member

Seems same config_element. No need data?

jitran added some commits Jan 20, 2017
@jitran jitran Renamed enable_catch_all param as emit_unmatched_lines for clarity; a…
…nd write tests for the new behaviour
eba4c61
@jitran jitran Deprecated Time.now.to_i in favor of Fluent::EventTime.now, and updat…
…ed unit tests
d874db1
@repeatedly repeatedly merged commit 19eb856 into fluent:master Jan 25, 2017

1 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@repeatedly
Member

Thanks!

@jitran
Contributor
jitran commented Jan 26, 2017

Thanks for accepting the PR and merging it into the v.0.12 branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment