Capture unmatched lines #1421

merged 4 commits into from Jan 25, 2017


None yet

2 participants

jitran commented Jan 17, 2017

Essentially all unmatched lines will be placed inside the parse_fail attribute, i.e.

   "parse_fail":"unmatched of the line",

This will enable our analytics tools to pick up any pattern matching regexp issues.

jitran added some commits Jan 16, 2017
@jitran jitran Add a new option for in_tail called 'enable_catch_all'. When set to t…
…rue, all singular and multiline log lines that don't match the parser regex will be stored in an attribute called parse_fail, and eventually saved in the output destination.
@jitran jitran Use as its backwards compatible with the earlier versio…
…ns of fluentd

Additional comments

  • Need test for this parameter
  • How about adding unmatched_tag for emitting unmatched lines to other tag, not including same EventStream.
@@ -65,6 +65,8 @@ def initialize
config_param :read_lines_limit, :integer, default: 1000
desc 'The interval of flushing the buffer for multiline format'
config_param :multiline_flush_interval, :time, default: nil
+ desc 'Enable the option to capture unmatched lines.'
+ config_param :enable_catch_all, :bool, default: false
repeatedly Jan 19, 2017 Member

emit_unmatched_lines or emit_parse_failed_lines is more better.
enable_catch_all is unclear name for me.

+ if @enable_catch_all
+ record = {'parse_fail' => line}
+ record[@path_key] ||= tail_watcher.path unless @path_key.nil?
+ es.add(, record)
repeatedly Jan 19, 2017 Member

Use Don't use Time directly for event time.

jitran Jan 19, 2017 Contributor

We're using the Redhat version of td-agent which comes with Fluentd v0.12. Do you know when the 0.14 branch will be stable? If it's still quite long away, should I merge my branch into the 0.12 branch instead?

repeatedly Jan 19, 2017 Member

If you want this feature in v0.12, you also need to send a patch to v0.12 branch.

@jitran jitran referenced this pull request Jan 20, 2017


jitran commented Jan 23, 2017

Hi @repeatedly, I have implemented the tests for the new parameter, but the existing unit tests seem to fail randomly between the different ruby releases, and I've noticed when the CI runs again, those tests pass again.

+ if @emit_unmatched_lines
+ record = {'unmatched_line' => line}
+ record[@path_key] ||= tail_watcher.path unless @path_key.nil?
+ es.add(, record)
repeatedly Jan 24, 2017 Member


@@ -149,6 +149,29 @@ def test_emit(data)
assert_equal(1, d.emit_count)
+ data(flat: config_element("", "", { "format" => "/^(?<message>test.*)/", "emit_unmatched_lines" => true }),
repeatedly Jan 24, 2017 Member

Seems same config_element. No need data?

jitran added some commits Jan 20, 2017
@jitran jitran Renamed enable_catch_all param as emit_unmatched_lines for clarity; a…
…nd write tests for the new behaviour
@jitran jitran Deprecated in favor of, and updat…
…ed unit tests
@repeatedly repeatedly merged commit 19eb856 into fluent:master Jan 25, 2017

1 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
continuous-integration/travis-ci/pr The Travis CI build passed


jitran commented Jan 26, 2017

Thanks for accepting the PR and merging it into the v.0.12 branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment