filter_join_partial: new filter for appending partial docker logs (#821)

[jlpettersson]

A new filter for appending log events. Solves issue #821
Currently only implemented for Docker json-log format, but can support CRI-O logs with little work.
This plugin is supposed to be used early in the filter-chain, before kubernetes-filter, e.g. right after tail-input.

Example configuration:

[INPUT]
    Name tail
    Path /var/logs/*.log
    Parser docker

[FILTER]
    Name    join_partial
    Match   *
    Log_Key log

[OUTPUT]
    Name  stdout
    Match *`

Example log file, for test input:

{"log":"Hello "}
{"log":"Middle "}
{"log":"World\n"}
{"log":"Normal line\n"}
{"log":"Another "}
{"log":"partial line\n"}
{"log":"Final line\n"}

Example output:

[0] tail.0: [1539781859.436474856, {"log"=>"Hello Middle World\n"}]
[1] tail.0: [1539781859.436479251, {"log"=>"Normal line\n"}]
[2] tail.0: [1539781859.436480171, {"log"=>"Another partial line\n"}]
[3] tail.0: [1539781859.436481998, {"log"=>"Final line\n"}]

[edsiper]

this looks good, despite this is similar to the new docker_mode in in_tail, I think is a good addition.

since Fluent Bit is a subproject of Fluentd which is part of CNCF and we are in the process of "donate the project", would you please assign the copyright to Treasure Data ?

/*  Fluent Bit
 *  ==========
 *  Copyright (C) 2015-2018 Treasure Data Inc.
 *
 *  Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 */

[edsiper]

Please fix the following two things:

1. single-line comments

Instead of comment a single line with //, use /* my comment */

2. memory leak at exit:

the following is reported by valgrind:

==28058== 
==28058== HEAP SUMMARY:
==28058==     in use at exit: 4 bytes in 1 blocks
==28058==   total heap usage: 1,953 allocs, 1,952 frees, 821,336 bytes allocated
==28058== 
==28058== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==28058==    at 0x483774F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28058==    by 0x1CCB2B: flb_malloc (flb_mem.h:57)
==28058==    by 0x1CCBE6: flb_strdup (flb_str.h:36)
==28058==    by 0x1CCC9F: join_partial_conf_create (join_partial.c:44)
==28058==    by 0x1CCDE6: cb_join_partial_init (join_partial.c:74)
==28058==    by 0x151739: flb_filter_initialize_all (flb_filter.c:264)
==28058==    by 0x158F9F: flb_engine_start (flb_engine.c:432)
==28058==    by 0x143199: main (fluent-bit.c:826)

[edsiper]

ping

[jlpettersson]

@edsiper Is there really use for this filter now when the same functionallity is available in the tail-input?

[StevenACoffman]

@jlpettersson My understanding is that appending partial CRI-O logs is not supported with the tail-input. I would like to see you pursue this specifically for that reason, and @edsiper already commented that he thought it was a good addition.

[edsiper]

Yes. Even if you consider that logs can come from a different source than in_tail we need a mechanism to correlate them.

[edsiper]

@jlpettersson we are ready to go once the minor memory leak is fixed.

[oanogin]

@edsiper Is there really use for this filter now when the same functionality is available in the tail-input?

hi, can you show any example of the same functionality with in_tail please?

[edsiper]

@oanogin it's coming in the next major release (v1.0), you can check the docs here:

https://github.com/fluent/fluent-bit-docs/blob/master/input/tail.md#docker_mode

[edsiper]

ping @jlpettersson

[jlpettersson]

@edsiper We have no use for this in our project, since this functionality now is in the tail input. So I will not invest more time on this one. For me, it can be closed.

[edsiper]

ack. thanks.

[ewbankkit]

@jlpettersson We're using containerd/CRI-O and have many mutli-line log events (Java stack dumps 😄). My reading of this issue is that it has been closed because the tail input filter can handle multi-line docker log outputs and there was no use case for the CRI parsing. Is that correct?

[g3kr]

@jlpettersson we are ready to go once the minor memory leak is fixed.

@edsiper hi, do we have this merged? the filter plugin join_partial is this supported for forward input plugin as well?

[Skyfallz]

@edsiper Hi, is there a chance this ever gets implemented ? Such a filter would be really useful, ie for re-assembling chunked >16M docker logs lines forwarded through Firelens (cannot use Docker_Mode, as I can't use in_tail module). The usecase is not the same, it would be really complementary, maybe you should reconsider openening this issue ?