Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_stdout: new STDOUT Output Plugin. #5

Merged
merged 1 commit into from Jun 2, 2015

Conversation

pandax381
Copy link
Contributor

I implemented a new STDOUT Output Plugin.
I hope that it will assist debugging.

$ ./fluent-bit -i cpu -o stdout
Fluent-Bit v0.1.0
Copyright (C) Treasure Data

[2015/06/02 18:32:54] [ info] starting engine
[0] {"time"=>1433237575, "cpu"=>0.000000}
[1] {"time"=>1433237576, "cpu"=>0.000000}
[2] {"time"=>1433237577, "cpu"=>0.000000}
[3] {"time"=>1433237578, "cpu"=>0.000000}
[2015/06/02 18:32:59] [ info] Flush buf 96 bytes
[0] {"time"=>1433237579, "cpu"=>0.000000}
[1] {"time"=>1433237580, "cpu"=>0.000000}
[2] {"time"=>1433237581, "cpu"=>0.000000}
[3] {"time"=>1433237582, "cpu"=>0.000000}
[4] {"time"=>1433237583, "cpu"=>0.000000}
[2015/06/02 18:33:04] [ info] Flush buf 120 bytes
...

Note: An error has occurred that "libmsgpack.a is not a PIC format" in the process of generating a shared library of fluent-bit. So, I added the code to add the -fPIC option to CMakeLists.txt of msgpack.

Scanning dependencies of target fluent-bit-shared
[ 81%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_pack.c.o
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_input.c.o
[ 87%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_output.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_config.c.o
[ 93%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_network.c.o
[ 96%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_utils.c.o
[100%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_engine.c.o
Linking C shared library ../library/libfluent-bit.so
/usr/bin/ld: ../library/libmsgpack.a(unpack.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
../library/libmsgpack.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [library/libfluent-bit.so] Error 1
make[1]: *** [src/CMakeFiles/fluent-bit-shared.dir/all] Error 2
make: *** [all] Error 2

Signed-off-by: Masaya YAMAMOTO pandax381@gmail.com

Signed-off-by: Masaya YAMAMOTO <pandax381@gmail.com>
edsiper added a commit that referenced this pull request Jun 2, 2015
out_stdout: new STDOUT Output Plugin.
@edsiper edsiper merged commit 634584e into fluent:master Jun 2, 2015
@edsiper
Copy link
Member

edsiper commented Jun 2, 2015

thanks!

@edsiper
Copy link
Member

edsiper commented Jun 3, 2015

would you please check the following issue ?

#6

@pandax381
Copy link
Contributor Author

Sorry, It's my mistake.
The cause of this problem was in in_mem plugin.
I will pull request to fix the problem urgently.

@pandax381 pandax381 deleted the out_stdout branch June 3, 2015 05:14
@prashantvicky prashantvicky mentioned this pull request Aug 28, 2018
fujimotos pushed a commit to fujimotos/fluent-bit that referenced this pull request Jul 22, 2019
api: add new API flb_service_set()
fujimotos pushed a commit to fujimotos/fluent-bit that referenced this pull request Jan 15, 2020
When Fluent Bit encounters with a partial parser definition, it
crashes badly with a segmentation fault.

    $ ./bin/fluent-bit -R parser.conf -c tail.conf
    ...
    [2020/01/15 16:11:21] [error] [parser] no parser 'format' found for 'simple' in file 'conf/timestamp.parser'
    [engine] caught signal (SIGSEGV)
    #0  0x558bc4a0a226      in  flb_parser_decoder_list_destroy() at src/flb_parser_decoder.c:700
    fluent#1  0x558bc4a05d75      in  flb_parser_conf_file() at src/flb_parser.c:566
    fluent#2  0x558bc49f4bdd      in  flb_config_set_property() at src/flb_config.c:406
    fluent#3  0x558bc49e24ae      in  flb_service_conf() at src/fluent-bit.c:446
    fluent#4  0x558bc49e2f90      in  main() at src/fluent-bit.c:807
    fluent#5  0x7fa1cb7f109a      in  ???() at ???:0
    fluent#6  0x558bc49e13a9      in  ???() at ???:0
    fluent#7  0xffffffffffffffff  in  ???() at ???:0
    Aborted

This is just because `decoders` is not being initialized properly,
and that confuses Fluent Bit to deallocate a random memmory block
on the cleanup path. Fix it.

Signed-off-by: Fujimoto Seiji <fujimoto@clear-code.com>
edsiper pushed a commit that referenced this pull request Jan 16, 2020
When Fluent Bit encounters with a partial parser definition, it
crashes badly with a segmentation fault.

    $ ./bin/fluent-bit -R parser.conf -c tail.conf
    ...
    [2020/01/15 16:11:21] [error] [parser] no parser 'format' found for 'simple' in file 'conf/timestamp.parser'
    [engine] caught signal (SIGSEGV)
    #0  0x558bc4a0a226      in  flb_parser_decoder_list_destroy() at src/flb_parser_decoder.c:700
    #1  0x558bc4a05d75      in  flb_parser_conf_file() at src/flb_parser.c:566
    #2  0x558bc49f4bdd      in  flb_config_set_property() at src/flb_config.c:406
    #3  0x558bc49e24ae      in  flb_service_conf() at src/fluent-bit.c:446
    #4  0x558bc49e2f90      in  main() at src/fluent-bit.c:807
    #5  0x7fa1cb7f109a      in  ???() at ???:0
    #6  0x558bc49e13a9      in  ???() at ???:0
    #7  0xffffffffffffffff  in  ???() at ???:0
    Aborted

This is just because `decoders` is not being initialized properly,
and that confuses Fluent Bit to deallocate a random memmory block
on the cleanup path. Fix it.

Signed-off-by: Fujimoto Seiji <fujimoto@clear-code.com>
edsiper pushed a commit that referenced this pull request Jan 17, 2020
When Fluent Bit encounters with a partial parser definition, it
crashes badly with a segmentation fault.

    $ ./bin/fluent-bit -R parser.conf -c tail.conf
    ...
    [2020/01/15 16:11:21] [error] [parser] no parser 'format' found for 'simple' in file 'conf/timestamp.parser'
    [engine] caught signal (SIGSEGV)
    #0  0x558bc4a0a226      in  flb_parser_decoder_list_destroy() at src/flb_parser_decoder.c:700
    #1  0x558bc4a05d75      in  flb_parser_conf_file() at src/flb_parser.c:566
    #2  0x558bc49f4bdd      in  flb_config_set_property() at src/flb_config.c:406
    #3  0x558bc49e24ae      in  flb_service_conf() at src/fluent-bit.c:446
    #4  0x558bc49e2f90      in  main() at src/fluent-bit.c:807
    #5  0x7fa1cb7f109a      in  ???() at ???:0
    #6  0x558bc49e13a9      in  ???() at ???:0
    #7  0xffffffffffffffff  in  ???() at ???:0
    Aborted

This is just because `decoders` is not being initialized properly,
and that confuses Fluent Bit to deallocate a random memmory block
on the cleanup path. Fix it.

Signed-off-by: Fujimoto Seiji <fujimoto@clear-code.com>
edsiper pushed a commit that referenced this pull request Jan 23, 2020
When Fluent Bit encounters with a partial parser definition, it
crashes badly with a segmentation fault.

    $ ./bin/fluent-bit -R parser.conf -c tail.conf
    ...
    [2020/01/15 16:11:21] [error] [parser] no parser 'format' found for 'simple' in file 'conf/timestamp.parser'
    [engine] caught signal (SIGSEGV)
    #0  0x558bc4a0a226      in  flb_parser_decoder_list_destroy() at src/flb_parser_decoder.c:700
    #1  0x558bc4a05d75      in  flb_parser_conf_file() at src/flb_parser.c:566
    #2  0x558bc49f4bdd      in  flb_config_set_property() at src/flb_config.c:406
    #3  0x558bc49e24ae      in  flb_service_conf() at src/fluent-bit.c:446
    #4  0x558bc49e2f90      in  main() at src/fluent-bit.c:807
    #5  0x7fa1cb7f109a      in  ???() at ???:0
    #6  0x558bc49e13a9      in  ???() at ???:0
    #7  0xffffffffffffffff  in  ???() at ???:0
    Aborted

This is just because `decoders` is not being initialized properly,
and that confuses Fluent Bit to deallocate a random memmory block
on the cleanup path. Fix it.

Signed-off-by: Fujimoto Seiji <fujimoto@clear-code.com>
allamand pushed a commit to allamand/fluent-bit that referenced this pull request Oct 26, 2020
Clean CloudWatch log group once the validation is done
edsiper added a commit that referenced this pull request Feb 25, 2021
When libco starts, it might enter in a race condition if multiple
threads are trying to initialize the 'co_swap' function, this check
is done on every coroutine creation:

  ==346246== Possible data race during read of size 8 at 0x5CA890 by thread #5
  ==346246== Locks held: none
  ==346246==    at 0x48EFAE: co_create (amd64.c:132)
  ==346246==    by 0x173035: flb_output_coro_create (flb_output.h:511)
  ==346246==    by 0x173035: output_thread (flb_output_thread.c:281)
  ==346246==    by 0x1889BE: step_callback (flb_worker.c:44)
  ==346246==    by 0x4843B1A: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_helgrind-amd64-linux.so)
  ==346246==    by 0x487E58F: start_thread (pthread_create.c:463)
  ==346246==    by 0x4F47222: clone (clone.S:95)
  ==346246==
  ==346246== This conflicts with a previous write of size 8 by thread #4
  ==346246== Locks held: none
  ==346246==    at 0x48EFCB: co_create (amd64.c:134)
  ==346246==    by 0x173035: flb_output_coro_create (flb_output.h:511)
  ==346246==    by 0x173035: output_thread (flb_output_thread.c:281)
  ==346246==    by 0x1889BE: step_callback (flb_worker.c:44)
  ==346246==    by 0x4843B1A: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_helgrind-amd64-linux.so)
  ==346246==    by 0x487E58F: start_thread (pthread_create.c:463)
  ==346246==    by 0x4F47222: clone (clone.S:95)
  ==346246==  Address 0x5ca890 is 0 bytes inside data symbol "co_swap"

This patch introduce a new API for flb_coro interface that aims to
be called inside every worker thread. The access to this first
initialization is protected.

No more race conditions on that piece of code has been seen with valgrind
after the usage of this new function (next patches).

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
edsiper added a commit that referenced this pull request Mar 2, 2021
When libco starts, it might enter in a race condition if multiple
threads are trying to initialize the 'co_swap' function, this check
is done on every coroutine creation:

  ==346246== Possible data race during read of size 8 at 0x5CA890 by thread #5
  ==346246== Locks held: none
  ==346246==    at 0x48EFAE: co_create (amd64.c:132)
  ==346246==    by 0x173035: flb_output_coro_create (flb_output.h:511)
  ==346246==    by 0x173035: output_thread (flb_output_thread.c:281)
  ==346246==    by 0x1889BE: step_callback (flb_worker.c:44)
  ==346246==    by 0x4843B1A: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_helgrind-amd64-linux.so)
  ==346246==    by 0x487E58F: start_thread (pthread_create.c:463)
  ==346246==    by 0x4F47222: clone (clone.S:95)
  ==346246==
  ==346246== This conflicts with a previous write of size 8 by thread #4
  ==346246== Locks held: none
  ==346246==    at 0x48EFCB: co_create (amd64.c:134)
  ==346246==    by 0x173035: flb_output_coro_create (flb_output.h:511)
  ==346246==    by 0x173035: output_thread (flb_output_thread.c:281)
  ==346246==    by 0x1889BE: step_callback (flb_worker.c:44)
  ==346246==    by 0x4843B1A: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_helgrind-amd64-linux.so)
  ==346246==    by 0x487E58F: start_thread (pthread_create.c:463)
  ==346246==    by 0x4F47222: clone (clone.S:95)
  ==346246==  Address 0x5ca890 is 0 bytes inside data symbol "co_swap"

This patch introduce a new API for flb_coro interface that aims to
be called inside every worker thread. The access to this first
initialization is protected.

No more race conditions on that piece of code has been seen with valgrind
after the usage of this new function (next patches).

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
cosmo0920 added a commit that referenced this pull request Oct 5, 2022
…es strictly

Without this check, the following weird error is occurred
intermittently:

```log
[0] dummy.0: [1664938706.407551000, {"message"=>"dummy"}]
[2022/10/05 11:58:27] [ info] [test] flush record
flb-rt-core_chunk_trace(32205,0x16fe87000) malloc: *** error for object 0x600002600074: pointer being realloc'd was not allocated
flb-rt-core_chunk_trace(32205,0x16fe87000) malloc: *** set a breakpoint in malloc_error_break to debug
```

The main reason is, num_records index is broken in some cases:

```
flb-rt-core_chunk_trace(32205,0x16fe87000) malloc: *** error for object 0x600002600074: pointer being realloc'd was not allocated
flb-rt-core_chunk_trace(32205,0x16fe87000) malloc: *** set a breakpoint in malloc_error_break to debug
[2022/10/05 11:58:27] [ info] [input] pausing dummy.0
Process 32205 stopped
* thread #2, name = 'flb-pipeline', stop reason = breakpoint 1.1
    frame #0: 0x00000001b34a3120 libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
->  0x1b34a3120 <+0>:  pacibsp
    0x1b34a3124 <+4>:  stp    x29, x30, [sp, #-0x10]!
    0x1b34a3128 <+8>:  mov    x29, sp
    0x1b34a312c <+12>: nop
Target 0: (flb-rt-core_chunk_trace) stopped.
(lldb) bt
* thread #2, name = 'flb-pipeline', stop reason = breakpoint 1.1
  * frame #0: 0x00000001b34a3120 libsystem_malloc.dylib`malloc_error_break
    frame #1: 0x00000001b3494844 libsystem_malloc.dylib`malloc_vreport + 428
    frame #2: 0x00000001b3497f34 libsystem_malloc.dylib`malloc_report + 64
    frame #3: 0x00000001b3488210 libsystem_malloc.dylib`realloc + 328
    frame #4: 0x0000000100006154 flb-rt-core_chunk_trace`flb_realloc(ptr=0x0000600002600074, size=18446744064764412176) at flb_mem.h:94:12
    frame #5: 0x0000000100005fc8 flb-rt-core_chunk_trace`callback_add_record(data=0x0000600003014000, size=135, cb_data=0x0000600000004010) at core_chunk_trace.c:51:28
    frame #6: 0x00000001001268b0 flb-rt-core_chunk_trace`out_lib_flush(event_chunk=0x0000600000c14000, out_flush=0x0000600001714000, i_ins=0x0000000100b09ab0, out_context=0x0000600000204a80, config=0x000000010181d200) at out_lib.c:197:9
    frame #7: 0x0000000100029d70 flb-rt-core_chunk_trace`output_pre_cb_flush at flb_output.h:517:5
    frame #8: 0x000000010044fa64 flb-rt-core_chunk_trace`co_switch(handle=0x000000010044fa64) at aarch64.c:133:4
(lldb) frane select 5
error: 'frane' is not a valid command.
(lldb) frame select 5
frame #5: 0x0000000100005fc8 flb-rt-core_chunk_trace`callback_add_record(data=0x0000600003014000, size=135, cb_data=0x0000600000004010) at core_chunk_trace.c:51:28
   48  	                           flb_calloc(1, sizeof(struct callback_record));
   49  	        } else {
   50  	            ctx->records = (struct callback_record *)
-> 51  	                           flb_realloc(ctx->records,
   52  	                                       (ctx->num_records+1)*sizeof(struct callback_record));
   53  	        }
   54  	        if (ctx->records ==  NULL) {
(lldb) po ctx->records
0x0000600002600074

(lldb) po ctx->records
0x0000600002600074

(lldb) po ctx->num_records
-559071216
```

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants