Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

in_systemd: backport 'lowercase' option #4992

Merged
merged 2 commits into from
Mar 11, 2022
Merged

in_systemd: backport 'lowercase' option #4992

merged 2 commits into from
Mar 11, 2022

Conversation

chitoku-k
Copy link
Contributor

@chitoku-k chitoku-k commented Mar 5, 2022 •
edited

This PR backports #4908 to 1.8 branch.

Fixes #1543

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@chitoku-k
in_systemd: fix coding style
7c8b862 
Signed-off-by: Chitoku <odango@chitoku.jp>
@chitoku-k
in_systemd: add 'lowercase' option
5b3a448 
Signed-off-by: Chitoku <odango@chitoku.jp>
@chitoku-k chitoku-k mentioned this pull request Mar 5, 2022
Merged
@chitoku-k
Copy link
Contributor Author

Config

[INPUT]
    Name                systemd
    Systemd_Filter      _SYSTEMD_UNIT=systemd-logind.service
    Read_From_Tail      On
    Lowercase           On

[OUTPUT]
    Name                stdout

Log

$ bin/fluent-bit -c config
Fluent Bit v1.8.13
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/05 14:58:06] [ info] [engine] started (pid=76132)
[2022/03/05 14:58:06] [ info] [storage] version=1.1.6, initializing...
[2022/03/05 14:58:06] [ info] [storage] in-memory
[2022/03/05 14:58:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/05 14:58:06] [ info] [cmetrics] version=0.2.2
[2022/03/05 14:58:06] [ info] [sp] stream processor started
[2022/03/05 14:58:06] [ info] [output:stdout:stdout.0] worker #0 started
[0] systemd.0: [1646459892.633159000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "code_line"=>"712", "code_func"=>"session_start", "syslog_identifier"=>"systemd-logind", "message_id"=>"8d45620c1a4348dbb17410da57c60c66", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "session_id"=>"6", "leader"=>"76166", "message"=>"New session 6 of user chitoku.", "_source_realtime_timestamp"=>"1646459892633130"}]
[0] systemd.0: [1646459898.379473000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "syslog_identifier"=>"systemd-logind", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "code_line"=>"783", "code_func"=>"session_stop_scope", "session_id"=>"6", "leader"=>"76166", "message"=>"Session 6 logged out. Waiting for processes to exit.", "_source_realtime_timestamp"=>"1646459898374946"}]
[1] systemd.0: [1646459898.379552000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "syslog_identifier"=>"systemd-logind", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "code_line"=>"840", "code_func"=>"session_finalize", "message_id"=>"3354939424b4456d9802ca8333ed424a", "session_id"=>"6", "leader"=>"76166", "message"=>"Removed session 6.", "_source_realtime_timestamp"=>"1646459898375464"}]
^C[2022/03/05 14:58:24] [engine] caught signal (SIGINT)
[2022/03/05 14:58:24] [ info] [input] pausing systemd.0
[2022/03/05 14:58:24] [ warn] [engine] service will shutdown in max 5 seconds
[2022/03/05 14:58:25] [ info] [engine] service has stopped (0 pending tasks)
[2022/03/05 14:58:25] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/03/05 14:58:25] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -c config
==76382== Memcheck, a memory error detector
==76382== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==76382== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==76382== Command: bin/fluent-bit -c config
==76382==
Fluent Bit v1.8.13
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/05 14:58:30] [ info] [engine] started (pid=76382)
[2022/03/05 14:58:30] [ info] [storage] version=1.1.6, initializing...
[2022/03/05 14:58:30] [ info] [storage] in-memory
[2022/03/05 14:58:30] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/05 14:58:30] [ info] [cmetrics] version=0.2.2
[2022/03/05 14:58:30] [ info] [sp] stream processor started
[2022/03/05 14:58:31] [ info] [output:stdout:stdout.0] worker #0 started
[0] systemd.0: [1646459912.965408000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "code_line"=>"712", "code_func"=>"session_start", "syslog_identifier"=>"systemd-logind", "message_id"=>"8d45620c1a4348dbb17410da57c60c66", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "session_id"=>"7", "leader"=>"76404", "message"=>"New session 7 of user chitoku.", "_source_realtime_timestamp"=>"1646459912965384"}]
[0] systemd.0: [1646459918.724016000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "syslog_identifier"=>"systemd-logind", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "code_line"=>"783", "code_func"=>"session_stop_scope", "session_id"=>"7", "leader"=>"76404", "message"=>"Session 7 logged out. Waiting for processes to exit.", "_source_realtime_timestamp"=>"1646459918723989"}]
[1] systemd.0: [1646459918.724560000, {"_uid"=>"0", "_gid"=>"0", "_systemd_slice"=>"system.slice", "_boot_id"=>"f451927c5a6841d79048dd5faee180e2", "_machine_id"=>"09cb1f91a4af4686bc1b57cad2710c32", "_hostname"=>"lychee", "priority"=>"6", "_transport"=>"journal", "syslog_facility"=>"4", "tid"=>"505", "code_file"=>"src/login/logind-session.c", "syslog_identifier"=>"systemd-logind", "user_id"=>"chitoku", "_pid"=>"505", "_comm"=>"systemd-logind", "_exe"=>"/usr/lib/systemd/systemd-logind", "_cmdline"=>"/usr/lib/systemd/systemd-logind", "_cap_effective"=>"24420020f", "_systemd_cgroup"=>"/system.slice/systemd-logind.service", "_systemd_unit"=>"systemd-logind.service", "_systemd_invocation_id"=>"0d9a542887a24db38ebf95a9e0a33728", "code_line"=>"840", "code_func"=>"session_finalize", "message_id"=>"3354939424b4456d9802ca8333ed424a", "session_id"=>"7", "leader"=>"76404", "message"=>"Removed session 7.", "_source_realtime_timestamp"=>"1646459918724544"}]
^C[2022/03/05 14:58:41] [engine] caught signal (SIGINT)
[2022/03/05 14:58:41] [ info] [input] pausing systemd.0
[2022/03/05 14:58:41] [ warn] [engine] service will shutdown in max 5 seconds
[2022/03/05 14:58:42] [ info] [engine] service has stopped (0 pending tasks)
==76382== Thread 4 flb-out-stdout.0:
==76382== Invalid read of size 8
==76382==    at 0x4FDD03: mk_event_timeout_destroy (mk_event.c:165)
==76382==    by 0x19520E: flb_sched_timer_destroy (flb_scheduler.c:649)
==76382==    by 0x195037: flb_sched_destroy (flb_scheduler.c:596)
==76382==    by 0x181C6B: output_thread (flb_output_thread.c:361)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Address 0x52d2940 is 16 bytes inside a block of size 24 free'd
==76382==    at 0x484827F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD27A: mk_mem_free (mk_memory.h:102)
==76382==    by 0x4FDB77: mk_event_loop_destroy (mk_event.c:82)
==76382==    by 0x181C5C: output_thread (flb_output_thread.c:359)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Block was alloc'd at
==76382==    at 0x484AA83: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD241: mk_mem_alloc_z (mk_memory.h:70)
==76382==    by 0x4FDAB7: mk_event_loop_create (mk_event.c:58)
==76382==    by 0x181F8B: flb_output_thread_pool_create (flb_output_thread.c:443)
==76382==    by 0x17F47D: flb_output_enable_multi_threading (flb_output.c:784)
==76382==    by 0x1802A5: flb_output_init_all (flb_output.c:1070)
==76382==    by 0x19060C: flb_engine_start (flb_engine.c:602)
==76382==    by 0x16E8EA: flb_lib_worker (flb_lib.c:627)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==
==76382== Invalid read of size 4
==76382==    at 0x4FD605: _mk_event_del (mk_event_epoll.c:153)
==76382==    by 0x4FD855: _mk_event_timeout_destroy (mk_event_epoll.c:325)
==76382==    by 0x4FDD1D: mk_event_timeout_destroy (mk_event.c:166)
==76382==    by 0x19520E: flb_sched_timer_destroy (flb_scheduler.c:649)
==76382==    by 0x195037: flb_sched_destroy (flb_scheduler.c:596)
==76382==    by 0x181C6B: output_thread (flb_output_thread.c:361)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Address 0x52d25a0 is 0 bytes inside a block of size 16 free'd
==76382==    at 0x484827F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD27A: mk_mem_free (mk_memory.h:102)
==76382==    by 0x4FD4BB: _mk_event_loop_destroy (mk_event_epoll.c:94)
==76382==    by 0x4FDB5B: mk_event_loop_destroy (mk_event.c:80)
==76382==    by 0x181C5C: output_thread (flb_output_thread.c:359)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Block was alloc'd at
==76382==    at 0x484AA83: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD241: mk_mem_alloc_z (mk_memory.h:70)
==76382==    by 0x4FD3C4: _mk_event_loop_create (mk_event_epoll.c:54)
==76382==    by 0x4FDA98: mk_event_loop_create (mk_event.c:53)
==76382==    by 0x181F8B: flb_output_thread_pool_create (flb_output_thread.c:443)
==76382==    by 0x17F47D: flb_output_enable_multi_threading (flb_output.c:784)
==76382==    by 0x1802A5: flb_output_init_all (flb_output.c:1070)
==76382==    by 0x19060C: flb_engine_start (flb_engine.c:602)
==76382==    by 0x16E8EA: flb_lib_worker (flb_lib.c:627)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==
==76382== Invalid read of size 8
==76382==    at 0x4FDC35: mk_event_del (mk_event.c:126)
==76382==    by 0x4FDCEC: mk_event_timeout_disable (mk_event.c:157)
==76382==    by 0x194D76: flb_sched_timer_cb_disable (flb_scheduler.c:494)
==76382==    by 0x195225: flb_sched_timer_destroy (flb_scheduler.c:652)
==76382==    by 0x195037: flb_sched_destroy (flb_scheduler.c:596)
==76382==    by 0x181C6B: output_thread (flb_output_thread.c:361)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Address 0x52d2940 is 16 bytes inside a block of size 24 free'd
==76382==    at 0x484827F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD27A: mk_mem_free (mk_memory.h:102)
==76382==    by 0x4FDB77: mk_event_loop_destroy (mk_event.c:82)
==76382==    by 0x181C5C: output_thread (flb_output_thread.c:359)
==76382==    by 0x19A19B: step_callback (flb_worker.c:44)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==  Block was alloc'd at
==76382==    at 0x484AA83: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==76382==    by 0x4FD241: mk_mem_alloc_z (mk_memory.h:70)
==76382==    by 0x4FDAB7: mk_event_loop_create (mk_event.c:58)
==76382==    by 0x181F8B: flb_output_thread_pool_create (flb_output_thread.c:443)
==76382==    by 0x17F47D: flb_output_enable_multi_threading (flb_output.c:784)
==76382==    by 0x1802A5: flb_output_init_all (flb_output.c:1070)
==76382==    by 0x19060C: flb_engine_start (flb_engine.c:602)
==76382==    by 0x16E8EA: flb_lib_worker (flb_lib.c:627)
==76382==    by 0x4E4B5C1: start_thread (in /usr/lib/libc.so.6)
==76382==    by 0x4ED0583: clone (in /usr/lib/libc.so.6)
==76382==
[2022/03/05 14:58:42] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/03/05 14:58:42] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==76382==
==76382== HEAP SUMMARY:
==76382==     in use at exit: 0 bytes in 0 blocks
==76382==   total heap usage: 1,719 allocs, 1,719 frees, 1,505,799 bytes allocated
==76382==
==76382== All heap blocks were freed -- no leaks are possible
==76382==
==76382== For lists of detected and suppressed errors, rerun with: -s
==76382== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 0 from 0)

@edsiper edsiper merged commit cdb89f0 into fluent:1.8 Mar 11, 2022
@chitoku-k chitoku-k deleted the 1.8-backport-in-systemd-lowercase branch March 11, 2022 05:29
@chitoku-k
Copy link
Contributor Author

@edsiper
Thank you for reviewing this! Now that both master and 1.8 have gotten #1543 solved, should it be closed accordingly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

Assignees
No one assigned
Labels
ci/integration-docker-ok ci/integration-test-gcp-ok ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@chitoku-k @edsiper @lecaros @niedbalski