AFSQL refactors & bugfixes from PE #8
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 13, 2013 16:10
A lot of the code that was previously in afsql_dd_insert_db() have been extracted to smaller functions, and afsql_dd_insert_db() was rebuilt on top of these. At the same time, memory leaks were plugged, and in case of a transaction error, backlog rewinding has been fixed too, to not loose messages since the last BEGIN command. Signed-off-by: Juhasz Viktor <jviktor@balabit.hu> Signed-off-by: Laszlo Budai <lbudai@balabit.hu>
When log_dest_driver_acquire_queue() fails to acquire the queue, afsql_dd_init() should fail too. Signed-off-by: Juhasz Viktor <jviktor@balabit.hu> Signed-off-by: Budai Laszlo <lbudai@balabit.hu>
Sequence numbering was restarted upon reload before, but with this patch, the number gets stored in the persist config, and read back from there upon reload. Signed-off-by: Juhasz Viktor <jviktor@balabit.hu> Signed-off-by: Budai Laszlo <lbudai@balabit.hu>
bazsi
added a commit
that referenced
this pull request
Feb 21, 2016
syslog-ng was crashing in my development environment at startup if the systemd-journal() driver was used, in a backtrace like this: Program received signal SIGSEGV, Segmentation fault. g_module_symbol (module=0x6771b0, symbol_name=0x7ffff4942a65 "sd_journal_open", symbol=0x7ffff7f838f0 <sd_journal_open>) at /build/glib2.0-ajuDY6/glib2.0-2.46.1/./gmodule/gmodule.c:818 818 /build/glib2.0-ajuDY6/glib2.0-2.46.1/./gmodule/gmodule.c: No such file or directory. (gdb) bt #0 g_module_symbol (module=0x6771b0, symbol_name=0x7ffff4942a65 "sd_journal_open", symbol=0x7ffff7f838f0 <sd_journal_open>) at /build/glib2.0-ajuDY6/glib2.0-2.46.1/./gmodule/gmodule.c:818 #1 0x00007ffff493e6cd in load_journald_subsystem () at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/modules/systemd-journal/journald-subsystem.c:105 #2 0x00007ffff493cd0c in systemd_journal_module_init (cfg=0x63a2a0, args=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/modules/systemd-journal/systemd-journal-plugin.c:44 #3 0x00007ffff7b4a9d3 in plugin_load_module (module_name=0x6572a0 "sdjournal", cfg=0x63a2a0, args=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/plugin.c:385 #4 0x00007ffff7b4a18c in plugin_find (cfg=0x63a2a0, plugin_type=3, plugin_name=0x675c80 "systemd-journal") at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/plugin.c:155 #5 0x00007ffff7b5ee8e in main_parse (lexer=0x63f010, dummy=0x7fffffffd0c8, arg=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/cfg-grammar.y:580 #6 0x00007ffff7b2a7a5 in cfg_parser_parse (self=0x7ffff7dd4ee0 <main_parser>, lexer=0x63f010, instance=0x7fffffffd0c8, arg=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/cfg-parser.h:83 #7 0x00007ffff7b2b7f7 in cfg_run_parser (self=0x63a2a0, lexer=0x63f010, parser=0x7ffff7dd4ee0 <main_parser>, result=0x7fffffffd0c8, arg=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/cfg.c:430 #8 0x00007ffff7b2ba3a in cfg_read_config (self=0x63a2a0, fname=0x60e940 "etc/syslog-ng.conf", syntax_only=0, preprocess_into=0x0) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/cfg.c:502 #9 0x00007ffff7b45368 in main_loop_read_and_init_config () at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/lib/mainloop.c:447 #10 0x0000000000401b31 in main (argc=1, argv=0x7fffffffd238) at /home/bazsi/zwa/projects/syslog-ng-ose-3.6/source/syslog-ng/syslog-ng/main.c:261 My belief is that the root cause was that the global variables in journald-subsystem.c are named the same as the symbols in libsystemd.so, and both are symbols with global visibility. My theory was that due to the name clash, the actual pointer passed to g_module_symbol() was pointing to the actual code of the functions, instead of the pointer we wanted to store the values. I didn't completely confirm this theory, but adding "static" qualifiers to the variables resolved the issue for me. Honestly, I don't really like how journal-systemd.c is organized, but this was only a distraction for me and I was satisfied with resolving the problem for now. Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>
Kokan
added a commit
to Kokan/syslog-ng
that referenced
this pull request
Oct 2, 2020
Every time a key created a counter names is allocated. But it is not
freed. The only time it is freed when a new stats-cluster is created
from it (first time looked up) and the counter_names owenership
transfared to the stats-cluster.
With this patch the allocation is delayed to the ownership change, thus
only allocated if there is still a pointer to the memory.
Reproduction:
1. syslog-ng configuration with at least one network source
2. start syslog-ng with a leak detector
(2.1 optional do restart (in order to have the exact leak as above))
3. stop syslog-ng
=================================================================
==2095056==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x55a4f86db781 in calloc (/tmp/install/sbin/syslog-ng+0xc5781)
#1 0x7f343f03a941 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x50941)
syslog-ng#2 0x7f343acb37c7 in _make_connection_conter_stats_queryable /home/user/src/syslog-ng/build/../modules/afsocket/afsocket-source.c:1007:9
syslog-ng#3 0x7f343acb30f3 in afsocket_sd_init_method /home/user/src/syslog-ng/build/../modules/afsocket/afsocket-source.c:1060:5
syslog-ng#4 0x7f343acbc358 in afinet_sd_init /home/user/src/syslog-ng/build/../modules/afsocket/afinet-source.c:103:8
syslog-ng#5 0x7f343f248e1a in log_pipe_init /home/user/src/syslog-ng/build/../lib/logpipe.h:301:23
syslog-ng#6 0x7f343f248c88 in cfg_tree_start /home/user/src/syslog-ng/build/../lib/cfg-tree.c:1419:12
syslog-ng#7 0x7f343f23af71 in cfg_init /home/user/src/syslog-ng/build/../lib/cfg.c:368:8
syslog-ng#8 0x7f343f270b3c in main_loop_reload_config_apply /home/user/src/syslog-ng/build/../lib/mainloop.c:276:41
syslog-ng#9 0x7f343f273162 in _consume_action /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:240:3
syslog-ng#10 0x7f343f27321f in _invoke_sync_call_actions /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:250:7
syslog-ng#11 0x7f343f2736ba in main_loop_worker_sync_call /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:396:7
syslog-ng#12 0x7f343f2709d1 in main_loop_reload_config_commence /home/user/src/syslog-ng/build/../lib/mainloop.c:340:3
syslog-ng#13 0x7f343f275852 in control_connection_reload /home/user/src/syslog-ng/build/../lib/mainloop-control.c:167:3
syslog-ng#14 0x7f343f290c80 in control_connection_io_input /home/user/src/syslog-ng/build/../lib/control/control-server.c:378:3
syslog-ng#15 0x7f343efd4b2c (/usr/lib/libivykis.so.0+0x6b2c)
syslog-ng#16 0x7f343f239c50 in run_application_hook /home/user/src/syslog-ng/build/../lib/apphook.c:125:11
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x55a4f86db781 in calloc (/tmp/install/sbin/syslog-ng+0xc5781)
#1 0x7f343f03a941 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x50941)
syslog-ng#2 0x7f343acb3e27 in _stop_connection_counter_stats_queryable /home/user/src/syslog-ng/build/../modules/afsocket/afsocket-source.c:1027:9
syslog-ng#3 0x7f343acb38bd in afsocket_sd_deinit_method /home/user/src/syslog-ng/build/../modules/afsocket/afsocket-source.c:1073:3
syslog-ng#4 0x7f343f2491d4 in log_pipe_deinit /home/user/src/syslog-ng/build/../lib/logpipe.h:316:25
syslog-ng#5 0x7f343f2490e5 in cfg_tree_stop /home/user/src/syslog-ng/build/../lib/cfg-tree.c:1439:12
syslog-ng#6 0x7f343f23b2d1 in cfg_deinit /home/user/src/syslog-ng/build/../lib/cfg.c:387:10
syslog-ng#7 0x7f343f271ec2 in main_loop_exit_finish /home/user/src/syslog-ng/build/../lib/mainloop.c:427:3
syslog-ng#8 0x7f343f273162 in _consume_action /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:240:3
syslog-ng#9 0x7f343f27321f in _invoke_sync_call_actions /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:250:7
syslog-ng#9 0x7f343f27321f in _invoke_sync_call_actions /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:250:7
syslog-ng#10 0x7f343f2736ba in main_loop_worker_sync_call /home/user/src/syslog-ng/build/../lib/mainloop-worker.c:396:7
syslog-ng#11 0x7f343f2709d1 in main_loop_reload_config_commence /home/user/src/syslog-ng/build/../lib/mainloop.c:340:3
syslog-ng#12 0x7f343f275852 in control_connection_reload /home/user/src/syslog-ng/build/../lib/mainloop-control.c:167:3
syslog-ng#13 0x7f343f290c80 in control_connection_io_input /home/user/src/syslog-ng/build/../lib/control/control-server.c:378:3
syslog-ng#14 0x7f343efd4b2c (/usr/lib/libivykis.so.0+0x6b2c)
syslog-ng#15 0x7f343f239c50 in run_application_hook /home/user/src/syslog-ng/build/../lib/apphook.c:125:11
Signed-off-by: Kokan <kokaipeter@gmail.com>
szemere
pushed a commit
that referenced
this pull request
Dec 15, 2020
This commit accumulates synthetic messages in an array, and forwards them after releasing the GroupingBy parser's lock. It is NOT allowed to call log_pipe_forward_msg() when locks are held, because it would slow down the entire pipeline, and it could also result in deadlock, for example: 1. A new message is produced in syslog-ng's worker pool, which goes through the parser, so it acquires GroupingBy::lock, and starts processing the message (continues in step 3). 2. The main thread schedules grouping_by_timer_tick(), which needs to hold GroupingBy::lock, so it blocks and waits for the other thread. 3. A new synthetic message is emitted in the worker thread (step 1). stateful_parser_emit_synthetic() is called in grouping_by_expire_entry(), which is protected by GroupingBy::lock, The message goes through the pipeline with this lock held, reaches a destination, where main_loop_call(wait=TRUE) is called. Worker waits for the main loop call to be scheduled and finished, main thread waits for GroupingBy::lock, which will never be released. A deadlock situation requires at least 2 locks: - one was GroupingBy::lock; - the other was main_task_lock and its GCond in main_loop_call(). Main thread: ``` #0 syscall () #1 g_mutex_lock_slowpath () #2 _grouping_by_timer_tick () #3 grouping_by_timer_tick () #4 iv_run_timers () #5 iv_main () #6 main_loop_run () #7 main () ``` Worker thread: ``` #0 syscall () #1 g_cond_wait () #2 main_loop_call () #3 ml_batched_timer_postpone () #4 log_writer_queue () #5 log_dest_driver_queue_method () #6 log_multiplexer_queue () #7 log_multiplexer_queue () #8 log_filter_pipe_queue () #9 stateful_parser_emit_synthetic () #10 grouping_by_expire_entry () #11 timer_wheel_set_time () #12 grouping_by_set_time () #13 grouping_by_process () ... ``` Important note: The unlock trick wouldn't have worked in the timer expire callback case, as we would be iterating through a non-locked data structure (TimerWheel) that could be modified by other threads. This commit is based on a00164c, which is a similar deadlock fix for PatternDB. Signed-off-by: László Várady <laszlo.varady@protonmail.com>
HofiOne
pushed a commit
to HofiOne/syslog-ng
that referenced
this pull request
Jun 10, 2024
* Adds initial readme.md * Adds Axoflow logo * Moves logo under /docs * Use different logo for light/dark GitHub color scheme * Formatting fix * Adds note about named tags being automatically updated * Typo fix * Adds some text about contributions * Adds discord invite link * Delete unneeded logo from the repo * Revert "Delete unneeded logo from the repo" This reverts commit 9bbf67cdc5437d2612099ff996e16c2f5c8bbf7d. * Fix logo url
HofiOne
pushed a commit
to HofiOne/syslog-ng
that referenced
this pull request
Jun 10, 2024
…rch-dest collector: add opensearch destination
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a set of patches to refactor some very ugly parts of the afsql driver, and also correct some errors (memory leaks and lost messages on transaction failure, for example).