Permalink
Cannot retrieve contributors at this time
15297 lines (15263 sloc)
860 KB
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2112.0 (aka 2021.12) 2021-12-?? | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2110.0 (aka 2021.10) 2021-10-19 | |
| - 2021-10-13: config bugfix: global(security.abortonidresolutionfail=) did not work | |
| when used with rscript based configuration, it was not checked. | |
| - 2021-10-13: config bugfix: global param $privDropToUser did not work correctly | |
| The parameter was not implemented for rscript based configuration and | |
| did not properly apply to legacy configuration. In essence, it almost always | |
| did not work as expected. | |
| see also: https://github.com/rsyslog/rsyslog/issues/4642 | |
| see also: https://github.com/rsyslog/rsyslog/commit/cbcaf2c7e5b67e5465e47bc7cc67af2eae47bd31 | |
| - 2021-10-12: rscript bugfix: ruleset called async when ruleset had queue.type="direct" | |
| The call rscript statement is able to call a rule set either synchronously or | |
| asynchronously. We did this, because practice showed that both modes | |
| are needed. For various reasons we decided to make async | |
| calls if the ruleset has a queue assigned and sync if not. | |
| To know if a "queue is assigned" we just checked if queue parameters were | |
| given. It was overlookeded the case of someone explicitly specifying a | |
| "direct queue", aka "no queue". As such, queue="direct" triggered async | |
| calls. That in turn meant that when a write operation to a variable was | |
| made inside that rule set, other rulesets could or could not see the | |
| write. While if was often not seen, this was a data race where the | |
| change could also be seen by the outside. | |
| This is now fixed. No matter if queue.type="direct" is specified or | |
| left out, the call will always by synchronous. Any values written to | |
| variables will also be seen by the "outside world" in later processing | |
| stages. | |
| Note that this has some potential to BREAK EXISTING CONFIGURATIONS. | |
| We deem this acceptable because: | |
| 1. this was racy at all, so unexpected behaviour could alwas occur | |
| 2. it is actually unlikely that someone used the triggering conditions | |
| in practice. But we can not outrule this, especially when the | |
| configuration was auto-generated. | |
| Potential compatibility issues can be solved by defining a small | |
| array-memory queue on the ruleset in question instead of specifying | |
| direct type. | |
| Again, we expect that almost all users will never experience any | |
| problems. If you do, however, please let us know: we may add an | |
| option to re-enable the bug. | |
| - 2021-10-12: ksi bugfix: locking bug fixed in rsksiCtxOpenFile | |
| Thanks to Taavi Valjaots for the patch. | |
| - 2021-10-11: core bugfix: fix typo in error message | |
| Thanks to github user jkschulz for the patch. | |
| - 2021-10-11: tcpsrv bugfix: compilation without exceptions | |
| tcpsrv.c:992:1: error: label at end of compound statement | |
| finalize_it: | |
| ^~~~~~~~~~~ | |
| Quoting from pthread.h: | |
| pthread_cleanup_push and pthread_cleanup_pop are macros and must always | |
| be used in matching pairs at the same nesting level of braces. | |
| Amends commit bcdd220142ec9eb106550195ba331fd114adb0bd. | |
| Thanks to Orgad Shaneh for the patch. | |
| - 2021-10-11: mkubernetes bugfix: no connection retry to kubernetes APP | |
| When connection to the kubernates API was not possible, mmkubernetes | |
| did not retry. This does now happen via regular rsyslog retry | |
| mechanism. | |
| Thanks to github user jayme-github for the analysis and patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/4669 | |
| - 2021-10-11: openssl bugfix: Correct gnutlsPriorityString (custom ciphers) behaviour | |
| - Only apply default anon ciphers if gnutlsPriorityString is NULL and | |
| Authentication Mode is set to anon. Otherwise we do not set them | |
| as they overwrite custom Ciphers. | |
| - Added two tests for custom cipher configuration (anon/certvalid mode). | |
| - Add call for applyGnutlsPriorityString if gnutlsPriorityString changes. | |
| - Merged openssl init code from Connect into osslInitSession | |
| closes: https://github.com/rsyslog/rsyslog/issues/4686 | |
| - 2021-10-11: build issue: handle undefined MAXPATHLEN, PATH_MAX | |
| While we handled missing PATH_MAX, we did not handle missing MAXPATHLEN. | |
| This happens under GNU/Hurd, because there is no official limit. However, | |
| extremely long pathes are extremely uncommon, so we do not want to | |
| use slow dynamic alloc each time we need to build pathes. So we | |
| impose a limit of 4KiB, which should be fairly enough. Note that | |
| this obviously increases stack requirements in GNU/Hurd. | |
| As suggested by Michael Biebl, we have now implemented a generic | |
| approach to handle this via autoconf. | |
| - 2021-09-12: openssl: extended output information on connection failure | |
| Now includes the remote client/server IP address in the log output. | |
| - 2021-09-12: imhttp enhancements - query parameter ingestion & basic auth support | |
| - Basic Authentication support & tests | |
| * configured via imhttp option "basicAuthFile". This option should be configured | |
| to point to your htpasswd file generated via a standard htpasswd tool. | |
| tests: | |
| * imhttp-post-payload-basic-auth.sh | |
| * imhttp-post-payload-basic-auth-vg.sh | |
| - Query parameter ingestion capability & tests | |
| use t `addmetadata` option to inject query parameters into | |
| metadata for imhttp input. | |
| DISTRO PACKAGERS BEWARE: NEW DEPENDENCY FOR IMHTTP: | |
| libaprutil (libaprutil1-dev on debian'ish, apr-util-devel on Red Hat) | |
| Thanks to Nelson Yen for the patch. | |
| - 2021-09-07: testbench bugfix: privdrop tests under root user did not work | |
| When running under root, the privdrop tests did not properly work. This | |
| patch fixes the issue and skips test where necessary. | |
| This also includes some modernization of the related tests. | |
| closes https://github.com/rsyslog/rsyslog/issues/4619 | |
| - 2021-09-07: core/ratelimiting: fix rate limiting for already parsed messages | |
| Rate limiting may not have worked if the considered message had already | |
| been parsed (not having NEEDS_PARSING in msgFlags). | |
| This affects also imuxsock in its default configuration | |
| (useSpecialParser="true" and ratelimit.severity="1") | |
| - 2021-09-07: core bugfix: use of property $wday terminates string | |
| When $wday is used inside a template, all template parts after it | |
| are ignored. For exmaple: | |
| template(name="json_filename" type="string" string="/var/log/%$wday%.log") | |
| would generate something like "/var/log/0" - the ".log" part would be | |
| missing. For the same reason, $wday can not reliably checked in script | |
| filters. | |
| Thanks to Alain Thivillon for reporting the bug and providing an | |
| excellent analysis, which essentiellay was exactly this fix here. | |
| closes https://github.com/rsyslog/rsyslog/issues/4670 | |
| - 2021-09-07: core/queue bugfix: potential misadressing when queue discarded messages | |
| When a discard mark was set, the queue was very busy and discarded messages, a | |
| NULL pointer access could happen. Depending on circumstances, several problems | |
| could occur, including a SEGFAULT. This is now fixed. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4437 | |
| - 2021-09-07: imdiga bugfix: iOverallQueueSize calculation could be incorrect | |
| This issue only affects testbench and rsyslog development debugging. The active | |
| messages counter, used for synchronizing test steps, went wrong when the queue | |
| discarded messages on it's consumer thread. Now fixed. | |
| - 2021-09-06: gnutls driver: SAN priority did not work correctly on server side | |
| PrioritizeSAN was not propagated when accepting a new connection, this is now fixed. | |
| Thanks to Attila Lakatos for the patch. | |
| - 2021-08-24: config: implement script-equavalent for $PrivDrop* statements | |
| closes https://github.com/rsyslog/rsyslog/issues/891 | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2108.0 (aka 2021.08) 2021-08-17 | |
| - 2021-08-16: openssl tls: Improved error message output on tls failures. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4645 | |
| - 2021-08-16: impstats: add percentile metrics tracking functionality | |
| Brief overview: | |
| TO configure tracking percentile metrics in rainerscript: | |
| User would need to define: | |
| - which percentile to track, such as [p50, p99, etc.] | |
| - window size - note, this correlates directly with memory usage to | |
| track the percentiles. | |
| To track a value, user would call built-in function `percentile_observe()` in their configurations to | |
| record an integer value, and percentile metrics would be emitted every | |
| impstats interval. | |
| Thanks to Nelson Yen for the patch. | |
| - 2021-08-12: imfile: add parameter "ignoreolderthanoption" | |
| instructs imfile not to ingest a file that has not been modified in the | |
| specified number of seconds. | |
| Thanks to github user yanjunli76 for the patch (submitted from Nelson Yen) | |
| - 2021-08-10: imklog bugfix: invalid memory adressing, could cause abort | |
| This is a regeression from commit 94c4a87. It introduced a free() call | |
| using an object that was no longer valid (the main pointer to the | |
| to-be-freed object) was already freed at time of use. This could | |
| cause various issues, including a segfault. | |
| Note: this bug was triggerred only during late phase of rsyslog | |
| shutdown, so it did not affect regular operation. | |
| Special thanks to github user wxiaoguang for analyzing the issue | |
| and providing a draft fix proposal, on which this patch builds. | |
| see also https://github.com/rsyslog/rsyslog/pull/4629 | |
| closes https://github.com/rsyslog/rsyslog/issues/4625 | |
| - 2021-08-09: imfile bugfix: deleteStateOnFileDelete missed some state files | |
| When the log file is deleted, imfile would attempt to delete the statefile but it | |
| was missing the file_id part of the statefile name. This means the statefiles were | |
| only removed in the log file was less than 512 characters, because for very small | |
| files the file ID hash is not created. This lead to some state files not being | |
| deleted. | |
| Thanks to pearseimperva for the patch. | |
| - 2021-08-09: imfile bugfix: hash char invalidly added in readmode != 0 | |
| If imfile is ingesting log files with readMode set to 2 or 1, the resulting | |
| messages all have a '#' character at the end. This patch corrects the behaviour. | |
| Note: if some external script "supported" the bug of extra hash character at | |
| the end of line, it may be necessary to update them. | |
| closes https://github.com/rsyslog/rsyslog/issues/4491 | |
| - 2021-08-09: omelasticsearch bugfix: errorFile mutex was not consistently locked | |
| Lock the file during SIGHUPs to avoid issues with concurrent accesses by | |
| writeDataError(). | |
| Thanks to François Poirotte for the patch. | |
| - 2021-08-09: imudp: add socket type (IPv4 vs. 6) to input name | |
| Most importantly, the input name is used for stats counter names as | |
| well. Previously, the same name was used for IPv4 and IPv6, so we had | |
| two counters with an equal name. That left users puzzled. | |
| Unfortunately, this change can potentially require changes to existing | |
| analysis scripts, as the name is now slightly different. | |
| closes https://github.com/rsyslog/rsyslog/issues/4364 | |
| - 2021-08-06: omfwd: add capability for action-specific TLS certificate settings | |
| This permits to override the global definitions for TLS certificates | |
| at the action() level. | |
| - 2021-08-06: imfile bugfix: file handle leak if "freshStartTail" was turned on | |
| - 2021-08-05: imtcp: permit to use different certificate files per input/action | |
| This completes the ability to override global/default TLS settings at the imtcp | |
| input() level. Support for using multiple CAs/Certs per Connection is now provided. | |
| - 2021-08-04: imptcp bugfix: keep alive interval was incorrectly set | |
| The interval was accidentally set to keep alive interval. This has been | |
| corrected. | |
| closes https://github.com/rsyslog/rsyslog/issues/4609 | |
| - 2021-07-08: openssl network driver bugfix: small memory leak | |
| Fixes a static, non-growing memory leak which existed when parameter | |
| "GnutTLSPriorityString" was used. This was primarily a cosmetic issue, | |
| but caused some grief during development in regard to memory leak | |
| detectors. | |
| Note: yes, this is for openssl -- the parameter name is historical. | |
| - 2021-07-07: psrv bugfix: abort if no listener could be started | |
| Modules (like imtcp and imdiag) which use tcpsrv could abort or | |
| otherwise malfunction if no listener for a specific input could | |
| be started. | |
| Found during implementing a new feature, no report from practice. | |
| But could very well happen. | |
| - 2021-07-07: mmkubernetes bugfix: apiserver error handling | |
| - Added graceful handling of apiserver errors with unexpected responses, | |
| i.e., anything other than 200, 404, or 429. Idea is that apiserver | |
| transient error state will recover. We don't want mmkubernetes to miss | |
| metadata resolution for containers that don't have cached metadata. | |
| During these transient error states, mmkubernetes will provide basic | |
| container file path based resolution of namespace and pod metadata for | |
| new pods whose metadata is not yet cached. After this error state | |
| recovers, mmkubernetes is expected to resume its metadata resolution as | |
| expected. | |
| - Added a unit test case for apiserver return 500 with changes to mock server | |
| - Fixed existing unit test that was failing due to missing expected results file | |
| - Added mmkubernetes unit tests to testbench | |
| Thanks to Abdul Waheed for the patch (submitted from Nelson Yen). | |
| - 2021-07-07: ommongodb bugfixes | |
| - Fix Segmentation fault when server is down | |
| - Add server connexion check while resuming | |
| Thanks to Kevin Guillemot for the patch. | |
| - 2021-06-28: omkafka improvements | |
| - drain librdkafka queues and retry later during rsyslog restart or hup. This | |
| re-injects messages into rsyslog's native queues. | |
| - add statsname on per kafka instance for better visibility | |
| - omkafka - count errors related ssl as "errors_ssl" | |
| Thanks to Nelson Yen for the patch. | |
| - 2021-06-23: some CI/QA improvements, Travis-CI disabled | |
| For the time being, Travis CI is disabled because it was outdated and Travis also | |
| changed their system. We will re-evaluate if we re-enable it. Since quite a while | |
| the Travits tests were redundant with the rest of CI, so this does not reduce | |
| coverage. | |
| - 2021-06-23: omhttp bugfix: dynrestpath param in batch mode invalid | |
| When batchmode was used, the templates could not be used to | |
| expand dynrestpath. We are now storing the restpath param | |
| within the batch data if we are in batch mode. | |
| When we are in batch mode, and the restpath value changes, the | |
| batch is submitted and reinitialized | |
| closes: https://github.com/rsyslog/rsyslog/issues/4567 | |
| - 2021-06-17: add predefined template RSYSLOG_SyslogRFC5424Format | |
| This is essentially the same as RSYSLOG_SyslogProtocol23Format with | |
| a better name and a fix to remove the unnecessary LF at the end of | |
| the message. | |
| The different name also enables us to fix the LF issue without | |
| any concern about backwards compatibility. | |
| closes https://github.com/rsyslog/rsyslog/issues/4384 | |
| - 2021-06-17: impstats/bugfix: _sender_stats reports integer counter as string | |
| Note that this introduces a small backwards incompatibility: in previous output | |
| the field was of string type, now it is integer (as intended). We discussed this | |
| on the mailing list and the overwhelming thought was that this is not a problem | |
| because almost all analysis backends are able to cover that format change. This made | |
| the bugfix essentially costmetic. | |
| HOWEVER, if you still experience issues, please let us know. We can add an option | |
| to provide the previous format, and just spared to do so because there was no | |
| evidence it was needed. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2106.0 (aka 2021.06) 2021-06-15 | |
| NOTE: the prime new feature is support for TLS and non-TLS connections | |
| via imtcp in parallel. Furthermore, most TLS parameters can now be overriden | |
| at the input() level. The notable exceptions are certificate files, something | |
| that is due to be implemented as next step. | |
| - 2021-06-14: new global option "parser.supportCompressionExtension" | |
| This permits to turn off rsyslog's single-message compression extension | |
| when it interferes with non-syslog message processing (the parser | |
| subsystem expects syslog messages, not generic text) | |
| closes https://github.com/rsyslog/rsyslog/issues/4598 | |
| - 2021-05-12: imtcp: add more override config params to input() | |
| It is now possible to override all module parameters at the input() level. Module | |
| parameters serve as defaults. Existing configs need no modification. | |
| - 2021-05-06: imtcp: add stream driver parameter to input() configuration | |
| This permits to have different inputs use different stream drivers | |
| and stream driver parameters. | |
| closes https://github.com/rsyslog/rsyslog/issues/3727 | |
| - 2021-04-29: imtcp: permit to run multiple inputs in parallel | |
| Previously, a single server was used to run all imtcp inputs. This | |
| had a couple of drawsbacks. First and foremost, we could not use | |
| different stream drivers in the varios inputs. This patch now | |
| provides a baseline to do that, but does still not implement the | |
| capability (in this sense it is a staging patch). | |
| Secondly, we now ensure that each input has at least one exclusive | |
| thread for processing, untangling the performance of multiple | |
| inputs from each other. | |
| see also: https://github.com/rsyslog/rsyslog/issues/3727 | |
| - 2021-04-27: tcpsrv bugfix: potential sluggishnes and hang on shutdown | |
| tcpsrv is used by multiple other modules (imtcp, imdiag, imgssapi, and, | |
| in theory, also others - even ones we do not know about). However, the | |
| internal synchornization did not properly take multiple tcpsrv users | |
| in consideration. | |
| As such, a single user could hang under some circumstances. This was | |
| caused by improperly awaking all users from a pthread condition wait. | |
| That in turn could lead to some sluggish behaviour and, in rare cases, | |
| a hang at shutdown. | |
| Note: it was highly unlikely to experience real problems with the | |
| officially provided modules. | |
| - 2021-04-22: refactoring of syslog/tcp driver parameter passing | |
| This has now been generalized to a parameter block, which makes it much cleaner and | |
| also easier to add new parameters in the future. | |
| - 2021-04-22: config script: add re_match_i() and re_extract_i() functions | |
| This provides case-insensitive regex functionality. | |
| closes https://github.com/rsyslog/rsyslog/issues/4429 | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2104.0 (aka 2021.04) 2021-04-20 | |
| - 2021-04-19: new contributed module imhiredis | |
| Thanks to Théo Bertin (frikilax) for the patch. | |
| - 2021-04-19: new built-in function get_property() to access property vars | |
| Provides ability to evaluate a rsyslog variable using dynamically | |
| evaluated parameters. | |
| 1st param is the rsyslog param, 2nd param is a key, can be an array | |
| index or key string. | |
| Useful for accessing json sub-objects, where a key | |
| needs to be evaluated at runtime. Can be used to access arrays as well. | |
| Thanks to Nelson Yen for contributing this module. | |
| - 2021-04-19: mmdblookup: add support for mmdb DB reload on HUP | |
| Thanks to Théo Bertin (frikilax) for the patch. | |
| - 2021-04-19: script bugfix: empty array in foreach() improperly handled | |
| When running a foreach() loop inside a ruleset, if the json array/object iterated | |
| over is empty but valid, the foreach will make the message processing in the | |
| ruleset abort operation, no following operation (such as actions) will be | |
| executed after this. | |
| Thanks to Théo Bertin (frikilax) for the patch. | |
| - 2021-04-19: imjournal bugfixes (handle leak, empty file) | |
| Flush the FILE* buffer before rename & fsync in order | |
| to not end up syncing an empty file. | |
| Also, close WorkDir on fsync in order to prevent | |
| file descriptor leakage. | |
| Thanks to github user gerd-rausch for the fix. | |
| - 2021-04-06: new contributed function module fmunflatten | |
| This commit adds a new rainerscript function to unflatten keys in a JSON tree. It | |
| provides a way to expand dot separated fields. | |
| <result> = unflatten(<source-tree>, <key-separator-character>); | |
| It allows for instance to produce this: { "source": { "ip": "1.2.3.4", "port": 443 } } | |
| from this source data: { "source.ip": "1.2.3.4", "source.port": 443 } | |
| Thanks to Julien Thomas for the contribution. | |
| - 2021-02-22: test bugfix: some tests did not work with newer TLS library versions | |
| Newer versions provide TLS versions that cannot be disabled in older versions as they | |
| are unknown there. This is solved by setting restrictions in multiple steps. For | |
| older library versions, the final step will error out, but the other one be applied. | |
| This permits to achieve proper test results. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4534 | |
| - some improvements to project CI | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2102.0 (aka 2021.02) 2021-02-16 | |
| - 2021-02-15: omfwd: add stats counter for sent bytes | |
| Thanks to John Chivian for suggesting this feature. | |
| - 2021-02-15: omfwd: add error reporting configuration option | |
| RSyslog on a plain TCP cannot guarantee the message delivery | |
| without using RELP protocol. Besides that the logs may be | |
| flooded with connection errors making the rest of messages | |
| difficult to find. To alleviate the problem (see issue 3910), | |
| this patch adds a configuration option that enables to reduce | |
| the number of network errors logged and reported. | |
| For example, if each 10th network error message should be logged, | |
| the rsyslog configuration has to be updated as follows. | |
| action(type="omfwd" Target="<IP_ADDR>" Port="<PORT>" Protocol="tcp" ConErrSkip="10") | |
| Thanks to Libor Bukata for the patch. | |
| - 2021-02-15: action stats counter bugfix: failure count was not properly incremented | |
| In some cases the counter was not incremented, most notably with transaction-enabled | |
| actions. | |
| Thanks to github user thinkst-marco for the patch. | |
| - 2021-02-15: action stats counter bugfix: resume count was not incremented | |
| And so it always stayed at zero. | |
| Thanks to github user thinkst-marco for the patch. | |
| - 2021-02-15: omfwd bugfix: segfault or error if port not given | |
| If omfwd is configured via RainerScript config format and the "port" | |
| parameter is not given, a segfault will most likely happen on | |
| connection establishment for TCP connections. For UDP, this is | |
| usually not the case. | |
| Alternatively, in any case, errors may happen. | |
| Note that the segfault will usually happen right on restart so this | |
| was easy to detect. | |
| We did not receive reports from practice. Instead, we found the bug | |
| while conducting other work. | |
| - 2021-01-29: lookup table bugfix: data race on lookup table reload | |
| A data race could happen when a lookup table was reloaded. We found | |
| this while moving to newer version of TSAN, but have no matching | |
| report from practice. However, there is a potential for this to cause | |
| a segfault under "bad circumstances". | |
| - 2021-01-18: testbench modernization | |
| Bump dependency versions, use newer distro versions for some tests. | |
| Make kafka distcheck separate to help diagnose flaky kafka tests. | |
| - 2021-01-16: testbench: fix invalid sequence of kafka tests runs | |
| kafka tests can not run well in parallel (mostly due to ressource | |
| constraints on CI machines). Accidentally, this was not enforced for | |
| one of the tests. That could lead to random failures and false positives. | |
| - 2021-01-14: testbench: fix kafkacat issues | |
| The kafkacat tool has an upper limit of how many messages it can send | |
| at once. Going over that limit causes messages loss. The exact limit | |
| seems to depend on the environment. This causes testbench false positives. | |
| This commit fixes two related issues: | |
| - errors during kafkacat run were not detected - this has been added | |
| - we now have a "max messages at once" setting, after which kafkacat | |
| is restarted for the next batch of messages. It currently is set | |
| to 25,000 msgs per incarnation. All tests loop now to send the | |
| required number of messages. This has been fixed at the testbench | |
| framework level, so no need to adjust individual tests. | |
| - 2021-01-14: testbench: fix year-dependendt clickhouse test | |
| A test had the year value hardcoded and as such failed whenever the | |
| year changed. This patch corrects that. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2012.0 (aka 2020.12) 2020-12-08 | |
| - 2020-12-07: testbench bugfix: some tests did not work in make distcheck | |
| - certificate file missing in dist tarball | |
| - some test cases did not properly specify path to cert file | |
| Thanks to Michael Biebl for alerting us and providing part of | |
| the fix. | |
| closes https://github.com/rsyslog/rsyslog/issues/4446 | |
| - 2020-12-07: immark: rewrite with many improvements | |
| - mark message text can now be specified | |
| - support for rulesets | |
| - support for using syslog API vs. regular internal interface | |
| - support for output template system | |
| - ability to specify is mark message flag can be set | |
| - minor changes and improvements | |
| - 2020-11-30: usability: re-phrase error message to help users better understand cause | |
| see also https://github.com/rsyslog/rsyslog/issues/3910 | |
| - 2020-11-10: add new system property $now-unixtimestamp | |
| Among others, this may be used as a monotonic counter | |
| for doing load-balancing and other things. | |
| Thanks to Nicholas Brown for suggesting this feature. | |
| - 2020-11-04: omfwd: add new rate limit option | |
| Adding new rate limit option to omfwd for rate limiting | |
| syslog messages sent to the remote server | |
| ratelimit.interval: | |
| Specifies the rate-limiting interval in seconds. | |
| Default value is 0, which turns off rate limiting. | |
| ratelimit.burst | |
| Specifies the rate-limiting burst in number of messages. | |
| closes https://github.com/rsyslog/rsyslog/issues/4423 | |
| Thanks to Dinesh-Ramakrishnan for the patch. | |
| - 2020-11-03: omfwd bug: param "StreamDriver.PermitExpiredCerts" is not "off" by default | |
| The default behaviour of expired certificates of stream driver in TLS mode, should | |
| have been that the see tcp transmission is closed due to expired certificates, and | |
| error messages emited in rsyslog status. This was not the case. That in turn could | |
| lead to permitting sessions which should not be permitted. | |
| Thanks to Vincent Zhu for alerting us and providing a great problem analysis | |
| closes: https://github.com/rsyslog/rsyslog/issues/4425 | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2010.0 (aka 2020.10) 2020-10-20 | |
| - 2020-10-13: gnutls TLS subsystem bugfix: handshake error handling | |
| If the tls handshake does not immediatelly finish, gnutls_handShake is called in | |
| doRetry handler again. However the error handling was not | |
| complete in the doRetry handler. A failed gnutls_handShake call | |
| did not abort the connection and properly caused unexpected | |
| problems like in issues: | |
| https://github.com/rsyslog/rsyslog/issues/4270 | |
| https://github.com/rsyslog/rsyslog/issues/4288 | |
| - 2020-10-13: core/msg bugfix: memory leak | |
| There is a missing call to json_object_put(json) if the call to | |
| jsonPathFindParent() failed. It's leaking memory. Depending on workload and config, | |
| this leak can potentially grow large (albeit we did not see reports from practice). | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-10-13: core/msg bugfix: segfault in jsonPathFindNext() when <root> not an object | |
| The segfault gets happens when <bCreate> is 1 and when the <root> | |
| container where to insert the <namebuf> key is not an object. | |
| Here is simple reproducible test case: | |
| // ensure we start fresh | |
| // unnecessary if there was no previous set | |
| unset $!; | |
| set $! = ""; | |
| set $!event!created = 123; | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-10-13: openssl TLS subsystem: improvments of error and status messages | |
| Adding error logs at the ssl handshake failure scenarios. | |
| Adding the header "nsd_ossl:" tag to these logs to identify | |
| the origin module from which logs are generated. | |
| Thanks to Anusha Pai G for the patch. | |
| - 2020-10-06: add 'exists()' script function to check if variable exists | |
| This implements a way to check if rsyslog variables (e.g. '$!path!var') is | |
| currently set of not. | |
| Sample: if exists($!somevar) then ... | |
| closes https://github.com/rsyslog/rsyslog/issues/4385 | |
| - 2020-10-03: core bugfix: do not create empty JSON objects on non-existent key access | |
| Performing a condition (eg: check for an empty string) on a subtree key that do not | |
| exists (depth > 1 from the root container), creates an empty "parent" object. | |
| Depending on your context, you may end up with (kind of...) annoying garbage when | |
| producing object documents (for instance to index in ES). | |
| Also fixes a hypothetical hang condition with an almost (?) unused plugin parameter | |
| passing mode, for details see | |
| https://github.com/rsyslog/rsyslog/issues/4436 | |
| closes https://github.com/rsyslog/rsyslog/issues/4430 | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-09-28: gnutls subsysem bugfix: potential hang on session closure | |
| Some TLS servers don't reply to graceful shutdown requests "for | |
| optimization". This results in rsyslog's omfwd+gtls client to wait | |
| forever for a reply of the TLS server which never comes, due to shutting | |
| down the connection with gnutls_bye(GNUTLS_SHUT_RDWR). | |
| On systemd systems, commands such as "systemctl restart rsyslog" just | |
| hang for 1m30 and rsyslogd gets killed upon timeout by systemd. | |
| This is fixed by replacing the call to gnutls_bye(GNUTLS_SHUT_RDWR) by calls to | |
| gnutls_bye(GNUTLS_SHUT_WR) which is sufficient and doesn't wait for a | |
| server reply. | |
| As an example, Kiwi Syslog server is known to cause this issue. | |
| Thanks to Renaud Métrich for the patch. | |
| - 2020-09-23: core/network bugfix: obey net.enableDNS=off when querying local hostname | |
| Local hostname resolution used DNS queries even if the enableDNS was set to off, and | |
| this could cause unexpected delays in the HUP signal handling if the DNS server was | |
| not responsive. | |
| Thanks to Samu Nuutamo for the fix. | |
| - 2020-09-14: core bugfix: potential segfault on query of PROGRAMNAME property | |
| A data race can happen on variable iLenProgram as it is not guarded | |
| by the message mutex at time of query. This can lead to it being | |
| non -1 while the buffer has not yet properly set up. | |
| Thanks to Leo Fang for alerting us and a related | |
| patch proposal. | |
| replaces https://github.com/rsyslog/rsyslog/pull/4300 | |
| - 2020-09-14: imtcp bugfix: broken connection not necessariy detected | |
| Due to an invalid return code check, broken TCP sessions could not | |
| necessarily be detected "right in time". This can result is the loss | |
| of one message. | |
| closes https://github.com/rsyslog/rsyslog/issues/4227 | |
| Thanks to Leo Fang for the patch. | |
| - 2020-09-14: new module: imhttp - http input | |
| permits to receive log data via HTTP. | |
| uses http library to provide http input. | |
| user would need to configure an 'endpoint' as input, along | |
| with a ruleset, defining how the input should be routed in | |
| rsyslog. | |
| Thanks to Nelson Yen for contributing this module. | |
| - 2020-09-11: mmdarwin bugfix: potential zero uuid when reusing existing one | |
| - fix a use-after-free variable during darwin uuid message extraction | |
| - improve debug/output by logging uuid parse errors | |
| Thanks to github user frikilax for the patch. | |
| - 2020-09-10: imdocker bugfix: build issue on some platforms | |
| An invalid variable type was used, leading to compile errors at least on | |
| all platform that use gcc 10 and above. Otherwise, however, it looks like the | |
| issue caused no real harm. | |
| - 2020-09-07: omudpspoof bugfix: make compatbile with Solaris build | |
| Thanks to Dagobert Michelsen for the patch. | |
| - 2020-09-03: testbench fix: python 3 incompatibility | |
| - 2020-09-02: core bugfix: segfault if disk-queue file cannot be created | |
| When using Disk Queue and a queue.filename that can not be created | |
| by rsyslog, the service does not switch to another queue type as | |
| supposed to and crashes at a later step. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4282 | |
| - 2020-08-26: cosmetic: fix dummy module name in debug output | |
| When we have optional components (like imjournal) a dummy module | |
| is used. It's sole purpose is to emit "this module is not available". | |
| During init, the module emitted an invalid module name into the debug | |
| log. This has now been replaced by the generic term "dummy". | |
| Note: it is highly unlikely that someone will ever see that message | |
| at all, as it is unlikely for the dummy modules to be build. | |
| see also: https://github.com/rsyslog/rsyslog/commit/84a7e3d80b80106dcc86c273ed8cf78a6c11c722#r41782830 | |
| Thanks to Thomas D. (whissi) for the patch. | |
| - 2020-08-26: config bugfix: intended warning emitted as error | |
| When there are actions configured after a STOP, a warning should be | |
| emitted. In fact, an error message is generated. This prevents the | |
| construct, which may have some legit uses in exotic settings. It | |
| may also break older configs, but as the message is an error | |
| for so long now, this should be no longer of concern. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2008.0 (aka 2020.08) 2020-08-25 | |
| - 2020-08-25: imdocker bugfix: error reporting not always correct | |
| A wrong function to obtain the error code was used. This | |
| could lead to invalid error messages. | |
| Thanks to Steve Grubb for the bug report and fix proposal. | |
| closes https://github.com/rsyslog/rsyslog/issues/4381 | |
| - 2020-08-25: imptcp: add max sessions config parameter | |
| The max is per-instance, not global across all instances. | |
| There is also a bugfix where if epoll failed I think we could leave a | |
| session linked in the list of sessions, this code unlinks it. | |
| Thank to Alfred Perlstein for the patch. | |
| - 2020-08-24: omelasticsearch bugfix: reply buffer reset after health check | |
| The issue happens when more than one server is defined on the | |
| action. On that condition a health check is made through | |
| checkConn() before sending the POST. The replyLen should be | |
| set back to 0 after the health check, otherwise the response | |
| data received from the POST gets appended to the end of the | |
| last health check. | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-08-14: omfile: do no longer limit dynafile cache size in legacy format | |
| When using obsolete legacy config format, omfile had a hard limit of | |
| 1,000 dynafile cache entries. This does not play well with very | |
| large installation. This limit is now removed and converted into | |
| a warning if cache size > 25,000 is specified. | |
| Note: the problem can easily be worked-around by using modern | |
| config format (RainerScript). | |
| closes: https://github.com/rsyslog/rsyslog/issues/4241 | |
| - 2020-08-13: imudp: fix very small, static memory leak | |
| When ruleset support was used, the ruleset name was not freed upon rsyslog | |
| termination. While this has no consequences for regular runs, it generates | |
| leak errors under memory debuggers and as such makes debugging harder than | |
| necessary. | |
| Thanks to github user frikilax for the patch. | |
| - 2020-08-13: omelasticsearch: add parameter skipPipelineIfEmpty | |
| When POST'ing a document, Elasticsearch does not allow an empty pipeline | |
| parameter value. This patch introduces boolean option skipPipelineIfEmpty | |
| to the omelasticsearch action. When set to true, the pipeline parameter | |
| won't be posted. Default is false so we do not modify current behavior. | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-08-12: systemd service file removed from project | |
| This was done as distros nowadays have very different service files and it no | |
| longer is useful to provide a "generic" (sic) example. | |
| see also: https://github.com/rsyslog/rsyslog/issues/4333 | |
| - 2020-08-11: gnutls TLS driver bugfix: EKU check not done properly | |
| When the server accepted a new connection, it did not properly set the | |
| dataTypeCheck field based on the listening socket. That resulted in | |
| skipping ExtendedKeyUsage (EKU) check on the client. | |
| Thanks to Daiki Ueno for the patch. | |
| - 2020-08-06: MMDARWIN:: improve configuration flexibility and UUID fix | |
| -t pu now able to get fields from local variables ($.) | |
| - now able to configure a custom root container for mmdarwin fields | |
| - now able to put nested keys ($!key1!key2) | |
| - don't regenerate a UUID each time, but instead check if one exists before | |
| creating it (allow successive calls without losing previous UUID) | |
| Thanks to github user frikilax for the contribution. | |
| - 2020-08-06: add --enable-imjournal=optional ./configure option | |
| - 2020-08-06: IMPCAP::Fixes: segfault, memory and build corrections | |
| * fix bug in ethernet packets parsing | |
| * fix removes build error with gcc10: 'multiple definition of...' | |
| * resolve memory leak during interface init failure (device not freed after post-create error) | |
| * add test 'impcap_bug_ether' to prove ethernet parser fix is working | |
| Thanks to github user frikilax for the contribution. | |
| closes https://github.com/rsyslog/rsyslog/issues/4332 | |
| - 2020-07-14: CI: add support for github actions | |
| - 2020-07-14: imklog: add ruleset support | |
| see also: https://github.com/rsyslog/rsyslog/issues/4344#issuecomment-658001854 | |
| see also: https://github.com/rsyslog/rsyslog/issues/106 | |
| - 2020-07-06: config system fix: ChkDisabled method to make config.enabled work | |
| There was wrong negation in the method so it returned 0/1 in reverse | |
| and also it did not mark the node to not be reported as unknown at all | |
| times which is needed after all. | |
| Thanks to Jiri Vymazal for the patch. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2006.0 (aka 2020.06) 2020-06-23 | |
| - 2020-06-22: queue: permit ability to double size at shutdown | |
| This prevents message loss due to "queue full" when re-enqueueing data | |
| under quite exotic settings. | |
| see also https://github.com/rsyslog/rsyslog/issues/3941#issuecomment-549765813 | |
| closes https://github.com/rsyslog/rsyslog/issues/4020 | |
| - 2020-06-22:Fixing imfile segfaulting on selinux denial | |
| If imfile is denied access to file watched trough symlink there is | |
| unchecked condition resulting in access to not initialized memory. | |
| - 2020-06-22: openssl: Fixed memory leak when tls handshake failed. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4319 | |
| - 2020-06-22: change systemd service file to wait for network | |
| now that rsyslog is usually only installed for real syslog servers, | |
| we should assume that some network listening or forwarding happens | |
| on start. As such we need to start a bit later, after the network. | |
| This poses no problem as systemd nowadays comes with journal which | |
| is in almost all cases configured to buffer log data while | |
| rsyslog is not yet running. | |
| see also https://github.com/rsyslog/rsyslog-pkg-rhel-centos/issues/72 | |
| - 2020-06-22: NEW INPUT MODULE:: impcap, network packets input parser | |
| Thanks to github user frikilax for the contribution. | |
| - 2020-06-22: ksi bugfix: Optimized code in KSI module initialization fixed. | |
| KSI module initialization will not stuck in infinite loop when code is | |
| built with optimization -O2. | |
| - 2020-06-05: operatingstatefile bugfix: month was given too low | |
| The month was printed with the range 0 (January) to 11 (December). | |
| This has now been corrected. | |
| closes https://github.com/rsyslog/rsyslog/issues/4292 | |
| - 2020-06-05: build system: add "optional" build functionality to some components | |
| Nameley: | |
| --enable-libdbi=optional | |
| --enable-mmdblookup=optional | |
| --enable-imkafka=optional | |
| --enable-omkafka=optional | |
| If used, builds a dummy module which just emits a "module not supported | |
| on this platform" error message when loaded. | |
| Primary use case for this system is Debian-ish builds on SUSE OBS, | |
| where we prefer to have a single package definition for all versions | |
| (else things get much more complicated). | |
| - 2020-05-23: config system bugfix: backticks cat segfault if file cannot be opened | |
| when a `cat <filename>` construct is used in rsyslog.conf and <filename> can not | |
| be accessed (does not exist, no permissions, ...), rsyslog segfaults. | |
| Thanks to Michael Skeffington for notifying us and providing root cause analysis. | |
| closes https://github.com/rsyslog/rsyslog/issues/4290 | |
| - 2020-05-15: imtcp bugfix: octet framing/stuffing problem with discardTruncatedMsg on | |
| When "discardTruncatedMsg" was enabled in imtcp, messages were incorrectly | |
| skipped if the last character before the truncation was the LFdelimiter. | |
| Also adds two testbench tests for this case. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4281 | |
| - 2020-05-12: ompipe bugfix: race during HUP | |
| When HUP was received, the write mutex was not aquired. This could | |
| lead to unexpected invalidation of the output file descriptor. | |
| Thanks to Julien Thomas for alerting us on this issue. | |
| see also https://github.com/rsyslog/rsyslog/pull/4136#issuecomment-578326278 | |
| - 2020-05-12: ompipe: add action parameter tryResumeReopen | |
| Sometimes we need to reopen a pipe after an ompipe action gets | |
| suspended. Sending an HUP signal to rsyslog does the job but requires | |
| an interraction with rsyslog. The patch adds support for a new boolean | |
| option, tryResumeReopen, for the ompipe action. It mimics what an HUP | |
| signal would do. | |
| Thanks to Julien Thomas for the patch. | |
| - 2020-05-12: imjournal: remove strcat call | |
| Thanks to Jeff Marckel for the patch. | |
| - 2020-05-12: build system: libzcmq version requirement needs to be bumped | |
| Thanks to Thomas Deutschmann for pointing this out. | |
| closes https://github.com/rsyslog/rsyslog/issues/3957 | |
| - 2020-05-12: testbench: download ElasticSearch binaries from rsyslog.com | |
| The official ElasticSearch download site sometimes denies the download. | |
| - 2020-05-11: openssl netstream driver bugfix: context leak | |
| The context object was not properly freed. | |
| Thanks to Michael Zimmermann for the fix. | |
| - 2020-05-11: omhttp: Add support for multiple http headers | |
| Allows the inclusion of multiple http headers on the REST call. | |
| Thanks to callmegar for the patch. | |
| - 2020-04-29: core bugfix: group id could not be obtained for very large groups | |
| Thanks to github user emilbart for the patch. | |
| - 2020-04-29: testbench additions (relp broken connection test) | |
| - 2020-04-29: omudpspoof bugfix: issues with oversized messages | |
| First issue was an incorrect packet length in UDP Header. It has to be the FULL UDP Packet | |
| regardless of the MTU Setting. As a result regardless of IP fragmentation, the MTU setting | |
| also limited the siizmax size of the UDP message. | |
| The second issue was incorrect calculation of the UDP Checksum with libnet if | |
| IP fragmentation was used (Based on MTU Setting). As a result, the network packets were | |
| dropped by the tcp stack before they even could reach there target. The workarround for this | |
| problem is, that we set the UDP Checksum to 0x0000 which allows skipping of the checksum | |
| test. Fixing the problem by calculating the correct UDP Checksum would require some | |
| code changes in the libnet. | |
| Also fixed the omudpspoof bigmsg test and increased the testing size to 16KB. | |
| - 2020-04-29: omprog: fix assert failed on HUP with output flag | |
| If the 'output' setting of omprog was used and rsyslog received a HUP | |
| signal just after starting (and before the omprog action received the | |
| first log to process), an internal assertion could fail, causing | |
| rsyslog to terminate. The failure message was "rsyslogd: omprog.c:660: | |
| closeOutputFile: Assertion `pCtx->bIsRunning' failed." | |
| The failure could also occur if rsyslog received a HUP signal during | |
| the shutdown sequence. | |
| This bug was introduced in v8.2004 by PR https://github.com/rsyslog/rsyslog/pull/4255 | |
| Although a test already existed that checked the interaction of HUPs | |
| with the 'output' setting, it didn't always fail in this particular case | |
| due to timing conditions. The test has been improved to cover this case | |
| more reliably. | |
| Thanks to Joan Sala Isern for the patch. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2004.0 (aka 2020.04) 2020-04-28 | |
| - 2020-04-28: ksi bugfix: When KSI module is suddenly closed, files are finalized | |
| In async. mode all pending signature requests are closed immediately and | |
| unsigned block marker is attached with message about sudden closure. | |
| Similar approach is used for blocks that already contain some records. | |
| Empty blocks are just closed without any metadata. | |
| Thanks to Taavi Väljaots for the patch. | |
| - 2020-04-28: ksi bugfix: Signer thread initialization is verified before usage. | |
| When signer thread is created in rsksiInitModule thread successful | |
| initialization is verified before returning the function. This will | |
| prevent adding records to not initialized module and in case of an | |
| error signature files opened will contain only magic bytes. | |
| Thread flags replaced with thread state. | |
| When init module fails, module is disabled. | |
| Thanks to Taavi Väljaots for the patch. | |
| - 2020-04-28: ksi bugfix: Hardcoded default hash algorithm replaced with 'default' | |
| Instead of hardcoded SHA-256 KSI_getHashAlgorithmByName("default") | |
| is used to get default hash function. | |
| Function rsksiSetHashFunction and SetCnfParam updated. | |
| Thanks to Taavi Väljaots for the patch. | |
| - 2020-04-28: imfile bugfix: poential segfault in stream object on file read | |
| - if cstrLen(pThis->prevMsgSegment) > maxMsgSize then len calculation | |
| become negative if cstrLen(thisLine) < cstrLen(pThis->prevMsgSegment) | |
| This causes illegal access to memory location and thus causing segfault. | |
| - assigning len = 0 if cstrLen(pThis->prevMsgSegment) > maxMsgSize so that | |
| it access the correct memory location. | |
| Thanks to github user jaankit | |
| - 2020-04-28: openssl TLS drivers: made more reliable for older openssl versions | |
| OpenSSL can retry some failed operations, but older versions need an explicit | |
| opt-in to do so. This is now done. | |
| - 2020-04-28: omprog: fix bad fd errors in daemon mode | |
| When omprog was used with the 'forceSingleInstance=on' option, and/or | |
| the 'output' setting, "bad file descriptor" errors occurred, which | |
| prevented the external program to be executed and/or the program output | |
| to be correctly captured. The bug could also manifest as "resource | |
| temporarily unavailable" errors, or other errors related to the use of | |
| invalid/reassigned file descriptors. These errors only happened when | |
| rsyslog ran in daemon mode (i.e. they didn't happen if rsyslogd was | |
| run with the '-n' option). | |
| The cause of the bug was that omprog opened the pipe fds needed by | |
| these flags during the configuration load phase (in the 'newActInst' | |
| module entrypoint). This is a bad place since the fork of the daemon | |
| occurs after this phase, and all fds are closed when the daemon process | |
| is started (see 'initAll' in rsyslogd.c), hence invalidating the | |
| previously opened fds. | |
| To correct this, the single child process and the output capture thread | |
| are now started later, when the first log message is received by the | |
| first worker thread. (Note: the 'activateCnf' module entrypoint, despite | |
| being invoked after the fork, cannot be used for this purpose, since it | |
| is invoked per module, not per action instance.) | |
| Currently no automated test exists for this use case since the testbench | |
| always runs rsyslog in non-daemon mode. | |
| Affected versions: v8.38 and later | |
| closes: https://github.com/rsyslog/rsyslog/issues/4247 | |
| Thanks to Joan Sala Isern for the patch. | |
| - 2020-04-28: omfile bugfix: $outchannel split log lines at rotation time | |
| - 2020-04-17: openssl: add support for libreSSL | |
| Disable use of "@SECLEVEL" in default cipher string and | |
| avoid SSL_CONF_CTX_set_flags() API when LIBRESSL is used. | |
| This means tlscommands will not work. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4210 | |
| - 2020-03-04: imudp bugfix: build problems on some Linux kernel versions | |
| Thanks to Wen Yang for the patch. | |
| - 2020-03-02: conf output bugfix: -o produces missing space between call and rulename | |
| Thanks to Tetiana Ohnieva for the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/3761 | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2002.0 (aka 2020.02) 2020-02-25 | |
| - 2020-02-25: imfile: add per minute rate limiting | |
| Add MaxBytesPerMinute and MaxLinesPerMinute options. | |
| These take integer values and, respectively, limit the number | |
| of bytes or lines that may be sent in a minute. | |
| This can be used to put a limit on the count or volume of logs | |
| that may be sent for an imfile. | |
| Thanks to Greg Farrell for the patch. | |
| - 2020-02-24: core: add global parameter "security.abortOnIDResolutionFail" | |
| This parameter controls whether or not rsyslog aborts when a name ID | |
| lookup fails (for user and group names). This is necessary as a security | |
| measure, as otherwise the wrong permissions can be assigned or privileges | |
| are not dropped. | |
| CHANGE OF BEHAVIOR | |
| The default for this parameter is "on". In previous versions, the default | |
| was "off" (by virtue of this parameter not existing). As such, existing | |
| configurations may now error out. | |
| We have decided to accept this change of behavior because of the potential | |
| security implications. | |
| closes https://github.com/rsyslog/rsyslog/issues/4164 | |
| - 2020-02-24: openssl TLS driver bugfix: chained certificates were not accepted | |
| This was supported since always inside GnuTLS driver, but was missing for openssl one. | |
| - 2020-02-24: core bugfix: too early parsing of incoming messages | |
| In theory, rsyslog should call parsers on the queue worker threads whenever | |
| possible. This enables the parsers to be executed in parallel. There are | |
| some cases where parsers needs to be called earlier, namely when parsed | |
| data is needed for rate-limiting. | |
| The logic to do this previously did not work correctly and was fixed six | |
| years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive: | |
| it actually makes the early parser call now mandatory, effectively moving | |
| parsing to the input side where there is no to little concurrency. | |
| We still do not need to call the parser when all messages, regardless of | |
| severity, need to be rate-limited. This is the default and very frequent | |
| case. This patch introduces support for this and as such makes parsers | |
| able to run in parallel in the frequent case again. | |
| closes https://github.com/rsyslog/rsyslog/issues/4187 | |
| - 2020-02-20: testbench bugfix: two minor issues in omkafkadynakey.sh test | |
| lead to false positives during test runs (depending on circumstances) | |
| closes: https://github.com/rsyslog/rsyslog/issues/4134 | |
| - 2020-02-20: testbench: set max extra data length for tcpflood from 200 to 512KiB | |
| Added a imrelp test for big messages (256KB). | |
| closes: https://github.com/rsyslog/rsyslog/issues/4158 | |
| - 2020-02-20: config system bugfix: 'config.enabled' directive oddities | |
| Previously the directive was processed way too late which caused false | |
| errors whenever it was set to 'off' and possibly other problems. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2020-02-09: imfile bugfix: timeout did not work on very busy system | |
| The timeout feature was soley based on timeouts of the poll() | |
| system call. On a very busy system, this would probably happen | |
| very seldomly. Moreover, the timeout could occur later than | |
| expected on any system with high load. | |
| The issue was not reported from practice but discovered during | |
| CI system improvements. | |
| - 2020-01-30: build system: change --enable-imfile-tests default to "yes" | |
| This was accidentally set to "no" some time ago (actual commit unknown). Tests for | |
| imfile should by default run when imfile is enabled. | |
| see also https://github.com/rsyslog/rsyslog/issues/4120 | |
| - 2020-01-27: build system: add option --enable-gnutls-tests | |
| This enables us to build GNUtls support but not necessarily | |
| test it in CI. This is useful for some specialised subcomponent | |
| test. The default is enabled if gnutls is enabled and disabled if not. | |
| - 2020-01-26: testbench: new test for loadbalancing via global vars | |
| This is a popular functionality which had not been routinely tested | |
| in the past. | |
| - 2020-01-26: mmdblookup bugfix: invalid data returned when no entry found | |
| Since the upgrade of the package libmaxminddb on FreeBSD (1.3.2_2 -> 1.4.2), | |
| the module mmdblookup returns the first entry of the mmdb database even if the entry | |
| is not found. After some debug, I found the solution in the official maxminddb | |
| repository : to check if the entry is in database, we must check the found_entry | |
| attribute, otherwise the function MMDB_get_entry_data_list will return the first | |
| entry of the database if the entry is not found in it. | |
| Thanks to Kevin Guillemot for the patch. | |
| - 2020-01-23: oversize message log bugfix: do not close fd -1 | |
| The oversize message log fd is always closed on HUP, even if it never | |
| was opened (and thus has -1 value). This patch corrects the issue. | |
| The bug had no know-bad effect in practice other than getting an | |
| (ignored) error status from close(). However, it introduced warnings | |
| in test runs (e.g. when running under valgrind). | |
| - 2020-01-22: imfile bugfix: saving of old file_id for statefiles | |
| Previously we saved old file_id unconditionally, which led to not | |
| deleting old statefiles if files changes without rsyslog running. | |
| Now it should work correctly. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2020-01-22: imfile bugfix: misadressing and potential segfault | |
| Commit 3f72e8c introduced an invalid memory allocation size. This lead to | |
| too-short alloc and thus to overwrite of non-owned memory. That in turn | |
| could lead to segfaults or other hard to find problems. | |
| The issue was detected by our upgraded CI system. We did not receive | |
| any problem reports in practice. Nevertheless, the problem is real and | |
| people should update affected versions to patched ones. | |
| The bug was present in scheduled stable release 8.1911.0 and 8.2001.0. | |
| see also: https://github.com/rsyslog/rsyslog/issues/4120 | |
| see also: https://github.com/rsyslog/rsyslog/pull/4141 | |
| - 2020-01-20: core bugfix: potential race during HUP | |
| when rsyslog is HUPed immediately after startup and before it is fully | |
| initialized, there is a potential race with the list of loaded modules. | |
| This patch ensures no bad things can happen in that case. | |
| Detected by LLVM TSAN, not seen in practice. | |
| - 2020-01-20: testbench improvements and fixes | |
| modernize tests, reduce robustness against slow machines, provide some | |
| test framework functional enhancements, and optimize some tests. | |
| Also includes some code changes to C testing components. Among others, | |
| tests have slightly been speeded up by reducing the wait time at queue | |
| shutdown. This is possible because of better overall completion checks. | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.2001.0 (aka 2020.01) 2020-01-14 | |
| - 2020-01-12: core bugfix: race condition related to libfastjson when using DA queue | |
| Rsyslogd aborts when writing to disk queue from multiple workers simultaneously. | |
| It is assumed that libfastjson is not thread-safe. | |
| Resolve libfastjson race condition when writing to disk queue. | |
| see also https://github.com/rsyslog/rsyslog/issues/4041 | |
| Thanks to MIZUTA Takeshi for the fix. | |
| - 2020-01-12: omfwd bugfix: parameter streamdriver.permitexpiredcerts did not work | |
| closes https://github.com/rsyslog/rsyslog/issues/4098 | |
| - 2020-01-11: Bugfix: KSI module + dynafile in asynchronous mode fixed | |
| Thanks to Taavi Valjaots for the patch | |
| - 2020-01-08: tls driver: add support to configure certificate verify depth | |
| Support added in omfwd as instance parameter: | |
| streamdriver.TlsVerifyDepth | |
| Support added in imtcp as module parameter: | |
| streamdriver.TlsVerifyDepth | |
| Can be 2 or higher. | |
| Support added into ossl driver | |
| Support added into gtls driver | |
| Added testcases for both drivers. | |
| closes: https://github.com/rsyslog/rsyslog/issues/4035 | |
| - 2020-01-08: modernization of testbench | |
| moved some tests to newer standards, hardened them against slow testbench machines, | |
| kafka component download improvements, and prevent dangling left-over test tool | |
| instances from aborted tests | |
| - 2020-01-07: tls subsystem bugfix: default for permitExpiredCerts was invalidly "on" | |
| The problem occured with commit 3d9b8df in December 2018 and went into | |
| scheduled stable 8.1901.0. Unfortunately, the change in default was not detected | |
| until a year later. This commit re-enables the previous default ("off"), which is | |
| also the only sensible default from a security PoV. Unfortunately, new 2019 | |
| deployments may begin to see connection rejection when usin expired certs. As | |
| expired certs should not be used, this hopefully will not cause problems in | |
| practice. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2020-01-01: testbench: improve ElasticSearch test speed | |
| We now support re-using suitable running ES instances, which reduces the | |
| number of restarts. | |
| - 2019-12-31: omelasticsearch: improve curl reply buffer handling | |
| The curl reply buffer (pWrkrData->reply) was allocated, realloced and freed with | |
| each request. This has now been reduced to once per module, slightly increasing | |
| overall performance. | |
| closes https://github.com/rsyslog/rsyslog/issues/1964 | |
| - 2019-12-31: config system: emit proper error message on $ in double-quoted string | |
| closes https://github.com/rsyslog/rsyslog/issues/2869 | |
| - 2019-12-30: core bugfix: rsyslog aborts when config parse error is detected | |
| In defaut settings, rsyslog tries to continue to run, but some data | |
| structures are not properly initialized due to the config parsing error. | |
| This causes a segfault. | |
| In the following tracker, this is the root cause of the abort: | |
| see also https://github.com/rsyslog/rsyslog/issues/2869 | |
| - 2019-12-30: fix some alignment issues | |
| So far, this worked everywhere (for years). But it may still have | |
| caused issues on some platforms. | |
| closes https://github.com/rsyslog/rsyslog/issues/2608 | |
| - 2019-12-27: core bugfix: APP-NAME fields could become empty | |
| RFC 5424 specifies that an empty APP-NAME needs to be indicated by | |
| "-". Instead, the field could become empty under certain conditions. | |
| If so, outgoing 5424 messages were invalidly formatted. | |
| This happened under quite unusual conditions, but could be seen | |
| in practice. | |
| closes https://github.com/rsyslog/rsyslog/issues/4043 | |
| - 2019-12-27: core bugfix: reopen /dev/urandom file descriptor after fork on Linux | |
| This patch updates prepareBackground() in tools/rsyslogd.c to reopen any file | |
| descriptors used for random number generation in the child process. This fixes | |
| an issue on Linux systems where the file descriptor obtained for /dev/urandom | |
| by seedRandomNumber() in runtime/srutils.c was left closed after the fork. This | |
| could be observed in procfs, where /proc/fd/ would show no open descriptors to | |
| /dev/urandom in the forked process. /dev/urandom is reopened as the child may be | |
| be operating in a jail, and so should not continue to use file descriptors from | |
| outside the jail (i.e. inherited from the parent process). | |
| I found that this issue led to rsyslog intermittently hanging during seedIV() | |
| in runtime/libgcry.c. After the fork, the closed file descriptor number tended | |
| to get re-assigned. randomNumber() would then read from an incorrect (although | |
| still valid) file descriptor, and could block (depending on the state of that | |
| file descriptor). This gave rise to the intermittent hang that I observed. | |
| Thanks to Simon Haggett for the patch. | |
| - 2019-12-20: imdocker bugfix: did not compile without atomic operations | |
| - 2019-12-20: omclickhouse: new parameter "timeout" | |
| Thanks to Pavlo Bashynskiy for the patch. | |
| - 2019-12-20: omhiredis: add 'set' mode plus some fixes | |
| - new mode 'set' to send SET/SETEX commands | |
| - new parameter 'expiration' to send SETEX instead of SET commands (only applicable to 'set' mode) | |
| - fixes to missing frees | |
| Thanks to github user frikilax for the patch. | |
| - 2019-12-18: relp: Add support setting openssl configuration commands. | |
| Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp. | |
| (Using relpSrvSetTlsConfigCmd and relpCltSetTlsConfigCmd) | |
| OpenSSL Version 1.0.2 or higher is required for this feature. | |
| A list of possible commands and their valid values can be found in the | |
| documentation: https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html | |
| The setting can be single or multiline, each configuration command is | |
| separated by linefeed (n). Command and value are separated by | |
| equal sign (=). Here are a few samples: | |
| tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.2" | |
| tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 | |
| MinProtocol=TLSv1.2" | |
| Add to new testcases for librelp and tlscfgcmd. | |
| closes https://github.com/rsyslog/rsyslog/issues/3959 | |
| - 2019-12-18: bugfix core: potential segfault in template engine | |
| under some circumstances (not entirely clear right now), memory | |
| was freed but later re-used as state-tracking structures were not | |
| properly maintained. Github issue mentioned below has full details. | |
| Thanks to github user snaix for analyzing this issue and providing | |
| a patch. I am committing as myself as snaix did not disclose his or | |
| her identity. | |
| closes https://github.com/rsyslog/rsyslog/issues/3019 | |
| closes https://github.com/rsyslog/rsyslog/issues/4040 | |
| - 2019-12-18: fixed some minor issues detected by clang static analyzer 9 | |
| - 2019-12-10: core/config bugfix: false error msg when config.enabled="on" is used | |
| When the 'config.enabled="on"' config parameter an invalid error message | |
| was emitted that this parameter is not supported. However, it was still | |
| applied properly. This commit removes the invalid error message. | |
| closes https://github.com/rsyslog/rsyslog/issues/4011 | |
| - 2019-12-03: omsnmp bugfix: "traptype" parameter invalidly rejected value 6 | |
| "Traptype" needs to support values 0 to 6. | |
| However, if value 6(ENTERPRISESPECIFIC) was set, an invalid error message | |
| was emitted. Otherwise processing was correct. | |
| This could lead to problems with automatic config deployment, | |
| as valid configurations were invalidly reported as incorrect. | |
| That in turn could make a deployment fail. | |
| closes https://github.com/rsyslog/rsyslog/issues/3973 | |
| - 2019-12-03: omsnmp: add new parameter "snmpv1dynsource" | |
| If set, the source field from SNMPv1 trap can be overwritten | |
| with a template, default is "%fromhost-ip%". The content should be a | |
| valid IPv4 Address that can be passed to inet_addr(). If the content | |
| is not a valid IPv4 Address, the source will not be set. | |
| closes: https://github.com/rsyslog/rsyslog/issues/3991 | |
| - 2019-12-02: imfile bugfix: state file renaming sometimes did not work properly | |
| Now checking if file-id changes and renaming - cleaning state file | |
| accordingly and always checking and cleaning old inode-only style | |
| state files. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2019-12-02: ratelimit: increase rate limit interval parameter max value | |
| The burst parameter in the ratelimit was increased to an unsigned int | |
| but the interval remained an unsigned short. While it may be unusual, | |
| there is possibly a chance to need to represent an interval longer than | |
| about 3/4 of a day. | |
| While here, go through and normalize all the various incarnations of | |
| rate limiting to be explicitly unsigned int for the burst and interval. | |
| Thanks to github user frikilax for the patch. | |
| - 2019-12-02: ommongodb: Add other supported formats for 'time' and 'date' fields | |
| Thanks to github user frikilax for the patch. | |
| - 2019-12-02: imjournal bugfix: too many messages in error case | |
| Under certain error conditions, `ignorePreviousMessages="on"` could be ignored | |
| an existing messages be processed. | |
| Thanks to github user 3chas3 for the patch. | |
| - 2019-11-27: core bugfix: action on retry mangles messages | |
| When a failed action goes into retry, template content is rendered | |
| invalid if the action uses more than 1 template. | |
| closes https://github.com/rsyslog/rsyslog/issues/3898 | |
| Thanks to Mikko Kortelainen for the patch. | |
| - 2019-11-27: testbench: improve mysql testing support | |
| tests can now run in parallel and are hardened against several glitches | |
| - 2019-11-22: omhttp: add basic support for Loki Rest | |
| Loki is a new message indexer and querier from Grafana Labs. See | |
| https://github.com/grafana/loki for details on Loki. | |
| This change provides the initial message structure to send bulk message | |
| payloads to the Loki Rest endpoint. omhttp, received a new bulk message | |
| format called lokirest. Additionally, the plugin relies on the user to | |
| provide the correct "stream" read message format. | |
| A loki template must be json compatible and include a "stream" key of | |
| key value tags, and a values key of an array of 2 element arrays, where | |
| each 2 element array is the unix epoch in nanoseconds followed by an | |
| unstructured message. | |
| An example: | |
| template(name="array_loki" type="string" string="{\"stream\":{\"host\":\"%HOSTNAME%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"syslogtag\":\"%syslogtag%\"},\"values\": [[ \"%timegenerated:::date-unixtimestamp%000000000\", \"%msg%\" ]]}") | |
| - 2019-11-22: testbench: obtain python binary path via AM_PATH_PYTHON | |
| see also https://github.com/rsyslog/rsyslog/issues/3853 | |
| - 2019-11-22: omprog: detect violation of interface protocol | |
| The spec for the omprog interaction with the program it calls specifies | |
| that the program receives one message via one line. In other words: | |
| it must be a string terminated by LF. | |
| However, omprog does currently rely on a proper template to fulfill this | |
| requirement, If the template does not provide for the LF, it is never | |
| written. For the called program, this looks like it does not receive any | |
| input at all. Even if it finally reads data (e.g. due to full buffer), | |
| it will not properly be able to discern the messages. | |
| This handling is improved with this commit. | |
| We cannot just check the template, because at the end of the template | |
| may by a non-constant value. As such, we do not know at config load | |
| time if there is this problem or not. | |
| So the correct approach is to, during runtime, check if each message | |
| is properly terminated. For those that are not: | |
| * we append a LF, because anything else makes matters worse | |
| * log a warning message, at least for a sample of the messages | |
| The warning is useful in the (expected most often) case that the template | |
| is simply missing the LF. While appending works, it slows down processing. | |
| As such the user should be given a chance to correct the config bug. | |
| To avoid clutter, the warning is emitted at most once every 30 seconds. | |
| This value is hardcoded as we do not envision a need to adjust it. Usually | |
| users should quickly fix the template. | |
| closes https://github.com/rsyslog/rsyslog/issues/3975 | |
| - 2019-11-19: core queue: emit warning if parameters are set for direct queue | |
| Direct queues do not apply queue parameters because they are actually | |
| no physical queue. As such, any parameter set is ignored. This can | |
| lead to unintentional results. | |
| The new code detects this case and warns the user. | |
| closes https://github.com/rsyslog/rsyslog/issues/77 | |
| - 2019-11-19: imjournal bugfix: do not wait too long on recovery try | |
| When trying to recover journal errors, imjournal waited a hardcoded | |
| period of 10s between tries. This was pretty long and could lead to | |
| loss of journal data. | |
| This commit adjust it to 100ms, which should still be fully sufficient | |
| to prevent the journal from "hammering" the CPU. | |
| It may be worth considering to make this setting configurable - but | |
| let's first see if there is real demand to actually do that. | |
| closes https://github.com/rsyslog/rsyslog/issues/3969 | |
| - 2019-11-19: mmutf8fix: enhance handling of incorrect UTF-8 sequences | |
| 1. Invalid utf8 detection didn't handle 3 and 4-byte overlong encodings (2 | |
| byte overlong encodings were handled explicitly by rejection E0 and E1 | |
| start bytes). Unified checks for overlong encodings. | |
| 2. Surrogates U+D800..U+DFFF are not valid codepoints (Unicode Standard, D92) | |
| 3. Replacement of characters in invalid 3 or 4-bytes encodings was too | |
| eager. It must not replace bytes which are valid UTF-8 sequences. For | |
| example, in [0xE0 0xC2 0xA7] sequence the 0xC2 is invalid as a continuation | |
| byte, but it starts a valid UTF8 symbol [0xC2 0xA7]. That is, with current | |
| code processing the sequence will result in "???" but the correct result is "?§" | |
| (provided that the replacement character is "?"). | |
| 4. Various tests for UTF-8 invalid/valid sequences. | |
| Thanks to Sergei Turchanov for the patch. | |
| - 2019-11-14: imfile: add new input parameter escapeLF.replacement | |
| The new parameter permits to specify a replacement to be configured | |
| when "escapeLF" is set to "on". Previously, a fixed replacement string | |
| was used ("#012"/"\n") depending on circumstances. If the parameter is | |
| set to an empty string, the LF is simply discarded. | |
| closes https://github.com/rsyslog/rsyslog/issues/3889 | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.1911.0 (aka 2019.11) 2019-11-12 | |
| - 2019-11-12: core queue: add config param "queue.takeFlowCtlFromMsg" | |
| This is a fine-tuning option which permits to control whether or not | |
| rsyslog shall alays take the flow control setting from the message. If | |
| so, non-primary queues may also block when reaching high water mark. | |
| This permits to add some synchronous processing to rsyslog core engine. | |
| However, it is dangerous, as improper use may make the core engine | |
| stall. As such, enabling this option requires very careful planning | |
| of the rsyslog configuration and deep understanding of the consequences. | |
| Note that the option is applied to individual queues, so a configuration | |
| with a large number of queues can (and must if use) be fine-tuned to | |
| the exact use case. | |
| The rsyslog team strongly recommends to let the option turned off, | |
| which is the default setting. | |
| see also https://github.com/rsyslog/rsyslog/issues/3941 | |
| - 2019-11-12: imrelp: add new config parameter "flowcontrol" | |
| This permits to fine-tune the flowControl parameter. Possible values are | |
| "no", "light", and "full". With light being the default and previously | |
| only value. | |
| Changing the flow control setting may be useful for some rare applications, | |
| but be sure to know exactly what you are doing when changing this setting. | |
| Most importantly, whole rsyslog may block and become unresponsive if you | |
| change flowcontrol to "full". While this may be a desired effect when | |
| intentionally trying to make it most unlikely that rsyslog needs to | |
| lose/discard messages, usually this is not what you want. | |
| see also https://github.com/rsyslog/rsyslog/issues/3941 | |
| - 2019-11-11: imrelp: remove unsafe debug instrumentation | |
| dbgprintf, which is not signal safe, was called from a signal handler | |
| to get better understanding during debugging. While this usually works, | |
| it can occasionally (5%) lead to a hang during shutdown. We have now | |
| removed that debug info as it is no longer vital. | |
| Note: this could only happen during debug runs. Production mode was | |
| not affected. As such, this fix is only relevant to developers. | |
| However, it caused some confusion in the following issue tracker. | |
| see also https://github.com/rsyslog/rsyslog/issues/3941 | |
| - 2019-11-06: ossl driver bugfix: fix wrong OpenSSL Version check | |
| Fix OpenSSL Version check in: | |
| - SetGnutlsPriorityString function in nsd_ossl.c | |
| - initTLS() function tcpflood.c | |
| See https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_VERSION_NUMBER.html | |
| for more. | |
| This bug lead to not enabling some functionality correctly. | |
| Removed "MinProtocol=TLSv1.1" from two testcases because MinProtocol | |
| is only supported by OpenSSl 1.1.0 or higher and was not really | |
| necessary for the testcases. | |
| closes https://github.com/rsyslog/rsyslog/issues/3939 | |
| - 2019-11-05: mmdarwin: Optimizations, new parameters, update to protocol header | |
| - use permanent worker-dependent buffers to avoid malloc/free for each entry | |
| - move socket structures to worker data, remove global mutex | |
| - add log lines for parameters and general workflow | |
| - don't send body if empty/incomplete (see new parameters) | |
| - don't close/reopen socket every time -> let session open or create new every X | |
| entry (see new parameters) | |
| - clean up code | |
| - added 'send_partial', to let mmdarwin send body if not all fields were | |
| retrieved, or not; default false = only send complete bodies | |
| - added 'socket_max_use' to open new session every X packet, useful for | |
| some versions of Darwin (prior to 1.1) | |
| default is 0 = do not open new session/keep only one | |
| - added 'evt_id' to the darwin header (Darwin v1+ compatibility) | |
| Note: mmdarwin is a contributed module | |
| Thanks to github user frikilax for the patch. | |
| - 2019-11-01: mmkubernetes bugfix: improper use of realloc() | |
| could cause problems under extreme memory shortage - very unlikely | |
| credits to LGTM.COM for detecting this | |
| - 2019-10-31: imjournal: set the journal data threshold to MaxMessageSize | |
| When data is read from the journal using sd_journal_get_data it may be | |
| truncated to a certain threshold (64K by default). | |
| If the rsyslog MaxMessageSize is larger than the threshold, there is a | |
| chance rsyslog will receive incomplete messages from the journal. | |
| Empirically, this appears to happen reliably when XZ compression is | |
| used by journald. Systems where journald uses LZ4 compression do not | |
| appear to suffer this issue reliably--if at all. | |
| This change sets the threshold to the MaxMessageSize when the | |
| journal is opened. | |
| Thanks to Robert Winslow Dalpe for the patch. | |
| - 2019-10-30: improg bugfix: allow improg to handle multi-line inputs | |
| miscellaneous bug fixes in improg: | |
| * properly truncate string after an input event is submitted | |
| * set msgoffset to 0. | |
| * tests added to check above fixes | |
| Thanks to Nelson Yen for the fix. | |
| - 2019-10-30: mmdblookup bugfix: missing space in city name | |
| This fixes the issue that spaces in city names are dropped. However, the | |
| fix is more or less a work-around. As it turns out, the libmaxminddb API | |
| is not correctly used. In the somewhat longer term, we should fix this. | |
| see also https://github.com/maxmind/libmaxminddb/issues/218 | |
| closes https://github.com/rsyslog/rsyslog/issues/1650 | |
| - 2019-10-30: core/queue: provide ability to run diskqueue on multiple threads | |
| Up until this release, disk queues could only use a single thread, | |
| what limited their performance with outputs like ElasticSearch. | |
| Now disk queues can utilize multiple threads just like any other | |
| queue type. Most importantly, the disk queue part of a DA queue | |
| now inherits the max number of threads from its memory queue | |
| counterpart. | |
| NOTE: the new multi-threaded DA disk queue is actually a change of | |
| behavior. We have not guarded it by a new config switch as we | |
| assume the new behavior is most often exactly within user | |
| expectations. In any case, we cannot see any harm from running | |
| the disk queue on multiple threads. | |
| see also https://github.com/rsyslog/rsyslog/issues/3543 | |
| closes https://github.com/rsyslog/rsyslog/issues/3833 | |
| - 2019-10-25: omfile bugfix: file handle leak | |
| The stream class does not close re-opened file descriptors. | |
| This lead to leaking file handles and ultimately to the inability | |
| to open any files/sockets/etc as rsyslog ran out of handles. | |
| The bug was depending on timing. This involved different OS | |
| thread scheduler timing as well as workload. The bug was more | |
| common under the following conditions: | |
| - async writing of files | |
| - dynafiles | |
| - not committing file data at end of transaction | |
| However it could be triggered under other conditions as well. | |
| The refactoring done in 8.1908 increased the likelihood of | |
| experiencing this bug. But it was not a real regression, the new | |
| code was valid, but changed the timing so that the race was more | |
| likely. | |
| Thanks to Michael Biebl for reporting this bug and helping to | |
| analyze it. | |
| closes https://github.com/rsyslog/rsyslog/issues/3885 | |
| - 2019-10-22: imfile bugfix: improper use of calloc() | |
| could cause problems under extreme memory shortage - very unlikely | |
| credits to LGTM.COM for detecting this | |
| - 2019-10-22: TLS driver bugfix: improper use of calloc() | |
| can cause problems under extreme memory shortage - very unlikely | |
| credits to LGTM.COM for detecting this | |
| - 2019-10-22: imuxsock bugfix: improper use of calloc() | |
| can cause problems under extreme memory shortage - very unlikely | |
| credits to LGTM.COM for detecting this | |
| - 2019-10-17: build system bugfix: incorrect default in ./configure help text | |
| closes https://github.com/rsyslog/rsyslog/issues/3904 | |
| Thanks to Michael Biebl for pointing this out. | |
| - 2019-10-17: mmkubernetes bugfix: improper use of calloc() | |
| can cause problems under extreme memory shortage - very unlikely | |
| credits to LGTM.COM for detecting this | |
| - 2019-10-16: core queue bugfix: propagate batch size to DA queue | |
| This was a long-standing bug where the DA queue always had a fixed small batch | |
| size because the setting was not propagated from the memory queue. This also | |
| removes a needless and counter-productive "debug aid" which seemed to be in | |
| the code for quite some while. It did not cause harm because of the batch | |
| size issue. | |
| - 2019-10-16: testbench: fix unreliable gzipwrite test | |
| The test was timing-sensitive as we did not properly check all data | |
| was output to the output file - we just relied on sleep periods. | |
| This has been changed. Also, we made some changes to the testing | |
| framework to fully support sequence checking of multiple ZIP files. | |
| - 2019-10-16: core queue bugfix: handle multi-queue-file delete correctly | |
| Rsyslog may leave some dangling disk queue files under the following | |
| conditions: | |
| - batch sizes and/or messages are large | |
| - queue files are comparatively small | |
| - a batch spans more than two queue files (from n to n+m with m>1) | |
| In this case, queue files n+1 to (n+m-1) are not deleted. This can | |
| lead to problems when the queue is re-opened again. In extreme cases | |
| this can also lead to stalled processing when the max disk space is | |
| used up by such left-over queue files. | |
| Using defaults this scenario is very unlikely, but it can happen, | |
| especially when large messages are being processed. | |
| - 2019-10-16: imjournal: fix regression from yesterday's patch | |
| commit 78976a9bc059 introduced a regression that caused writing | |
| the journal state file to fail. This happens when the state file | |
| is given as relative file name and the working directory is also | |
| a relative path. This situation is very uncommon. So most deployments | |
| will never experience it. We discovered the issue during CI runs | |
| where the trigger condition is given. Note that it also takes | |
| multiple times of loading the journal to actually see the bug. | |
| see also https://github.com/rsyslog/rsyslog/pull/3878 | |
| - 2019-10-15: imjournal plugin code restructuring, added remote option | |
| Decomposed ReadJournal() a bit, also now coupling journald | |
| variables in one struct, added few warning messages and debug | |
| prints to help with bug hunts in future, also got rid of two | |
| needless journald calls. WorkAroundJournalBug now deprecated. | |
| Added option to pull journald records from outside local machine. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2019-10-11: core bugfix: potential abort on very long action name | |
| The action name is stored in modified form for the debug header and | |
| some messages. If it is extremely long, a buffer can be overrun, | |
| resulting in misaddressing and potential segfault for rsyslog. This | |
| can also happen if the action is NOT named, but a custom path to | |
| the output module is given and that path is very long. This triggers | |
| the same issue because by default the module load path is included | |
| in the action name. | |
| This patch corrects the problem and truncates overly long names | |
| when being used for name generation. | |
| The problem was detected during testbench work. We did never receive | |
| a bug report from practice. | |
| - 2019-10-10: testbench: add test for mmpstrucdata with RFC5424 escape sequences | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.1910.0 (aka 2019.10) 2019-10-01 | |
| - 2019-10-01: core bugfix: incorrect error message on duplicate module load | |
| A Null-pointer was passed to printf instead of the module name. | |
| On some platforms this may lead to a segfault. On most platforms | |
| printf check's for NULL pointers and uses the string "(null)" | |
| instead. In any case, the module name is missing from the error message. | |
| - 2019-10-01: imczmq nitfix: potential NULL ptr in printf on out-of-memory condition | |
| very unlikely to happen but if it does without any real issue on most platforms. | |
| - 2019-10-01: work around some compiler warning messages induced by pthreads API | |
| - 2019-10-01: core ratelimiting: more verbose message when rate-limiting happens | |
| When messages are rate-limited, the error message now also contains the | |
| rate limiter setting. This enables the user to more quickly understand what | |
| the problem is (especially if default values apply). | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2019-10-01: openssl TLS driver: do not emit unnecessary error message | |
| On older openssl versions, an API was missing to set user-defined parameters. If we | |
| had such an older version, rsyslog emitted an error message even if the user did | |
| not configure such parameters. This has been corrected, so that a message is only | |
| emitted if there really is a problem. Based on user feedback the severity has also | |
| been downgraded to "warning". | |
| - 2019-10-01: pmcisconames (contributed module) bugfix: potential misaddressing | |
| - 2019-09-30: pmaixforwardedfrom (contributed module) bugfix: potential misaddressing | |
| - 2019-09-30: pmdb2diag (contributed module) bugfix: Out of bounds issue | |
| Add a new sanity check after determining the level len. | |
| Thanks to Philippe Duveau for the patch. | |
| see also: https://nvd.nist.gov/vuln/detail/CVE-2019-17040 | |
| - 2019-09-02: ability to set stricter TLS operation modes | |
| - checking of extendedKeyUsage certificate field | |
| - stricter checking of certificate name/addresses | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2019-08-21: testbench: add basic test for immark | |
| - 2019-08-20: core: do not unnecessarily set hostname on each HUP | |
| - 2019-08-20: build system: support cross-platform build for mysql/mariadb | |
| rsyslog fails to cross build from source, because it uses mysql_config | |
| and mysql_config is unfixably broken for cross compilation. It would be | |
| better to use pkg-config. The attached patch makes rsyslog try | |
| pkg-config first and fall back to mysql_config. | |
| Thanks to Helmut Grohne for providing a base patch. | |
| - 2019-08-20: core/tcpsrv: potential race on startup/shutdown | |
| if the tcpsrv component is started and quickly terminated, it may hang | |
| for a short period of time. Also a very small amount of memory is leaked | |
| immediately before shutdown. While this leak is irrelevant in practice | |
| (the OS clean up the process anyways), it leads to CI failures. The hang, | |
| however, can lead to longer than expected shutdown times for rsyslog. | |
| The problem can be experienced via imtcp, imgssapi and imdiag (users | |
| of affected core component). | |
| ---------------------------------------------------------------------------------------- | |
| Scheduled Release 8.1908.0 (aka 2019.08) 2019-08-20 | |
| - 2019-08-19: testbench: add test for $allowedSender functionality | |
| - 2019-08-19: testbench: harden some tests against very slow CI machines | |
| - 2019-08-16: testbench: make most tests use a port file and assign listen port 0 | |
| This makes the test much more robust against heavily loaded test systems. | |
| - 2019-08-16: core/action: guard action.externalstate.file content against whitespace | |
| remove trailing whitespace before checking the status string. This is | |
| most important as a line usually ends with \n, which is considered | |
| trailing whitespace. Accepting this increases usability. | |
| - 2019-08-16: imtcp bugfix: multiple listenerPortFile parameter did not work | |
| ... because they were treated as module-global. If we had multiple imtcp | |
| listeners with multiple port files, only the last filename was always used. | |
| closes https://github.com/rsyslog/rsyslog/issues/3817 | |
| - 2019-08-16: testbench: improve testbench plumbing for gzip and fail cases | |
| We have added new capabilities to the testbench plumbing to automatically | |
| deal with gzip-compressed files. This also permits to use the wait_seq_check | |
| function to work for gzip tests as well. The known-timing-sensitive | |
| gzipwr_large test now makes use of the new capabilities. This enables us | |
| to more reliably detect when we can savely shutdown the tested instance. | |
| This commit also adds an ability to "abort" the full testbench run on | |
| first test failure. This is especially useful during CI. | |
| - 2019-08-13: testbench: add test for imuxsock legacy format | |
| This was never tested. Ensures we don't accidentally break existing | |
| configurations. | |
| - 2019-08-13: omelasticsearch bugfix: segfault on unknown retryRuleset | |
| omelasticsearch does some "interesting tricks" for an output module. | |
| This causes a segfault if the retryRuleset is now known. | |
| The action module interface currently expects that all config errors | |
| be detected during instance creation. Instead omelasticsearch defers | |
| the retry ruleset check to a later state. The reason is that it wants | |
| to support the use the same rulesetname it is defined in - and this | |
| is not yet available at action parsing. | |
| We fix this by ensuring that any deleted instance is properly unlinked | |
| from the instance list. One may argue the module interface should get | |
| upgrade for such cases, but this is a longer-term approach. | |
| closes https://github.com/rsyslog/rsyslog/pull/3796 | |
| - 2019-08-12: imptcp bugfix: port="0" parameter did not work as expected | |
| when multiple interfaces and/or protocols could be bound, each of | |
| them used a different listener ports were assigned. While this is | |
| basically correct, it makes things unusable, especially as | |
| listenPortFileName will only contain the port number used for | |
| the latest listener. | |
| This patch now follows the model of nsd_ptcp.c to assign only | |
| the first port randomly and then use that port consistently. | |
| - 2019-08-10: omelasticsearch bugfix: potential resource leak with "rebindinterval" | |
| If the "rebindInterval" parameter was used connections could be linked. This | |
| was especially the case with small intervals (such as "2"). This is fixed by | |
| forcing libcurl to close the connection on rebind. | |
| Thanks to Noriko Hosoi for providing the patch. | |
| - 2019-08-10: imjournal bugfix: state file close with fsync() was incorrect | |
| This lead to fsync() now always applied where expected. | |
| Thanks to Jiri Vymazal for the patch. | |
| - 2019-08-10: testbench: add addtl test for multithreading and HUP | |
| - 2019-08-10: imptcp bugfix: received bytes counter improperly maintained | |
| imptcp counts the number of bytes received. However, receives | |
| happen on different worker thread. The access to the counter | |
| was not synchronized, which can cause loss of updates. Also, | |
| thread debuggers validly flag this as an error, which creates | |
| problems under CI. | |
| This commit fixes the situation via atomic operations and | |
| falls back to mutex calls if they are not available. | |
| Detected by LLVM thread sanitizer. | |
| closes https://github.com/rsyslog/rsyslog/issues/3798 | |
| - 2019-08-07: testbench: add basic tests for omusrmsg | |
| - 2019-08-05: omhttp bugfix: enable checkpath configuration parameter | |
| omhttp, 'checkpath' option, was not configurable in the past. | |
| - add 'checkpath' to the cnfparamdescr table. | |
| - fix issue with checkpath passing extra garbage characters in string. | |
| - add 'checkpath' into unit test - omhttp-retry.sh | |
| Thanks to Nelson Yen for the fix. | |
| - 2019-08-05: testbench bugfix: some tests were executed when req module was missing | |
| In actual case if --enable-impstats was not given some other tests failed. | |
| - 2019-08-03: iminternal bugfix: race on termination | |
| This could in theory lead to loss of shutdown messages, but was mostly a | |
| cosmetic issues. We primarily fixed it to get TSAN-clean so that we can | |
| utilize LLVM TSAN in CI. | |
| - 2019-08-02: testbench: new test for omfile outchannel functionality | |
| - 2019-08-02: core/janitor bugfix: properly maintain dynafile cache | |
| When the janitor cleans out timed-out files, it does not | |
| properly indicate the entry is gone. Especially when running | |
| in async mode this can lead to use-after-free and thus | |
| memory corruption or segfault. | |
| see also https://github.com/rsyslog/rsyslog/issues/3756 | |
| - 2019-08-01: omfile bugfix: race file when async writing is enabled | |
| This seems to be a long-standing bug, introduced around 7 years ago. | |
| It became more visible by properly closing files during HUP, which | |
| was done in 8.1905.0 (and was another bugfix). Note that due to this | |
| race a memory corruption can occur under bad circumstances. As such, | |
| this may have also caused segfaults or system hangs (mutexes could | |
| have been affected). | |
| closes https://github.com/rsyslog/rsyslog/issues/3772 | |
| - 2019-08-01: testbench: additional tests for HUP | |
| - 2019-07-31: imrelp bugfix: hang after HUP | |
| termination condition was not properly checked; this lead to | |
| premature termination after patch 1c8712415b9 was applied. | |
| It is open to debate if patch 1c8712415b9 changed the module | |
| interface. Actually it looks like this was previously not | |
| well thought out. | |
| closes https://github.com/rsyslog/rsyslog/issues/3760 | |
| - 2019-07-24: mmdarwin: add new module | |
| This is a contributed module. For details see doc. | |
| Thanks to the Advens team for contributing it. | |
| - 2019-07-23 iminternal bugfix: suppress mutex double-unlock | |
| If there is a burst of log messages during a time when rsyslog is unable | |
| to output (either during log rotation, an out-of-space condition, or | |
| some other similar condition), rsyslog can SEGFAULT due to a mutex | |
| double-unlock. | |
| - 2019-07-23 imtcp: enable listenPortFileName parameter | |
| this parameter was added, but it had no effect as it was not | |
| passed down to the driver layer. This has been fixed. That also | |
| now enables us to use dynamically-assigned port, which are | |
| very useful for further testbench stabilization. Quite some | |
| false positives occurred because the pre-selected port was | |
| already in use again when rsyslog started. | |
| - 2019-07-19 imtcp: enable listenPortFileName parameter | |
| this parameter was added, but it had no effect as it was not | |
| passed down to the driver layer. This has been fixed. That also | |
| now enables us to use dynamically-assigned port, which are | |
| very useful for further testbench stabilization. Quite some | |
| false positives occurred because the pre-selected port was | |
| already in use again when rsyslog started. | |
| - 2019-07-18 core/action: no error file written if act suspended on TX commit | |
| when an action was already disabled while the action was tried to be | |
| committed, no error file was written. Note that this state is highly | |
| unlikely to happen. Most probably, it can only happen if parameter | |
| action.externalstate.file is used. | |
| ---------------------------------------------------------------------------------------- | |
| Version 8.1907.0 (aka 2019.07) 2019-07-09 | |
| NOTE TO MAINTAINERS: libee is not used by rsyslog for quite some while. | |
| However, we never included this info into the changelog. So if you still | |
| make rsyslog depend on libee (some do this), you should stop doing so now. | |
| Libee is dead and no longer been maintained nor hosted by us. Old versions | |
| can still be found at github for those in need. | |
| GENERAL NOTE: during 8.1907 scheduled release timeframe we changed the ChangeLog | |
| format to include the date a change went into master branch. This is to provide | |
| an easy way to identify which changes went into the respective daily stable. | |
| - 2019-07-05 imuxsock: support FreeBSD 12 out of the box | |
| FreeBSD 12 uses RFC5424 on the system log socket by default. This | |
| format is not supported by the special parser used in imuxsock. | |
| Thus for FreeBSD the default needs to be changed to use the | |
| regular parser chain by default. That is all this commit does. | |
| closes https://github.com/rsyslog/rsyslog/issues/3694 | |
| - 2019-07-05 function bugfix: "ipv42num" misspelled as "ip42mum" (without "v") | |
| To fix the issue but keep compatible with existing deployments | |
| both function names are now supported. | |
| closes https://github.com/rsyslog/rsyslog/issues/3676 | |
| - 2019-07-04 fix leading double space in rsyslog startup messages | |
| see also https://github.com/rsyslog/rsyslog/issues/2979 | |
| - omamqp1: port to latest api, add tests | |
| This brings omamqp1 up-to-date with the latest qpid-proton-c | |
| api version. This also adds a test for the plugin, to test | |
| the basic functionality. The test requires the user to | |
| install qdrouterd and the python qpid-proton library in order | |
| to use the simple_recv.py test program. | |
| Thanks to Richard Megginson for the patch. | |
| - omclickhouse bugfix: potential segfault on omclickhouse batchmode | |
| segfault happened when the template did not contain the string | |
| "VALUES". | |
| Thanks to github user wdjwxh for the fix. | |
| - core bugfix: message duplication copied incorrect timestamp | |
| MsgDup() placed timereported into timegenerated property, resulting | |
| in invalid property values. Original timegenerated was lost. This | |
| occured always when a message needed to be duplicated. Most | |
| importantly this is the case when queues are used. | |
| closes https://github.com/rsyslog/rsyslog/issues/3716 | |
| - core bugfix: segfault on startup depending on queue file names | |
| rsyslog will segfault on startup when a main queue file name has | |
| been set and at least on other queue contains a file name. This | |
| was cased by too-early freeing config error-detection data | |
| structures. It is a regression caused by commit e22fb205a3. | |
| Thanks to Wade Simmons for reporting this issue and providing | |
| detailed analysis. That greatly helps fixing it quickly. | |
| closes https://github.com/rsyslog/rsyslog/issues/3681 | |
| - core "bugfix": alignment issue | |
| This was not a hard error on current platforms, but a | |
| to-be-considered compiler warning regarding invalid alignment. | |
| While it works well on current platforms, alignment issues may | |
| turn into real issues in future platforms. So we try to fix them | |
| if possible. As not only a side-effect this resolves compiler | |
| warnings even on current platforms. | |
| This fix has some regression potential. If so, the problems | |
| may occur during IP address resolution. | |
| see also https://github.com/rsyslog/rsyslog/issues/2608 | |
| - omfile bugfix: potential hang/segfault on HUP of dynafile action | |
| when omfile was HUPed it did not sufficiently clear all dynafile | |
| cache maintenance data structures. This usually lead to misaddressing | |
| and could result in various issues, including a hang of rsyslog | |
| processing or segfaults. It could also have "no effect" by pure | |
| luck of not hitting anything important. This actually seems to | |
| have been the most frequent case. | |
| This seems to be a long-standing bug, but the likelihood of its | |
| appearance seems to have been increased by commit 62fbef7 | |
| introduced in 8.1905. Note: the commit itself has no regression, | |
| just increases the likelihood to trigger the pre-existing bug. | |
| special thanks to Alexandre Guédon for his help in analyzing | |
| the issue - without him, we would probably still not know | |
| what actually went wrong. | |
| closes https://github.com/rsyslog/rsyslog/issues/3686 | |
| - imjournal bugfix: potential message duplication | |
| When journal was preloaded from previously saved cursor it was not advanced | |
| to next entry so reading begun from last message which was therefore | |
| duplicated. | |
| Thanks to Jiri Vymazal for the patch. | |
| - rfc5424 parser bugfix: leading space sometimes lost | |
| if structured data is present a leading space in MSG field is lost | |
| - queue subsystem bugfix: oversize queue warning message shown as error | |
| The warning message was emitted as an error message, which is misleading | |
| and may also break some automated procedures. | |
| - core bugfix: HUP did not work reliable on all platforms | |
| most notably not on FreeBSD, maybe others. The reason was obviously | |
| different handling of signals in respect to multiple threads. | |
| - build system bugfix: missing files in distribution tarball | |
| - testbench | |
| * fixed "make distcheck" settings which were missing some modules | |
| This lead to incomplete "make distcheck" run; some errors were not | |
| detected due to that. | |
| * testbench framework: use ip tool instead of outdated ifconfig | |
| The framework now first checks if "ip" is available and falls back | |
| to "ifconfig" only if this is not the case. | |
| Thanks to Michael Biebl for the suggestion. | |
| closes https://github.com/rsyslog/rsyslog/issues/3682 | |
| ------------------------------------------------------------------------------ | |
| Version 8.1905.0 (aka 2019.05) 2019-05-28 | |
| - templates: add datatype template option for JSON generation | |
| The new "datatype" and "onEmpty" template options permits to | |
| generate non-string data rather easily. It works together with | |
| jsonf formatting, which is what people should use nowadays. | |
| closes https://github.com/rsyslog/rsyslog/issues/2827 | |
| - config processing: check disk queue file is unique | |
| If the same name is specified for multiple queues, the queue files | |
| will become corrupted. This commit adds a check during config parsing. | |
| If duplicate names are detected the config parser errors out and the | |
| related object is not created. | |
| Note: this may look to a change-of-behavior to some users. However, | |
| this never worked and it was pure luck that these users did not run | |
| into big problems (e.g. DA queues were never going to disk at the | |
| same time). So it is acceptable to error out in this hard error case. | |
| closes https://github.com/rsyslog/rsyslog/issues/1385 | |
| - global config: new parameters for ruleset queue defaults | |
| specifically: | |
| * default.ruleset.queue.timeoutshutdown | |
| * default.ruleset.queue.timeoutactioncompletion | |
| * default.ruleset.queue.timeoutenqueue | |
| * default.ruleset.queue.timeoutworkerthreadshutdown | |
| closes https://github.com/rsyslog/rsyslog/issues/3656 | |
| - add capability to write full config file (-o cmdline option) | |
| Introduces the capability to create an output config file that explodes | |
| all "includes" into a single file. This provides a much better overview | |
| of how exactly the configuration is crafted. That could often be a great | |
| troubleshooting aid. | |
| This commit also contains some slight not-really-related cleanup. | |
| closes https://github.com/rsyslog/rsyslog/issues/3634 | |
| - queue subsystem: permit to disable "light delay mark" | |
| New semantic: if lightDelayMark is 0, it is set to the max queue | |
| size, effectively disabling the "light delay" functionality. | |
| Thanks to Yury Bushmelev to mentioning issues related to light | |
| delay mark and proposing the solution (which actually is what | |
| this commit does). | |
| closes https://github.com/rsyslog/rsyslog/issues/1778 | |
| - queue subsystem: provide better user status messages | |
| The queue subsystem now provides additional information messages which | |
| may help a regular user to maintain system health. Most importantly, | |
| DA queues now output when they persist queue data at end of run and | |
| when they restart the queue based on persisted data. | |
| - core: emit a warning message for ultra-large queue size definitions | |
| We see error reports from users who have configured excessively large queues | |
| and receive an OOM condition or other problems. | |
| With that patch we generate a warning message if a queue is configured very | |
| large. "Very large" is defined to be in excess of 500000 messages. | |
| see also https://github.com/rsyslog/rsyslog/issues/3314 | |
| closes https://github.com/rsyslog/rsyslog/issues/3334 | |
| - new global config parameter "internalmsg.severity" | |
| permits to specify a severity filter for internal message. Only | |
| messages with this severity level or more severe are logged. | |
| Originally this was done in rsyslog.conf as usual: you can filter | |
| rsyslog messages on severity, just like any other. But with systemd, | |
| we now emit primarily to the journal, and this is outside of rsyslog's | |
| rule engine and so regular filters do not apply (at least in regard | |
| to the journal). Logging to journal is good, because finally | |
| folks begin to see the messages (traditional distro configs discard | |
| them, for whatever is the reason). | |
| This commit implements a global setting for a severity-based filter | |
| for internal messages, before submitted to journal. So it's not 100% | |
| of what rsyslog can do, but at least some way to customize. | |
| see also https://github.com/rsyslog/rsyslog/issues/3639 | |
| - config processing bugfix: error messages if config.enabled="off" is used | |
| Using config.enabled="off" could lead to error messages on | |
| "parameter xxx not known", which were invalid. They occured | |
| because the config handler expected them to be used, which | |
| was not the case due to being disabled. | |
| This commit fixes that issue. | |
| closes https://github.com/rsyslog/rsyslog/issues/2520 | |
| - core portability bugfix: harden shutdown processing on FreeBSD | |
| On FreeBSD, rsyslog does not always terminate immediately on SIGTERM. | |
| Root cause seems to be that SIGTERM is delivered differently under | |
| FreeBSD. This causes the main thread to not be awaken, and so it | |
| takes until the next janitor interval to come back to life - which | |
| can be far too long. Fixed this bug explicitly awaking the main | |
| thread. | |
| - imtcp bugfix: oversize message truncation causes log to be garbled | |
| The actual problem is in the tcpserver component. However, the prime user | |
| is imtcp and so users will likely experience this as imtcp problem. | |
| When a too-long message is truncated, the byte after the truncation | |
| position becomes the first byte of the next message. This will garble | |
| the next messages and in almost all cases render it is syslog-noncompliant. | |
| The same problem does NOT occur when the message is split. | |
| This commit fixes the issue. It also includes a testbench fix. | |
| Unfortunately the test for exactly this feature was not properly | |
| crafted and so could not detect the problem. | |
| closes https://github.com/rsyslog/rsyslog/issues/3580 | |
| - omfile bugfix: FlushOnTXEnd does not work reliably with dynafiles | |
| The flush was only done to the last dynafile in use at end of | |
| transactions. Dynafiles that were also modified during the | |
| transaction were not flushed. | |
| Special thanks to Duy Nguyen for pointing us to the bug and | |
| suggesting a solution. | |
| This commit also contains a bit of cosmetic cleanup inside | |
| the file stream class. | |
| closes https://github.com/rsyslog/rsyslog/issues/2502 | |
| - lmcry_gcry build bugfix: was not always properly build | |
| Due to an invalid definition in build system this seems to have not | |
| been correctly build on at least some platforms (but it worked on | |
| others as it passed CI testing). This has now been corrected. | |
| Thanks to Remi Locherer for the patch. | |
| - dnscache bugfix: very unlikely memory leak | |
| This fixes a memory leak that can only occur under OOM conditions. | |
| Detected by Coverity Scan, CID 203717 | |
| - testbench bugfix: wrong parameter check in diag.sh (tcpflood()) | |
| When first parameter is check_only, the tcpflood funtion shall not | |
| abort the test itself (The fail is intended if this option is set). | |
| closes issue #3625 | |
| - testbench bugfix: imfile-symlink test failed w/ parallel test run | |
| The test sometimes failed. It used a symlink to a hardcoded name | |
| rsyslog-link.*.log. This symlink was created but then disappears. | |
| The reason is that upon (every!) test exit, rsyslog-link.*.log is | |
| deleted. So a parallel test running the exit procedure just at the | |
| "right" time can removed that file. | |
| The bug is that the file name should be created using the tests's | |
| dynamic name. This is done now. | |
| closes https://github.com/rsyslog/rsyslog/issues/3550 | |
| ------------------------------------------------------------------------------ | |
| Version 8.1904.0 (aka 2019.04) 2019-04-16 | |
| - omfile: provide more helpful error message on file write errors | |
| now contains actual file name plus a link to probable causes for this type | |
| of problem | |
| - imfile: emit error on startup if no working directory is set | |
| When the work directory has not been set or is invalid, state files | |
| are created in the root of the file system. This is neither expected | |
| nor desirable. We now complain loudly about this fact. For backwards | |
| compatibility reasons, we still need to support running imfile in | |
| this case. | |
| closes https://github.com/rsyslog/rsyslog/issues/1296 | |
| - dnscache: add global parameter dnscache.default.ttl | |
| This permits to control default TTL for cache entries. If set | |
| to 0, the DNS cache is effectively disabled. | |
| closes https://github.com/rsyslog/rsyslog/issues/49 | |
| closes https://github.com/rsyslog/rsyslog/issues/1487 | |
| - omelasticsearch: new parameter rebindinterval | |
| Thanks to Richard Megginson for the patch. | |
| - omelasticsearch: new parameter skipverifyhost | |
| Add ability to specify the libcurl CURLOPT_SSL_VERIFYHOST | |
| option to skip verification of the hostname in the peer cert. | |
| WARNING: This option is insecure, and should only be used | |
| for testing. The default value is off, meaning, the hostname | |
| will be verified by default. | |
| Thanks to Richard Megginson for the patch. | |
| - omelasticsearch: set rawmsg to data from original request | |
| Previously, when constructing the message to submit for a retry | |
| for an original request, if the original request did not contain | |
| the field `message`, the system property `rawmsg` was set to | |
| the entire metadata + data from the original request. This was | |
| causing problems with Elasticsearch. This patch changes | |
| the code so that the `rawmsg` will be set to only the data part | |
| of the original request if there is no `message` field. | |
| closes https://github.com/rsyslog/rsyslog/issues/3573 | |
| Thanks to Richard Megginson for the patch. | |
| - mmkubernetes - support for metadata cache expiration | |
| New parameters for mmkubernetes (module and action): | |
| * `cacheexpireinterval` | |
| If `cacheexpireinterval` is -1, then do not check for cache expiration. | |
| If `cacheexpireinterval` is 0, then check for cache expiration. | |
| If `cacheexpireinterval` is greater than 0, check for cache expiration | |
| if the last time we checked was more than this many seconds ago. | |
| * `cacheentryttl` - maximum age in seconds for cache entries | |
| New statistics counters: | |
| * `podcachenumentries` - the number of entries in the pod metadata cache. | |
| * `namespacecachenumentries` - the number of entries in the namespace | |
| metadata cache. | |
| * `podcachehits` - the number of times a requested entry was found in the | |
| pod metadata cache. | |
| * `namespacecachehits` - the number of times a requested entry was found | |
| in the namespace metadata cache. | |
| * `podcachemisses` - the number of times a requested entry was not found | |
| in the pod metadata cache, and had to be requested from Kubernetes. | |
| * `namespacecachemisses` - the number of times a requested entry was not | |
| found in the namespace metadata cache, and had to be requested from | |
| Kubernetes. | |
| - imdocker: new contributed module | |
| imdocker will get (docker) container logs from a host as well as filling | |
| out some basic container metadata as id, name, image, labels. | |
| Thanks to Nelson Yen for the contribution. | |
| - mmtaghostname: new contributed module | |
| This module allows to force hostname after parsing to the localhostname of | |
| rsyslog and/or add a tag to messages received from input modules without | |
| tag parameter. | |
| Thanks to Philippe Duveau for the contribution. | |
| - imbatchreport: new contributed input module | |
| This input module manage batches' reports : complete file as a single log. | |
| Thanks to Philippe Duveau for the contribution. | |
| - imtuxedolog: new contributed input module for Tuxedo ULOG | |
| Thanks to Philippe Duveau for the contribution. | |
| - openssl network driver: Added support setting openssl configcommands | |
| We are using the gnutlsPriorityString setting variable, to pass | |
| configuration commands to openssl. | |
| closes: https://github.com/rsyslog/rsyslog/issues/3605 | |
| - omkafka: drop messages rejected due to being too large | |
| Drop messages that were rejected due to | |
| 'RD_KAFKA_RESP_ERR_MSG_SIZE_TOO_LARGE' error | |
| Thanks to Nelson Yen for the patch | |
| - core/action: implement capability to resume/suspend via external file | |
| It has been reported that some TCP receivers exists that accept syslog tcp | |
| messages at any rate, even if they do not manage to actually process them. | |
| Instead, they silently drop the message. This behavior is not configurable. | |
| All in all, it can lead to considerate message loss. | |
| To support such use cases, we need to provide an ability to externally | |
| trigger actions suspension and resumption. | |
| We do this via a configured file which contains the status of the action. | |
| Rsyslog periodically reads the file and if it contains "SUSPEND", it | |
| suspend the action (and likewise for resume). | |
| closes https://github.com/rsyslog/rsyslog/issues/2924 | |
| - improg bugfix: some memory leaks | |
| Thanks to Philippe Duveau for the contribution. | |
| - msg object bugfix: regression from 1255a67 | |
| closes https://github.com/rsyslog/rsyslog/issues/3570 | |
| - pmnormalize: fix memory leaks, improve tests | |
| This patch fixes a set of problems plus provides more and enhanced | |
| tests for the module. | |
| Most important problem was a memory leak that occured when a message | |
| could not be passed at all. For each message that could not be parsed | |
| memory of at least the size the message is leaked. Depending on | |
| traffic pattern this can quickly lead to OOM. Note, however, that | |
| this leak was never reported - it was discovered as part of code | |
| review. | |
| closes https://github.com/rsyslog/rsyslog/issues/2007 | |
| - omkafka bugfix: build failure due to inconsistent type | |
| fails depending on platform and settings; was somehow undetected by CI | |
| - imjournal bugfix: potential segfault on some API failure returns | |
| In one case there was possibility of free()'d value of journal | |
| cursor not being reset, causing double-free and crash later on. | |
| closes https://github.com/rsyslog/rsyslog/issues/3537 | |
| - openssl subsystem bugfix: better error handling | |
| Handling of SSL_ERROR_SYSCALL has been hardened. | |
| Handling for SSL_Shutdown errors has been corrected. | |
| Also fixed SSL Shutdown handling in tcpflood (openssl code). | |
| If SSL_Shutdown returns error, we call SSL_read as described in | |
| the documentation to do a bidirectional shutdown. | |
| Closes https://github.com/rsyslog/rsyslog/issues/3561 | |
| - imjournal bugfix: Fetching journal cursor only for valid journal | |
| The sd_journal_get_cursor() got called regardless of previous | |
| retcodes from other journal calls which flooded logs with journald | |
| errors. Now skipping the call in case of previous journal call | |
| non-zero result. Fixed success checking of get_cursor() call | |
| to eliminate double-free possibility. | |
| Also, making WorkAroundJournalBug true by default, as there were no | |
| confirmed performance regressions for a quite long time. | |
| Thanks to Jiri Vymazal for the patch. | |
| - omamqp: fix build errors | |
| They occur on some, newer, platforms. We do not really fix them, but rather | |
| make the compiler ignore them. This is not really good, but the module is | |
| contributed and so that's for now the best thing we can do. | |
| - testbench: change manytcp.sh to use a larger connection count again | |
| not sure why it was reduced, maybe related to | |
| https://github.com/rsyslog/rsyslog/issues/1108 | |
| also, modernize this and another test | |
| - tcpflood bugfix: make soft connection limit work again | |
| It looks like the soft limit became defunct when tcpflood was enhanced to | |
| request more open file handles from OS. | |
| closes https://github.com/rsyslog/rsyslog/issues/1108 | |
| - testbench bugfix: omhttp tests were not run during "make distcheck" | |
| - build system bugfix: omhttp test files were not included in dist tarball | |
| Thanks to Thomas D. (whissi) for the patch. | |
| ------------------------------------------------------------------------------ | |
| Version 8.1903.0 (aka 2019.03) 2019-03-05 | |
| - omrabbitmq: add features (RabbitMQ HA management, templatize routing_key, | |
| populate amqp message headers, delivery_mode and expiration parameters) | |
| - improg: create input module to use external program as input datas | |
| - imtuxedoulog: create input module to consume Tuxedo ULOG files | |
| - omhttp: rewritten with large feature enhancements | |
| Many thanks to Gabriel Intrator for this work. Gabriel also has adopted the | |
| module and plans to support it in the future. | |
| - pmdb2diag: create parser module for DB2 diag logs | |
| - TLS subsystem: add support for certless communication | |
| both openssl and GnuTLS drivers have been updated to support certless | |
| communications. In this case e.g. Diffie-Helman is used. | |
| NOTE: this is an insecure mode, as it does NOT guard against | |
| man-in-the-middle attacks. We implemented it because of the large demand, | |
| not because we think it makes sense to use this mode. We strongly recommend | |
| against it. | |
| closes https://github.com/rsyslog/rsyslog/issues/1068 | |
| - imrelp/omrelp: add capability to specify tlslib for librelp | |
| closes https://github.com/rsyslog/rsyslog/issues/3451 | |
| - build system: introduce a better way to handle compiler pragmas | |
| we now use macros and _Pragma(). This requires less code lines and is more | |
| portable. | |
| - omkafka: add support for dynamic keys | |
| A new configuration property "dynaKey" is added that, when "on", changes the | |
| value of property "key" to a template names instead of a constant value. | |
| This is similar in approach to the DynaTopic implementation. | |
| Thanks to Ludo Brands for the patch. | |
| - AIX port: add AIX linking extensions on many plugins and contributions to | |
| allow building them on this os. | |
| - template: add Time-Related System Property $wday which is the day of week | |
| This allow to get a week based rotation of log as AIX does. | |
| - ksi subsystem: add high availability mode | |
| Note: ksi subsystem now REQUIRES libksi 3.19.0 or above | |
| Thanks to Allan Park for the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/3338 | |
| - imfile bugfix: file reader could get stuck | |
| State file handling was invalid. When a file was moved and re-created | |
| rsyslog could use the file_id if the new file to write the old files' | |
| state file. This could make the file reader stuck until it reached the | |
| previous offset. Depending on file sizes this could never happen AND | |
| would cause large message loss. This situation was timing dependent | |
| (a race) and most frequently occurred under log rotation. In polling | |
| mode the bug was less likely, but could also occur. | |
| closes: https://github.com/rsyslog/rsyslog/issues/3465 | |
| closes: https://github.com/rsyslog/rsyslog/issues/3468 | |
| - imfile bugfix: potential segfault when working with directories or symlinks | |
| see also https://github.com/rsyslog/rsyslog/pull/3496 | |
| Thanks to Nelson Yen for the patch | |
| - omhttp bugfix: header items could not have spaces in them | |
| Thanks to Nathan Brown for the patch. | |
| - core bugfix: enlarged msg offset types for bigger structured messages | |
| using a large enough (dozens of kBs) structured message | |
| it is possible to overflow the signed short type which leads | |
| to rsyslog crash. (applies to msg.c, the message object) | |
| Thanks to Jiri Vymazal for the patch. | |
| - core bugfix for AIX: timeval2syslogTime now handle the bias according to | |
| local time zone as documented by IBM. | |
| - imfile feature: add configuration parameter to force parsing of read logs | |
| - imczmq bugfix: | |
| Release zframe following read from socket | |
| Make the 0MQ frame pointer local to the receive loop and destroy the | |
| frame as soon as the contents have been copied. This avoids: | |
| * a memory leak should the receive loop execute more than once | |
| * referencing an un-initialized value during cleanup (finalize_it) | |
| Thanks to Mark Gillott for the patch. | |
| - omclickhouse bugfix: default template unusable | |
| STDSQL option added to the default template used in output module of clickhouse | |
| Thanks to gagandeep trivedi for the patch. | |
| - omclickhouse "bugfix": work-around failed error detection | |
| omclickhouse uses a questionable method to check if a request generated | |
| an error. We have seen the method to fail when we slightly upgraded clickhouse | |
| server in CI testing. | |
| This commit makes the method a bit more reliable without really fixing it. | |
| But it's at least a short-term solution. | |
| This should be changed to a proper status check. I assume such is possible. | |
| see also https://github.com/rsyslog/rsyslog/issues/3485 | |
| - imptcp bugfix: overly long socket bind path can lead to segfault | |
| if the `path` input parameter is overly long (e.g. more than 108 | |
| characters on some platforms) a non-terminated string is generated | |
| and then passed to OS API. This can lead to all sorts of problems | |
| including segfault. | |
| We detected that based on gcc-8 warnings during code inspection. | |
| No real-world problem case is known. | |
| - ommongodb bugfix: improper stpncpy() calls | |
| - testbench tcpflood: add new transport option relp-tls | |
| Tcpflood can now send messages via relp with tls support. | |
| closes https://github.com/rsyslog/rsyslog/issues/3448 | |
| - testbench: mmdb valgrind tests failed is srcdir env was not set | |
| - testbench: add omclickhouse tests | |
| - testbench bugfix: some long-running tests had too low runtime allowance | |
| closes https://github.com/rsyslog/rsyslog/issues/3493 | |
| - testbench bugfix: daqueue-dirty-shutdown test | |
| This test occasionally failed with left-over spool files. As far as we | |
| have analyzed, this is due to the use of an invalid shutdown timeout | |
| (very short) in the second phase of the test. It looks like this is | |
| actually a copy&paste error from phase one. Behavior of rsyslog was | |
| correct, but the test itself created a false positive. | |
| We have corrected the timeout now and also modernized the test | |
| a bit. | |
| closes https://github.com/rsyslog/rsyslog/issues/2122 | |
| - testbench bugfix: some omhttp tests had compatibility issues with Python 3 | |
| Thanks to Thomas D. (whissi) for the patch. | |
| ------------------------------------------------------------------------------ | |
| Version 8.1901.0 (aka 2019.01) 2019-01-22 | |
| - new version scheme: 8.yymm.0 - version now depends on release date | |
| see also https://rainer.gerhards.net/2018/12/rsyslog-version-numbering-change.html | |
| - queue: add support for minimum batch sizes | |
| closes https://github.com/rsyslog/rsyslog/issues/495 | |
| - change queue.timeoutshutdown default to 10 for action queues | |
| The previous default of 0 gave action queues no real chance to | |
| shutdown - at the time they were applied, they were usually already | |
| expired (computing the absolute timeout took a small amount of time). | |
| So we change this now to 10ms, which still is very quick but gives | |
| the queue at least a chance to shutdown itself. That in turn | |
| smoothes the whole shutdown process. | |
| If a very large number of action queues is used this may lead | |
| to a very slightly longer shutdown time, albeit this is very | |
| improbable. | |
| - omclickhouse: new output module for clickhouse | |
| This output module adds the possibility to send | |
| INSERT querys to a Clickhouse database. See doc for details. | |
| The messages are sent via a REST interface. | |
| This commit also adds support of the testbench | |
| for clickhouse tests, as well as various tests. | |
| Closes https://github.com/rsyslog/rsyslog/issues/2272 | |
| - omkafka: Add ability to dump librdkafka statistics to a file | |
| Use statsFile to specify statistics output file; also requires | |
| setting statistics.interval.ms confparam to a non-zero value. | |
| Thanks to github user pcullen65 for the contribution. | |
| - tls(ossl/gtls): add new Option "StreamDriver.PermitExpiredCerts" | |
| The new Option can have one of the following values: | |
| on = Expired certificates are allowed | |
| off = Expired certificates are not allowed | |
| warn = Expired certificates are allowed but warning will be logged (Default) | |
| Includes necessary tests to validate new code. | |
| closes https://github.com/rsyslog/rsyslog/issues/3364 | |
| - action: add "action.resumeIntervalMax" parameter | |
| This parameter permits to set an upper limit on the growth of the | |
| retry interval. This is most useful when a target has extended | |
| outage, in which case retries can happen very infrequently. | |
| closes https://github.com/rsyslog/rsyslog/issues/3401 | |
| - report child process exit status according to config parameter | |
| Add new global setting 'reportChildProcessExits' with possible values | |
| 'none|errors|all' (default 'errors'), and new global function | |
| 'glblReportChildProcessExit' to report the exit status of a child | |
| process according to the setting. | |
| Invoke the report function whenever rsyslog reaps a child, namely in: | |
| - rsyslogd.c (SIGCHLD signal handler) | |
| - omprog | |
| - mmexternal | |
| - srutils.c (execProg function, invoked from stream.c and omshell) | |
| Remove redundant "reaped by main loop" info log in omprog. | |
| Promote debug message in mmexternal indicating that the child has | |
| terminated prematurely to a warning log, like in omprog. | |
| closes https://github.com/rsyslog/rsyslog/issues/3281 | |
| Thanks to Joan Sala for contributing this. | |
| - build system: add capability to turn off helgrind tests | |
| we add configure switch --enable-helgrind. We need to turn helgrind off | |
| when we use clang coverage instrumentation. The instrumentation injects | |
| mt-unsafe counter updates which we seem to be unable to suppress. | |
| Note: for gcc this was possible, because they all occured in a utility | |
| function. For clang, they are inlined so we get many -and changing- violations. | |
| see also https://github.com/rsyslog/rsyslog/issues/3361#issuecomment-450502569 | |
| - imzmq3/omzmq3: remove modules | |
| according to @brianknox (their author) these modules are outdated: | |
| https://twitter.com/taotetek/status/931860786959540224 | |
| They are replaced by imczmq/omczmq and are no longer maintained. We put a | |
| depreciation notice into the modules a year ago, and now it finally is time | |
| to remove them. They do NOT build in any case, except if very old versions | |
| of the 0mq ecosystem are used. | |
| see also https://github.com/rsyslog/rsyslog/issues/2100 | |
| closes https://github.com/rsyslog/rsyslog/issues/2103 | |
| - bugfix omusrmsg: don't overwrite previous set _PATH_DEV value | |
| Since commit 56ace5e418d149af27586c7c1264fccfbc6badf1, omusrmsg was broken | |
| because "memcpy()" is not a suitable substitute for "strncat()" in this | |
| context, it is actually replacing the previous added content. | |
| Bug: https://bugs.gentoo.org/673004 | |
| Closes: https://github.com/rsyslog/rsyslog/issues/3346 | |
| Thanks to Thomas D. (whissi) for the patch. | |
| - bugfix ossl TLS driver: fixed authentication mode anon | |
| authentication mode "anon" was not properly supported in ossl TLS | |
| driver; if selected, did still require a full certificate. | |
| closes: https://github.com/rsyslog/rsyslog/issues/3037 | |
| - bugfix tls subsystem: Receiver hang due to insufficient TLS buffersize. | |
| gtls and ossl driver used a default buffersize of 8KiB to store received | |
| TLS packets. When tls read returned more than buffersize, the additional | |
| buffer was not processed until new data arrived on the socket again. | |
| TLS RFCs require up to 16KiB+1 buffer size for a single TLS record. | |
| closes https://github.com/rsyslog/rsyslog/issues/3325 | |
| - bugfix pmpanngfw: build issue due to non-matching data types in comparison | |
| Thanks to Narasimha Datta for the patch. | |
| - omfile: work-around for "Bad file descriptor" errors | |
| This works-around an issue we can reproduce e.g. via the | |
| imtcp-tls-ossl-x509fingerprint.sh test. Here, omfile gets a write | |
| error with reason EBADF. So far, I was not able to see an actual | |
| coding error. However I traced this down to a multithreaded race | |
| on open and close calls. I am very surprised to see this type | |
| of issue, as I think the kernel guarantees that it does not happen. | |
| Here is what I see in strace -f: | |
| openssl accepts a socket: | |
| [pid 66386] accept(4, {sa_family=AF_INET, sin_port=htons(59054), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 10 | |
| then, it works a bit with that socket, detects a failure and shuts it down. Sometimes, at the very same instant omfile on another thread tries to open on output file. Then the following happens: | |
| [pid 66386] close(10) = 0 | |
| [pid 66389] openat(AT_FDCWD, "./rstb_356100_31fa9d20.out.log", O_WRONLY|O_CREAT|O_NOCTTY|O_APPEND|O_CLOEXEC, 0644 <unfinished ...> | |
| [pid 66386] close(10 <unfinished ...> | |
| [pid 66389] <... openat resumed> ) = 10 | |
| [pid 66386] <... close resumed> ) = 0 | |
| [pid 66386] poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, -1 <unfinished ...> | |
| [pid 66389] write(2, "file './rstb_356100_31fa9d20.out"..., 66file './rstb_356100_31fa9d20.out.log' opened as #10 with mode 420 | |
| ) = 66 | |
| [pid 66389] ioctl(10, TCGETS, 0x7f59aeb89540) = -1 EBADF (Bad file descriptor) | |
| This is **literally** from the log, without deleting or reordering | |
| lines. I read it so that there is a race between `open` and `close` | |
| where fd 10 is reused, but seemingly closed - resulting in the `EBADF` | |
| While it smells like a kernel issue, it may be a well-hidden program | |
| bug - if so, one I currently do not find. HOWEVER, this commit | |
| works around the issue by reopening the file when we receive EBADF. | |
| That's the best thing to do in that case, especially if it really is | |
| a kernel bug. Data loss should not occur, as the previous writes | |
| succeeded in that case. | |
| The backdraw of this work-around is that it only "fixes" omfile. In | |
| theory every part of rsyslog can be affected by this issues (queue | |
| files, for example). So this is not to be considered a final solution | |
| of the root issues (but a big step forward for known problem cases). | |
| see also https://github.com/rsyslog/rsyslog/issues/3404 | |
| - omhttp bugfix: segfault due to NULL pointer access | |
| many thanks to Gerardo Puerta for the patch | |
| - omkafka bugfix: segfault when running in debug mode using dynamic topics | |
| This should only affect test environments, as debug mode is not | |
| suitable for production (and really does not work when running for | |
| extended period of time). | |
| - testbench bugfix: TLS syslog tests for "anon" mode were broken | |
| They did not detect when "anon" mode was not properly supported by the | |
| drivers. | |
| - test tooling bugfix: correct tcpflood error messages | |
| it looks like tcpflood's openssl code stems partly back to tcpdump, at | |
| least the error messages indicate this. Thankfully tcpdump is BSD licensed, | |
| so this should not be a big issue. Nevertheless, the incorrect program name | |
| in error messages needs to be corrected, and this is what this commit does. | |
| - tcpflood bugfix: tool did not terminate on certificate error | |
| when tcpflood detected a certificate error, it reported an | |
| error message but did not abort. This could make errors undetectable | |
| during CI runs. | |
| also fix tests which did not properly provide CA cert (which than | |
| caused the error). | |
| - testbench: fix issues with journal testing | |
| The configure/Makefile checks were not correct, leading to the | |
| build of journal components when not necessary, even if not | |
| supported by the platform. Thus lead to invalid build and test | |
| failures. | |
| - testbench: add tests for "certless" tcp/tls | |
| This adds a test to ensure that a client without certificate can | |
| connect to a server with certificates. So it is not exactly | |
| "certless". | |
| The prime intent of this test is to match config suggestions given | |
| by log hosting companies (like loggly) and so ensure that we do | |
| not accidentally break them. This is especially important as the | |
| capability for certless clients was not properly documented and | |
| also become forgotten by the rsyslog team. | |
| see also https://github.com/rsyslog/rsyslog/issues/3413 | |
| - CI | |
| - further improve testbench robustness against slow machines | |
| - testbench: add tests for parser.EscapeControlCharacterTab global option | |
| - testbench: Updated all expired x.509 certs | |
| Closes https://github.com/rsyslog/rsyslog/issues/3348 | |
| - fix a potential race in CI debug mode which can lead to segfault | |
| only when instructed to do so, rsyslog may emit a "final worker thread shutdown" | |
| messages. This is usually only enabled in CI and/or other testing. If enabled, | |
| the code has a race on the pWti object which can lead to segfault or abort. | |
| Only system which explicitly enable this CI aid are affected (running in debug | |
| mode alone is NOT sufficient). | |
| This is a regression from 8.40.0. | |
| - testbench: improve robustness against slow CI, gen. improvements | |
| * add an overall timeout value for tests - if running longer, | |
| testbench framework tries to FAIL and end test. Note that | |
| this is not bullet-proof and not intended to be so. | |
| * guard against hanging rsyslog instances via a new imdiag | |
| feature to abort after n number of seconds; among others, | |
| this guards as against timeout-cancel in CI, which is always | |
| pretty hard to diagnose - now we see these errors in test-suite.log | |
| * fix a bug in tcp zip test, which actually did not use zip mode | |
| * experimentally add debug output to better understand | |
| shutdown_when_empty operation; goal is to improve understanding | |
| and then remove that code again. | |
| * improve shutdown predicate for a couple of tests | |
| * made travis run make check with two parallel threads, for which | |
| we seem ready now. Nevertheless, it's still experimental and we | |
| may roll this back if required. | |
| * testbench: disable omprog tests that hang under coverage instrumentation | |
| When gcc coverage instrumentation is used, these tests hang. They work | |
| with clang coverage instrumentation, but for some reason clang does not | |
| give us full reports (at least not when used together with CodeCov.io). | |
| We have tried to troubleshoot this for hours and hours - now is time to | |
| give up until someone comes up with a bright idea. So we make the affected | |
| tests skip themselves when they detect gcc with coverage instrumentation. | |
| * testbench: add new test for imfile and logrotate in copytruncate mode | |
| * testbench: add new omkafka tests for dynamic topics | |
| * travis: do no longer run 0mq tests | |
| This often causes trouble when the packages are rebuild by the 0mq project | |
| (which happens frequently). We already do intensive testing of the 0mq | |
| components in the buildbot infrastructure, where we use dedicated containers. | |
| This is reliable, as the containers already contain everything needed and so | |
| do not need to reach out to the 0mq package archives. In the light of this, | |
| let's save us the trouble of Travis failures. The only downside is that | |
| users cannot pre-test with their local Travis when modifying 0mq modules, | |
| which is quite acceptable. | |
| ------------------------------------------------------------------------------ | |
| Version 8.40.0 [v8-stable] 2018-12-11 | |
| - mmkubernetes: add support for sslpartialchain for openssl | |
| If `"on"`, this will set the OpenSSL certificate store flag | |
| `X509_V_FLAG_PARTIAL_CHAIN`. This will allow you to verify the Kubernetes API | |
| server cert with only an intermediate CA cert in your local trust store, rather | |
| than having to have the entire intermediate CA + root CA chain in your local | |
| trust store. See also `man s_client` - the `-partial_chain` flag. | |
| This option is only available if rsyslog was built with support for OpenSSL and | |
| only if the `X509_V_FLAG_PARTIAL_CHAIN` flag is available. If you attempt to | |
| set this parameter on other platforms, you will get an `INFO` level log | |
| message. This was done so that you could use the same configuration on | |
| different platforms. | |
| - openssl driver: improved error messages | |
| also fixes misleading wording of some error messages | |
| closes https://github.com/rsyslog/rsyslog/issues/3238 | |
| - imfile: disable file vs directory error on symlinks | |
| The file/directory node-object alignment now ignores symlinks. Previously | |
| it reported error on each directory symlink spamming user error logs. | |
| Thanks to Jiri Vymazal for the patch. | |
| - cleanup: remove no longer needed --enable-rtinst code | |
| configure option --enable-rtinst is gone-away since a while, but there were | |
| still some supporting code left. It required careful analysis what could | |
| actually be removed. This is now done and the code fully cleaned up. This | |
| greatly simplifies the code and also makes it better readable for | |
| developers which are not deep inside the rsyslog code base. | |
| As a positive side effect, we could eliminate mutex calls inside | |
| the debug system. This means we are more likely to reproduce race | |
| conditions in runs with debugging enabled. | |
| closes https://github.com/rsyslog/rsyslog/issues/2211 | |
| - bugfix imfile: rsyslog re-sends data for files larger 2GiB | |
| This occurs always if and only if | |
| - reopenOnTruncate="on" is set | |
| - file grows over 2GiB in size | |
| Then, the data is continuously re-sent until the file becomes smaller | |
| 2GiB (due to truncation) or is deleted. | |
| It is a regression introduced by 2d15cbc8221e385c5aa821e4a851d7498ed81850 | |
| closes https://github.com/rsyslog/rsyslog/issues/3249 | |
| - config: fix segfault in backticks "echo" expansion of undefined variables | |
| The bug was introduced in commit abe0434 (config: enhance backticks "echo" | |
| capability). The getenv() result passed to strlen() and es_addBuf() may be | |
| NULL if the environment variable does not exist, resulting in a segfault. | |
| Thanks to Julien Thomas for the patch. | |
| fixes https://github.com/rsyslog/rsyslog/issues/3006 | |
| - bugfix imsolaris: message timestamps on Solaris | |
| On Solaris messages don't have their time directly in the raw body but in | |
| a separate log_ctl structure which is currently not used. | |
| When message is logged and processed, rsyslogd gives it current time because | |
| it ignores the actual one. That means that old messages (e.g. from system | |
| reboot) get timestamp of processing instead of the reboot itself (it is | |
| not a problem for live logging where now is used anyway). | |
| Thanks to Jakub Kulik for the patch. | |
| - bugfix build system: "make distcheck" did not work for mysql tests | |
| - bugfix build system: don't link liblogging-stdlog when available but not enabled | |
| When liblogging-stdlog was available but configure option "--disable-liblogging-stdlog" | |
| was set, rsyslog was still linking against liblogging-stdlog. | |
| This commit will ensure that rsyslog will only link against liblogging-stdlog when | |
| "--enable-liblogging-stdlog" was set. | |
| see also: https://bugs.gentoo.org/667836 | |
| - bugfix RainerScript: abs() could return negative value, now in range [0..max] | |
| Thanks to Harshvardhan Shrivastava for providing the patch | |
| - bugfix debug output: date property options output wrongly | |
| inside debug logging, the date property options were not all | |
| properly converted into strings. Some of the newer ones were | |
| invalidly flagged as "UNKNOWN". This is primarily a cosmetic | |
| problem and has no effect other than puzzling folks looking at | |
| the debug log. | |
| - bugfix omhttp: did not compile on some platforms | |
| - CI | |
| * made mysql-based tests (ommysql and omlibdbi) work inside containers | |
| * bugfix testbench: do not execute libgcrypt tests if disabled | |
| closes https://github.com/rsyslog/rsyslog/issues/3228 | |
| * testbench: grep failed when string starting with "-" was used | |
| The search term was mistakenly interpreted as an option. | |
| * testbench: support auto-start/-stop of mysqld | |
| This is required to run mysql/mariadb tests inside containers. | |
| closes https://github.com/rsyslog/rsyslog/issues/3223 | |
| * improve bash coding style and fix a some bug in testbench | |
| - duplicate diag.sh init call was not detected due to typo | |
| - queue-persists test did not work correctly | |
| - some general testbench framework improvements | |
| issues found be shellcheck, fixes brought up other work to do | |
| * testbench: improve journal tests and testbench framework | |
| improving both style and reliability of journal tests; along that way | |
| also improve testbench framework: | |
| - do cleanup on error_exit and skip | |
| - explicit skip handler (vs exit 77) | |
| this permits us to do better cleanup | |
| - new testbench functions for journal-specific functionality | |
| reduce code duplication and make things easier to maintain in the | |
| future | |
| - provide a way to do valgrind and non-valgrind tests with a single | |
| test file | |
| see also https://github.com/rsyslog/rsyslog/issues/2564 | |
| * testbench: improve framework, harden rscript http test | |
| - the test now tries to detect unavailable http server, which | |
| should not result in test failure | |
| - equivalent valgrind test changed to new method, removing code | |
| duplication | |
| - testbench supports | |
| * new exit code 177, which indicates environment error, makes | |
| test SKIP but still reports the failure | |
| * new exitcode, logurl stats reporting fields | |
| * report buildbot builder (if provided) in failure report | |
| * testbench: add test for mmjsonparse with unparsable data | |
| * testbench: make es-bulk-retry test more reliable | |
| We now no longer depend on a fixed 'sleep' command but rather | |
| check the output file for what we expect. This is much more | |
| robust on slow test machines. | |
| We believe this closes the below-mentioned issue. If not, it | |
| should be re-opened. | |
| closes https://github.com/rsyslog/rsyslog/issues/3104 | |
| * testbench: suppress valgrind error caused by pthreads lib | |
| finally I give up and honestly think this is a problem in pthreads and | |
| not in rsyslog code. See issue below and previous commit for more | |
| information. | |
| Unfortunately, this will also mask off cases where we do not properly | |
| call pthread_join() albeit it is needed. Nevertheless, this bug is | |
| causing so much CI grief that it is definitely worth it. | |
| closes https://github.com/rsyslog/rsyslog/issues/2902 | |
| * testbench: made a couple of (unnamed due to too many) test more robust | |
| against slow (CI) machines | |
| ------------------------------------------------------------------------------ | |
| Version 8.39.0 [v8-stable] 2018-10-30 | |
| - imfile: improve truncation detection | |
| previously, truncation was only detected at end of file. Especially with | |
| busy files that could cause loss of data and possibly also stall imfile | |
| reading. The new code now also checks during each read. Obviously, there | |
| is some additional overhead associated with that, but this is unavoidable. | |
| It still is highly recommended NOT to turn on "reopenOnTruncate" in imfile. | |
| Note that there are also inherent reliability issues. There is no way to | |
| "fix" these, as they are caused by races between the process(es) who truncate | |
| and rsyslog reading the file. But with the new code, the "problem window" | |
| should be much smaller and, more importantly, imfile should not stall. | |
| see also https://github.com/rsyslog/rsyslog/issues/2659 | |
| see also https://github.com/rsyslog/rsyslog/issues/1605 | |
| - imjournal: work around journald excessive reloading behavior | |
| This is workaround for possible imjournal interaction with systemd | |
| where journal invalidate fix is not present. The code tries to | |
| detect SD_JOURNAL_INVALIDATE loop and not reload after each call. | |
| Thanks to Jiri Vymazal for the patch. | |
| - errmsg: remove no longer needed code | |
| refactored code (over a long time) so that object-ish style is no longer | |
| needed and could now finally be removed; We also refactored the last | |
| component (omhttp contrib module) that used the old interface. | |
| closes https://github.com/rsyslog/rsyslog/issues/1684 | |
| - queue bugfix: invalid error message on queue startup | |
| due to some old regression (commit not exactly identified, but for | |
| sure a regression, 9 years ago it was correct) an error message | |
| is emitted when no .qi file exists on startup of the queue, which | |
| is a normal condition. | |
| Actually, the code should not have tried to open the .qi file in | |
| the first place because it detected that it did not exist. That | |
| (necessary) shortcut had been removed a while ago. | |
| closes https://github.com/rsyslog/rsyslog/issues/3117 | |
| - bugfix imrelp: regression with legacy configuration startup fail | |
| Startup of a relp listener failed if legacy configuration was used. | |
| caused by commit: 32b71daa8aadb8f16fe0ca2945e54d593f47a824 | |
| closes https://github.com/rsyslog/rsyslog/issues/3106 | |
| - bugfix imudp: stall of connection and/or potential segfault | |
| There was a regression in 493279b790a8cdace8ccbc2c5136985e820dd2fa. | |
| This regression may cause stop (or delay) of reception from some systems | |
| and may also cause a segfault. Triggering condition is that at least | |
| one listener could not be created. | |
| Thanks to Jens Låås for the patch. | |
| - bugfix gcry crypto driver: small memleak | |
| If a crypto key is specified directly via the key="" parameter, | |
| the storage for that key is not freed, causing a small memleak. | |
| Note that the problem occurs only once per context, so this | |
| should not cause real issues. Even more so, as specifying a | |
| key directly is meant only for testing purposes and is strongly | |
| discouraged for production use. | |
| Detected by internal testing, no actual fail case known. | |
| - fix potential misaddressing in encryption subsystem | |
| could happen if e.g. disk queues were encrypted | |
| not seen in practice but caught by testbench test | |
| - ksi subsystem changes | |
| * enhance debug logging | |
| * disable unsafe SHA1 algorithm | |
| Thanks to Allan Park for the patch. | |
| - bugfix core: regex compile error messages could be incorrect | |
| - bugfix core: potential hang on rsyslog termination | |
| The root cause was a deadlock during worker startup. This could | |
| happen for example when a DA queue needed to persist data during | |
| shutdown. | |
| Fail condition: | |
| * startup request for a new worker | |
| * initialization of that worker | |
| * immediate detection that the worker can or must shutdown | |
| * main thread waiting for worker running state, which it skips, | |
| and so the main thread hangs inside a loop | |
| closes https://github.com/rsyslog/rsyslog/issues/3094 | |
| - bugfix imkafka: system hang when backgrounded | |
| imkafka initializes librdkafka too early (before the fork). This leads | |
| to hangs in various parts of the system - not only im imkafka but | |
| other functions as well (e.g. getaddrinfo() calls). | |
| closes https://github.com/rsyslog/rsyslog/issues/3180 | |
| - bugfix imfile: file change was not reliably detected | |
| A change in the inode was not detected under all circumstances, | |
| most importantly not in some logrotate cases. | |
| Includes new tests made by Andre Lorbach. They now use the | |
| logrotate tool natively to reproduce the issue. | |
| closes https://github.com/rsyslog/rsyslog/issues/2659 | |
| closes https://github.com/rsyslog/rsyslog/issues/1605 | |
| - bugfix imrelp: do not fail build if librelp does not have relpSrvSetLstnAddr | |
| closes https://github.com/rsyslog/rsyslog/issues/2938 | |
| - bugfix queue subsystem: DA queue did ignore encryption settings | |
| closes https://github.com/rsyslog/rsyslog/issues/3066 | |
| closes https://github.com/rsyslog/rsyslog/issues/2575 | |
| - bugfix KSI: lmsig-ksils12 module skips signing the last block | |
| Thanks to Allan Park for the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/3105 | |
| - bugfix fmhash: function hash64mod sometimes returned wrong result | |
| Thanks to Harshvardhan Shrivastava for providing the patch | |
| closes https://github.com/rsyslog/rsyslog/issues/3025 | |
| - bugfix core/debug: data written to random fd 2 under some debug settings | |
| This happens only during auto-backgrounding, where we cannot any longer | |
| access stderr. Whatever is opened with fd2 receives some debug messages. | |
| Note that the specific feature is usually turned on only in CI runs. | |
| - cleanup: removed no longer needed code | |
| Code that was unused for quite a while or did not really belong to the | |
| project identified and removed. | |
| - overall code cleanup | |
| e.g. remove unused code, replace bad bash constructs, etc... | |
| - CI: | |
| * some small improvements in testbench plumbing | |
| e.g. (`cmd` replaced by $(cmd), removed useless use of cat, ...) | |
| * testbench: improve plumbing for kafka tests | |
| - Removed all sleeps where possible. | |
| - Moved all kafka start/stop/download logic into functions. | |
| - Moved kafka/zookeeper stop into error_exit and exit_test. | |
| - Kafka/Zookeeper cleanup only done on success now. | |
| - Kafka/Zookeeper logfiles automatically dumped on error_exit only now. | |
| - Added cleanup for Kafka/Zookeeper instances into CI/buildbot_cleanup.sh | |
| - added new tests | |
| * testbench: fix incompatibility of one omprog test with Python3 | |
| Python3 writes to stderr immediately, and this caused the | |
| captured output to differ with respect to Python2. Simplified | |
| the test to do a single write to stderr. Also a cast to int | |
| was needed when calculating 'numRepeats'. | |
| closes https://github.com/rsyslog/rsyslog/issues/3030 | |
| * testbench: fixed imfile parallel issues | |
| - Fixed timing issues in some imfile wildcard/regex tests | |
| - Added touch command in imfile wildcard tests to make sure directories | |
| exist before files are created in it if IO is under stress. | |
| - changed content checking in some tests to use "content_check_with_count" | |
| with check timeouts instead of using fixed sleeptimes. | |
| * testbench: new basic tests | |
| These ensure that for some modules that did not have any tests at all | |
| we have at least a minimal coverage (module loads, activates, is able | |
| to emit error messages). Of course, further improvements would make | |
| much sense. Modules: | |
| - ommail | |
| - testbench: new tests for disk queue encryption | |
| - testbench: improved auto-diagnostics for hanging instance | |
| - testbench: hardened kafka test against failing kafka subsystem, | |
| not in 100% of the cases, but at least in some that frequently occur | |
| - failing tests now report failure status so that we can get stats | |
| on unreliable tests | |
| - testbench tooling: fix incorrect tcpflood TLS parameter check | |
| could lead to segfault when started | |
| - bugfix testbench tooling: tcpflood invalid type in calloc (openssl mode) | |
| It is unlikely that this has caused a real issue, as long as pointers | |
| are all of the same size (what is highly probable). | |
| detected by cppcheck via Codacy.com | |
| ------------------------------------------------------------------------------ | |
| Version 8.38.0 [v8-stable] 2018-09-18 | |
| - AIX: make basic modules work again | |
| - make rsyslog build on AIX again | |
| ... at least for a limited set of default modules | |
| - imfile: support for endmsg.regex | |
| This adds support for endmsg.regex. It is similar to | |
| startmsg.regex except that it matches the line that denotes | |
| the end of the message, rather than the start of the next message. | |
| This is primarily for container log file use cases such as this: | |
| date stdout P start of message | |
| date stdout P middle of message | |
| date stdout F end of message | |
| The `F` means this is the line which contains the final part of | |
| the message. The fully assembled message should be | |
| `start of message middle of message end of message`. | |
| `startmsg.regex="^[^ ]+ stdout F "` will match. | |
| Thanks to Richard Megginson for the patch. | |
| - imkafka: add parameter "parseHostName" | |
| This enables imkafka to parse the hostname from log message. | |
| Previously that was not possible. It was most likely a bug, but | |
| one that users may count on. The new parameter "ParseHostName" | |
| (default is off) controls this behavior. Default is to NOT | |
| parse the hostname. | |
| Thanks to github user snaix for the contribution. | |
| - im[p]tcp: improve error message on connect failure | |
| Now a message with the actual OS error is emitted, making things far | |
| easier to troubleshoot. | |
| - imkafka: implement multithreading support for kafka consumers. | |
| Each consumer runs in it's own consumer thread now. New tests have also | |
| been added for this. | |
| - omelasticsearch: write all header metadata to $.omes for retries | |
| Write all of the original request metadata fields to $.omes for | |
| the retry, if present. This may include all of the following: | |
| _index, _type, _id, _parent, pipeline | |
| This is in addition to the fields from the response. If the same | |
| field name exists in the request metadata and the response, the | |
| field from the request will be used, in order to facilitate | |
| retrying the exact same request. | |
| Thanks to Richard Megginson for the patch. | |
| - core: improve error message on module load fail | |
| The error message now lists all dlopen() errors in depth. This is | |
| especially useful if the error is due to missing symbols or file | |
| format errors. | |
| - core/queue: add error message if queue file cannot be accessed | |
| When having a disk-assisted queue without permission to write to the specified | |
| queue file an error will now be generated. | |
| closes https://github.com/rsyslog/rsyslog/issues/323 | |
| - imtcp/imudp: new option preservecase for managing the case of FROMHOST value | |
| default is left at current behavior | |
| see also https://github.com/rsyslog/rsyslog/pull/2774 | |
| see also https://bugzilla.redhat.com/show_bug.cgi?id=1309698 | |
| - omprog: add feedback timeout and keep-alive feature | |
| - Restart the program if it does not respond within timeout. | |
| - New setting 'confirmTimeout' (default 10 seconds). | |
| - Allow the program to provide keep-alive feedback when a | |
| message requires long-running processing. | |
| - Improve efficiency when reading feedback line (use buffer). | |
| Retry interrupted writes/reads to/from pipe. | |
| - New setting 'reportFailures' for reporting error messages | |
| from the program. | |
| - Report child termination when writing to pipe. | |
| - Minor refactor: renamed writePipe function to sendMessage, | |
| renamed readPipe to readStatus. | |
| Thanks to Joan Sala for contributing this. | |
| - omprog: fix forceSingleInstance configuration option | |
| The forceSingleInstance option did not work as intended. Even | |
| if set multiple instances were spawned. This most probably | |
| was a regression from 0453b1670fc34c96d31ee7c9a370f0f5ec24744a | |
| The code was broken roughly 3.5yrs ago, so it looks like the | |
| issue was little-noticed. This also means that potentially some users | |
| may see the bugfix as change of behavior. If so, just remove | |
| the option. | |
| closes https://github.com/rsyslog/rsyslog/issues/2813 | |
| closes https://github.com/rsyslog/rsyslog/issues/2468 | |
| Thanks to Joan Sala for contributing this. | |
| - imfile: implement file-id, used in state file | |
| This ensures that files with the same inodes are not accidentally treated | |
| as equal, at least within the limits of the file id hash (see doc for | |
| details). | |
| We use the siphash reference implementation to generate our non-cryptographic | |
| hash. | |
| closes https://github.com/rsyslog/rsyslog/issues/2530 | |
| closes https://github.com/rsyslog/rsyslog/issues/2231 | |
| - imfile: experimental input throttling feature | |
| The new input parameter delay.message has been added. It specifies | |
| a delay in microseconds after each line read. | |
| closes https://github.com/rsyslog/rsyslog/issues/2960 | |
| - core: emit TZ warning on startup not on Linux non-container | |
| On Linux it seems common that the TZ variable is NOT properly set. | |
| There are some concerns that the warning related to rsyslog correcting | |
| this confuses users. It also seems that the corrective action rsyslog | |
| takes is right, and so there is no hard need to inform users on that. | |
| In Linux containers, however, the warning seems to be useful as the | |
| timezone setup there seems to be frequently-enough different and | |
| rsyslog's corrective action may not be correct. | |
| So we now check if we are running under Linux and not within a container. | |
| If so, we do not emit the warning. In all other case, we do. This is | |
| based on the assumption that other unixoid systems still should have | |
| TZ properly set. | |
| closes https://github.com/rsyslog/rsyslog/issues/2994 | |
| - omkafka: | |
| * better debug information | |
| * Fixed minor issue in omkafka producing wrong kafka timestamps when | |
| msgTimestamp was NULL. | |
| * Setting RD_KAFKA_V_KEY(NULL, 0) in rd_kafka_producev now when KEY is not | |
| configured. | |
| * Fixed minor issue when rsyslog is compiled with --enable-debug and | |
| librdkafka is too old. | |
| - omfile bugfix: errant error message when dynafile param needed | |
| also fixes related message in contributed module omfile-hardened | |
| closes https://github.com/rsyslog/rsyslog/issues/2975 | |
| Thanks to Frank Bicknell for the patch | |
| - omhttp: new contributed module | |
| Thanks to Christian Tramnitz for contributing it. | |
| Some more info at https://github.com/rsyslog/rsyslog/pull/2782 | |
| - mmkubernetes: action fails preparation cycle if kubernetes API ... | |
| ... destroys resource during bootup sequence | |
| The plugin was not handling 404 Not Found correctly when looking | |
| up pods and namespaces. In this case, we assume the pod/namespace | |
| was deleted, annotate the record with whatever metadata we have, | |
| and cache the fact that the pod/namespace is missing so we don't | |
| attempt to look it up again. | |
| In addition, the plugin was not handling error 429 Busy correctly. | |
| In this case, it should also annotate the record with whatever | |
| metadata it has, and _not_ cache anything. By default the plugin | |
| will retry every 5 seconds to connect to Kubernetes. This | |
| behavior is controlled by the new config param `busyretryinterval`. | |
| This commit also adds impstats counters so that admins can | |
| view the state of the plugin to see if the lookups are working | |
| or are returning errors. The stats are reported per-instance | |
| or per-action to facilitate using multiple different actions | |
| for different Kubernetes servers. | |
| This commit also adds support for client cert auth to | |
| Kubernetes via the two new config params `tls.mycert` and | |
| `tls.myprivkey`. | |
| Thanks to Richard Megginson for the patch. | |
| - bugfix pmnormalize/core: several memory leaks, invld property handling | |
| - major memory leak which occurred once per message processed | |
| So this could lead to OOM. Caused by improper free of json | |
| structure | |
| - another two major leaks of similar magnitude could occur if | |
| "fromhost-ip" and/or "fromhost" properties were set | |
| - minor leaks upon termination. these were unproblematic as | |
| static and only occured immediately before shutdown. | |
| But they triggered memory debugger errors. | |
| - fixed test which did not check for mem leaks albeit it should | |
| - core invalid handling of the "fromhost" property, if set via | |
| the MsgSetPropsViaJSON() call. This was primarily of concern | |
| for pmnormalize and mmexternal, and only if these properties | |
| were used by either the rulebase or the external program | |
| response. | |
| Actually, most of the leaks go back to rsyslog core, but that | |
| core functionality was not used by other modules in the same | |
| way. But if some other would have used it, the effects would | |
| have been the same (so be aware if you wrote custom modules). | |
| - bugfix imptcp: fixed pointers for session counting | |
| imptcp open, failedopen, and closed pstats counters were assigned the wrong | |
| name, thus pstats values did provide a totally wrong picture of what was | |
| going on. | |
| Thanks to github user jeverakes for the patch. | |
| - bugfix omprog: invalid memory access on partial writes to pipe | |
| When sending logs to the program, in case of a partial write to the pipe, | |
| invalid data was sent, or an invalid memory access could occur. (A | |
| partial write can occur if the syscall is interrupted or the pipe is full.) | |
| Thanks to Joan Sala for contributing this. | |
| - bugfix omprog: rsyslog's environment was not passed to script | |
| closes https://github.com/rsyslog/rsyslog/issues/2921 | |
| - bugfix omprog: severity of some log messages in waitForChild corrected | |
| Log some messages related to child process termination as info/warn | |
| instead of error. | |
| - bugfix imfile: files which were loaded via symlink were not always followed | |
| They were stopped watching after being rotated. | |
| closes https://github.com/rsyslog/rsyslog/issues/2229 | |
| Thanks to Jiri Vymazal for the patch. | |
| - bugfix imfile: potential misaddressing when processing symlinks | |
| Fixed parent name when processing symlinks. Detected during code review. | |
| There was a garbage byte left before which could cause errors down the | |
| road. | |
| Thanks to Jiri Vymazal for the patch. | |
| - bugfix ommongodb: build issue if mongo-c-driver is not compiled with TLS | |
| Let ommongodb module works even if mongo-c-driver is not compiled with SSL support. | |
| Thanks to Jérémie Jourdin for the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/2907 | |
| - CI: | |
| * many changes with the goal to support parallel test execution, e.g. | |
| use dynamic ports and file names, changes to testing tools, etc. | |
| * kafka tests re-enabled, as they should now no longer be racy. However, | |
| this has yet to be proven in practice. | |
| * upgrading kafka server version to current | |
| * Fixed server configuration issues holding the kafka tests back from working | |
| * Fixed some config issues in all sndrcv kafka tests. | |
| * Generating dynamically kafka topics now for each kafka test. | |
| * Reenabled kafka_multi test which runs a test on 3 kafka/zookeeper instances | |
| ------------------------------------------------------------------------------ | |
| Version 8.37.0 [v8-stable] 2018-08-07 | |
| - build system: add --enable-default-tests ./configure option | |
| This permits to control the "default tests" in testbench runs. These | |
| are those tests that do not need a special configure option. There are | |
| some situations where we really want to turn them of so that we can | |
| run tests only for a specific component (e.g. ElasticSearch). | |
| This commit also removes the --enable-testbench[12] configure switches, | |
| which were introduced just to work-around travis runtime restrictions. | |
| With the new CI setup and new options we could reduce the Travis runtime | |
| dramatically and so we do not need them any longer. | |
| - overall adaptation to gcc 8 which emits new warnings | |
| - fix some build warnings on 32bit systems, namely armhf architecture | |
| - ommail change of behavior: "enable.body" default now "on" | |
| This was always documented to be "on", but actually was "off". Usually, we | |
| fix the doc, but after long discussion the agreement was that in this | |
| specific case it was actually better to change the default. | |
| see also: https://github.com/rsyslog/rsyslog/pull/2791 | |
| - core/omfile: race in async writing mode | |
| mutex was not properly locked at all times when the async writing buffer | |
| was flushed | |
| Thanks to Radovan Sroka for the patch. | |
| - core: provide a somewhat better default action name | |
| We now include the module name (e.g. "omelasticsearch" or "builtin:omfile") | |
| as part of the name. This is still not perfect, but hopefully a bit | |
| easier to grasp. | |
| see also https://github.com/rsyslog/rsyslog/issues/342 | |
| - new global() parameter "abortOnUncleanConfig" | |
| This provides a new-style alternative to $AbortOnUncleanConfig. | |
| closes https://github.com/rsyslog/rsyslog/issues/2744 | |
| - tcpflood no longer links with -lgrcypt | |
| as this is no longer necessary for GnuTLS | |
| Thanks to Michael Biebl for the patch. | |
| - imjournal: add journal-specific impstats counters | |
| these provide some additional insight into journal operations | |
| Thanks to Abdul Waheed for the patch. | |
| - imjournal: fixed startup on missing state file | |
| When starting rsyslog with imjournal for first time it outputs | |
| an error and plugin does not run because no state file exists yet. | |
| Now it skips the loading and creates state file on first persist. | |
| Thanks to Jiri Vymazal for the patch. | |
| - imjournal: fetching cursor on readJournal() and simplified pollJournal() | |
| Fetching journal cursor in persistJournal could cause us to save | |
| invalid cursor leading to duplicating messages further on, when new | |
| WorkAroundJournalBug option is set we are saving it on each | |
| readJournal() where we now that the state is good. | |
| pollJournal() is now cleaner and faster, correctly handles INVALIDATE | |
| status from journald and is able to continue polling after journal | |
| flush. Also reduced POLL_TIMEOUT a bit as it caused rsyslog to exit | |
| with error in corner cases for some ppc when left at full second. | |
| re-factored imjournal CI tests with journal_print tool to have more | |
| detailed error reporting. | |
| Thanks to Jiri Vymazal for the patch. | |
| - config: enhance backticks "echo" capability | |
| This is now more along the lines of what bash does. We now support | |
| multiple environment variable expansions as well as constant text | |
| between them. | |
| example: | |
| env SOMEPATH is set to "/var/log/custompath" | |
| config is: param=`echo $SOMEPATH/myfile` | |
| param than is expanded to "/var/log/custompath/myfile" | |
| among others, this is also needed inside the testbench to properly | |
| support "make distcheck". | |
| Note: testbench tests follows via separate commit. There will be | |
| no special test, as the testbench itself requires the functionality | |
| at several places, so the coverage will be very good even without | |
| a dedicated test. | |
| - imrelp: add support for setting address to bind to (#894) | |
| This adds a new optional `address` parameter to `imrelp` inputs in order | |
| to specify an address to bind to. | |
| Based on support added by rsyslog/librelp@96eb5be | |
| Thanks to Simon Wachter for the patch. | |
| - omrelp: permit all authmodes; updated tests | |
| omrelp for some time limited authentication modes to those | |
| that were known. While this was OK, it prevented the easy | |
| introduction of new auth modes into librel. | |
| This has now been changed; omrelp now checks the validity of | |
| the authmode directly via librelp by doing some librelp calls | |
| upon processing the configuration. | |
| Also, some tests have been updated to check this feature and | |
| also ensure that the new librelp mode "certvalid" works | |
| (if it is available). | |
| - regexp.c: reduce lock contention when using glibc. | |
| When using glibc, we enable per-thread regex to avoid lock contention. | |
| See: | |
| - https://github.com/rsyslog/rsyslog/issues/2759 | |
| - https://github.com/rsyslog/rsyslog/pull/2786 | |
| - https://sourceware.org/bugzilla/show_bug.cgi?id=11159 | |
| This should not affect BSD as they don't seem to take a lock in regexec. | |
| NOTE: it is assumed that we can craft an even better solution than | |
| this patch, but it improves the situation and we do not have time to | |
| craft more. So we decided to merge. For details see | |
| https://github.com/rsyslog/rsyslog/pull/2786 | |
| - mmpstrucdata: better error message, support $! in var names | |
| see also https://github.com/rsyslog/rsyslog/issues/1262#issuecomment-404773495 | |
| - more explicit error msg with message modification mod on queue | |
| Message modification modules do not work if used with a non-direct queue. | |
| We now make this more explicit in the config parsing error message. | |
| closes https://github.com/rsyslog/rsyslog/issues/1323 | |
| - omrabbitmq: improve high-load performance | |
| A different pthread mutex is created for each connection (action) | |
| instead of a single one shared by all connections. This will | |
| improve performance when using multiple concurrent connections | |
| to a single (or multiple) RabbitMQ instance(s) (e.g. for load balancing) | |
| Thanks to github user micoq for contributing the patch. | |
| - imudp: replace select() calls by poll() | |
| This improves reliability in extreme cases (more than 1024 fds open when | |
| imudp begins to listen) and potentially improves performance a little. | |
| - ommysql: support mysql unix domain socket: | |
| via action(.. socket="/tmp/mysqld.sock" ..) | |
| Thanks to JoungKyun Kim for contributing this. | |
| - impstats: emit warning if log.syslog="off" and ruleset name given | |
| With this config, "ruleset" is silently ignored, what probably is | |
| not obvious to a user. | |
| closes https://github.com/rsyslog/rsyslog/issues/2821 | |
| - build system cleanup: remove no longer needed --enable-memcheck | |
| This was used for a very old testing capability, no longer functional but | |
| causes build to fail if enabled. Replaced by ASAN/valgrind. | |
| Issue detected while testing some other CI settings. | |
| - tools: Updated python based statslog analyzer sample scripts | |
| - developer tools: make devcontainer tool more developer friendly | |
| slight improvement for easy interactive use | |
| - enable better testing via "make distcheck" | |
| Also a couple of changes to testbench worth mentioning: | |
| * use cp -f to ensure files can be overwritten in VBUILD | |
| * fix issue of missing include test file in EXTRA_DIST | |
| * new suppressions | |
| * testbench: try to use local system dependency cache | |
| avoid going to Internet repos if not absolutely necessary. For | |
| development containers, they should be pre-populated with the | |
| important dependencies. | |
| * do not enable libfaketime if ASAN is selected | |
| unfortunately, libfaketime does not work in that case | |
| Note: for modules with non-standard dependencies (e.g. databases), | |
| "make distcheck" only enables what on the original ./configure line | |
| was enabled. This is done in order to ensure that "distcheck" adapts | |
| to what is actually available on the system in question. Rsyslog's | |
| own CI system installs the maximum set of possible dependencies and | |
| so tries the maximum set "make distcheck" can support on a platform. | |
| see also https://github.com/rsyslog/rsyslog/issues/174 | |
| - add new global config parameter "inputs.timeout.shutdown" | |
| - omusrmsg: do not fall back to max username length of 8 | |
| This happens if utmp.h and friends are not available and stems back to | |
| the original syslogd. Nowadas, 32 is more appropriate and now being used | |
| in that (now very unlikely) case. The detection logic for UT_NAMESIZE has | |
| also been streamlined. | |
| closes https://github.com/rsyslog/rsyslog/issues/2834 | |
| - bugfix build system: fix race in parallel builds | |
| If libgcry.la is built later than lmcry_gcry.la, there is a failure: | |
| [snip] | |
| |../aarch64-wrs-linux-libtool --tag=CC --mode=link aarch64-wrs-linux-gcc | |
| -o lmcry_gcry.la lmcry_gcry_la-lmcry_gcry.lo libgcry.la -lgcrypt | |
| |aarch64-wrs-linux-libtool: error: cannot find the library 'libgcry.la' | |
| or unhandled argument 'libgcry.la' | |
| |Makefile:1049: recipe for target 'lmcry_gcry.la' failed | |
| |make[2]: *** [lmcry_gcry.la] Error 1 | |
| [snip] | |
| The LIBADD of lmcry_gcry.la contains libgcry.la, we should also add libgcry.la | |
| to lmcry_gcry.la's DEPENDENCIES. | |
| Thanks to Hongxu Jia for the patch. | |
| - bugfix imfile: memory leak upon shutdown (cosmetic) | |
| When rsyslog shuts down and imfile is inside a change polling loop, | |
| it does not properly free memory returned by glob(). This is a cosmetic | |
| bug as the process terminates within the next few milliseconds. However, | |
| it causes memory analyzer reports and thus makes CI fail. | |
| - bugfix core msg: potential deadlock (and rsyslog hang) | |
| can happen e.g. with headerless messages when app-name | |
| property is used | |
| closes https://github.com/rsyslog/rsyslog/issues/3135 | |
| - bugfix core: do not abort startup on problems setting scheduling policy | |
| rsyslog creates a default scheduling policy on startup. This code | |
| invalidly used CHKiRet (our exception handler) to check pthreads | |
| return codes, what this macro cannot do. This lead to hard to | |
| diagnose startup problems in cases where there were problems | |
| setting the scheduling defaults (e.g. when rsyslog is set to run | |
| at idle priority). Even more so, this blocked startup altogether, | |
| which is not the right thing to do. Actually, this can be considered | |
| a regression from commit 7742b21. That commit was 8 years ago, so | |
| in general this cannot be a big issues ;-) | |
| The code now emits proper error messages (to stderr, as at this point | |
| no other output is available as it is during the initial state of | |
| rsyslog initialization) and continues the startup. | |
| closes https://github.com/rsyslog/rsyslog/issues/2855 | |
| - bugfix core: input shutdown timeout not properly applied | |
| The timeout could be reduced by mutex wait time, which was not the | |
| intended behavior and could lead the the input thread being | |
| cancelled while it would have been perfectly legal to shut it down | |
| cleanly. | |
| Noticed during working on the CI system. May explain some testbench | |
| instability and may have caused trouble with state files (not) | |
| properly being written by inputs. | |
| - bugfix config optimizer: error in constant folding | |
| did not work properly if a string and a number were to be folded. | |
| Detected by gcc 8. | |
| - build: fix improper function casts | |
| no real issue, but generated warnings under gcc 8 and thus | |
| broke CI | |
| - bugfix omlibdbi: fix potential small memory leak | |
| detected by clang static analyzer | |
| - bugfix ommysql: unsafe use of strncpy() | |
| also now reports oversize names as user error vs. silent truncation | |
| overly long names only could affect config load phase | |
| - bugfix omhttpfs: fix insecure usage if strncmp() | |
| consequences not evaluated as this is a contributed module. | |
| Detected by gcc 8. | |
| - bugfix mmgrok: cosmetic build issue - compiler warnings | |
| caused build under gcc 7 to fail with warning | |
| - bugfix mmkubernetes: stops working with non-kubernetes container names | |
| When mmkubernetes encounters a record with a CONTAINER_NAME field, | |
| but the value does not match the rulebase, mmkubernetes returns | |
| an error, and mmkubernetes does not do any further processing | |
| of any records. | |
| The fix is to check the return value of ln_normalize to see if | |
| it is a "hard" error or a "does not match" error. | |
| This also adds a test for pod names with dots in them. | |
| Thanks to Richard Megginson for the patch. | |
| - bugfix mmkubernetes: potential NULL pointer access | |
| If token file could not be opened, fclose() was passed a NULL pointer. | |
| Thanks to github user jvymazal for finding and Richard Megginson | |
| for fixing the issue. | |
| - bugfix omsnmp: invalid traptype was not detected | |
| this could leave config errors unreported and cause unexpected | |
| behavior | |
| - bugfix mmkubernetes: default rules use container_name_and_id | |
| also include rulebase files in dist and fix rule so that dot inside | |
| pod name is supported. | |
| Thanks to Richard Megginson for fixing the issue. | |
| - bugfix omelasticsearch: build regression | |
| Commit 6d4635efbb13907bf651b1a6e5a545effe84d9d9 introduced some compile | |
| problems, which were only detected on CentOS6, which unfortunately did | |
| not compile omelasticsearch during CI runs | |
| - bugfix ommongodb: do not force MongoDB to use "PLAIN" auth mechanism | |
| ... which also seems not to be handled by current MongoDB. | |
| Remove ?authMechanism=PLAIN URI part to let the mongo library chooses the | |
| default mechanism. One can force a specific authentication mechanism by | |
| adding ?authMechanism=XXX into the uristr argument of the module | |
| Thanks to Jérémie Jourdin for the fix. | |
| closes https://github.com/rsyslog/rsyslog/issues/2753 | |
| - build system: do not disable tests via --disable-liblogging-stdlog | |
| This setting controlled both the actual rsyslog functionality as well | |
| as some testbench tests, which use liblogging-stdlog to provide some | |
| specific functionality. This meant those tests were not run since | |
| changing the default. Now untangling the dependency. | |
| - CI: | |
| * most test refactored to use newer testbench plumbing | |
| while no functional change, this permits further enhancements | |
| * ElasticSearch startup timeout in tests increased to care for | |
| slower test systems | |
| * imjournal: fixed tests to actually test plugin functionality | |
| Thanks to Jiri Vymazal for the patch. | |
| * new test for gnutls priority string in librelp | |
| Thanks to github user jvymazal for the patch | |
| * testbench: relax hanging instance detection | |
| This does not work reliably if multiple instances of rsyslog | |
| builds run on a single machine. We need to improve, but this | |
| commit makes conflict less likely and provides some diagnostic | |
| info to help guide us towards a final solution. | |
| * testbench: fix tests that look awfully wrong | |
| These tests indicated they terminate rsyslog forcefully without | |
| draining the queues, but then checked if they were drained (all | |
| messages processed). That does not make sense, and we cannot | |
| envision why this was written the first place. So we assume some | |
| copy&paste problem was the root of that. | |
| * testbench: refactor tests which used "nettester" tool | |
| Some old tests are carried out via the nettester tool. This was | |
| our initial shot at a testbench a couple of years ago. While it | |
| worked back then, the testbench framework has been much enhanced. | |
| These old tests are nowadays very hard to handle, as they miss | |
| debug support etc. So it is time to refactor them to new style. | |
| As a side-activity, the testbench plumbing has been enhanced to | |
| support some operations commonly needed by these tests. Contrary | |
| to pre-existing plumbing, these new operations are now crafted | |
| using bash functions, which we consider superior to the current | |
| method. So this is also the start of converting the older-style | |
| functionality into bash functions. We just did this now because | |
| it was required and we entangled it into the test refactoring | |
| because it was really needed. Else we had to write old-style | |
| operations and convert them in another commit, which would | |
| have been a waste of time. | |
| Special thanks to Pascal Withopf for the initial step of taking | |
| old tests and putting config as well as test data together into | |
| the refactored tests, on which Rainer Gerhards than could build | |
| to create the new tests and update testbench plumbing. | |
| * testbench: ensure uxsock test leaves no dangling listener instances | |
| ..in case the test aborts. We utilize the timeout utility for now | |
| to prevent this. | |
| * testbench: make port for imdiag dynamic | |
| This is prep work to support parallel test runs | |
| ------------------------------------------------------------------------------ | |
| Version 8.36.0 [v8-stable] 2018-06-26 | |
| - build system change: | |
| Liblogging-stdlog was introduced to provide a broader ability to send rsyslog | |
| internal logs to different sources. However, most distros did not pick up | |
| that capability and so instead we do a regular syslog() call. We assume that | |
| the actual functionality is never used in practice, so we plan to retire it. | |
| That makes building rsyslog from source easier. | |
| The plan is to disable use of liblogging-stdlog by default during | |
| configure. So users (and distros!) can still opt-in to have it enabled if | |
| they desire. | |
| A couple of releases later, we want to completely remove the functionality, | |
| except if there has desire been shown in the meantime which justifies to keep | |
| liblogging-stdlog. | |
| This version disabled liblogging-stdlog by default. We now also | |
| emit a warning message ("liblogging-stdlog will go away") so that users | |
| know what is going on and my react. | |
| closes https://github.com/rsyslog/rsyslog/issues/2705 | |
| see also https://github.com/rsyslog/rsyslog/issues/2706 | |
| - add openssl driver alongside GnuTLS one for TLS communication | |
| The openssl driver is currently experimental. It will become the new preferred | |
| driver as it permits us to provide much better end-user error message than | |
| we could provide with GnuTLS. It is also less picky with certificate files | |
| and provides specific error messages if there are certificate problems. | |
| closes: https://github.com/rsyslog/rsyslog/issues/1390 | |
| closes: https://github.com/rsyslog/rsyslog/issues/1840 | |
| closes: https://github.com/rsyslog/rsyslog/issues/1352 | |
| closes: https://github.com/rsyslog/rsyslog/issues/1702 | |
| closes: https://github.com/rsyslog/rsyslog/issues/2547 | |
| - GnuTLS TLS driver: support intermediate certificates | |
| this is necessary for certificate chains | |
| Thanks to Arne Nordmark for providing the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/2762 | |
| - omelasticsearch: write op types; bulk rejection retries | |
| * Add support for a 'create' write operation type in addition to | |
| the default 'index'. Using create allows specifying a unique id | |
| for each record, and allows duplicate document detection. | |
| * Add support for checking each record returned in a bulk index | |
| request response. Allow specifying a ruleset to send each failed | |
| record to. Add a local variable `omes` which contains the | |
| information in the error response, so that users can control how | |
| to handle responses e.g. retry, or send to an error file. | |
| * Add support for response stats - count successes, duplicates, and | |
| different types of failures. | |
| * Add testing for bulk index rejections. | |
| Thanks to Richard Megginson for the patch. | |
| - lookup tables: reload message now with "info" severity (was "error") | |
| thanks to Adam Chalkley for the patch | |
| - imptcp: add support for regex-based framing | |
| for complex multi-line messages (XML in particular), the multiLine method | |
| does not work well. We now have a capability to specify via a regex when | |
| a frame starts (and the previous thus ends). | |
| adds imptcp input parameter "framing.delimiter.regex" | |
| - imjournal: add statistics counter | |
| following statistics counter are now supported by imjournal | |
| - submitted = total number of messages submitted for processing | |
| closes https://github.com/rsyslog/rsyslog/issues/2549 | |
| - config: permit 4-digit file creation modes | |
| permit 4-digit file creation modes (actually 5 with the leading zero) so | |
| that the setgid bit can also be set (and anything else on that position. | |
| closes https://github.com/rsyslog/rsyslog/issues/1092 | |
| - ommongodb: add possibility to ignore some insertion error code | |
| new config parameter "allowed_error_codes", which will be ignored if | |
| they happen. For example, 11000 DuplicateKey in case of collection | |
| containing a unique field. | |
| Thanks to Hugo Soszynski for contributing this work | |
| - omprog: simplify 'plugin-with-feedback.py' example | |
| Make the skeleton easier to understand by removing transaction support. | |
| Also, transaction failures did not work as explained in the skeleton, | |
| because of issue #2420. In the future, a 'plugin-with-transactions.py' | |
| example can be added, ideally once the issue is solved. | |
| Thanks to Joan Sala for contributing this. | |
| - core: misaddressing when writing disk queue files | |
| when writing disk queue files during shutdown, access to freed | |
| memory can occur under these circumstances: | |
| - action A is processing data, but could not complete it | |
| most importantly, the current in-process batch needs not to | |
| be totally completed. Most probable cause for this scenario | |
| is a suspended action in retry mode. | |
| - action A is called from a ruleset RA which | |
| - does not have a queue assigned | |
| - where RA is called from a ruleset RO which is bound | |
| to the input from which the message originated | |
| - RO must be defined before RA inside the expanded config | |
| - Disk queues (or the disk part of a DA queue) must be utilized by A | |
| When re-injecting the unprocessed messages from A into the disk queue, the | |
| name of ruleset RO is accessed (for persisting to disk). However, RO is | |
| already destructed at this point in time. | |
| The patch changes the shutdown processing of rulesets, so that all | |
| shutdown processing is done before any ruleset data is destructed. This | |
| ensures that all data items which potentially need to be accessed | |
| remain valid as long as some part may potentially try to access them. | |
| This follows a the approach used in | |
| https://github.com/rsyslog/rsyslog/pull/1857 | |
| where obviously that part of the problem was not noticed. | |
| see also https://github.com/rsyslog/rsyslog/issues/1122 | |
| closes https://github.com/rsyslog/rsyslog/issues/2742 | |
| - core: fix message loss on target unavailability during shutdown | |
| Triggering condition: | |
| - action queue in disk mode (or DA) | |
| - batch is being processed by failed action in retry mode | |
| - rsyslog is shut down without resuming action | |
| In these cases messages may be lost by not properly writing them | |
| back to the disk queue. | |
| closes https://github.com/rsyslog/rsyslog/issues/2760 | |
| - imrelp bugfix: error message "librelp too old" is always emitted ... | |
| ... even if librelp is current. The condition check was actually missing. | |
| This commit adds it. | |
| closes https://github.com/rsyslog/rsyslog/issues/2712 | |
| - imrelp: segfault on startup when cert without priv key is configured | |
| closes https://github.com/rsyslog/rsyslog/issues/2747 | |
| - omrelp bugfix: segfault on first message sent when authmode was wrong | |
| A segfault could occur if the authmode was configured to an invalid value. | |
| This is now caught during config processing and an error is reported. | |
| closes https://github.com/rsyslog/rsyslog/issues/2743 | |
| - imfile bugfix: double-free on module shutdown | |
| detected by code review, not seen in practice | |
| - imfile/core bugfix: potential misaddressing in string copy routine | |
| This can be exposed via imfile, as follows: | |
| - use a regex to process multiline messages | |
| - configure timeouts | |
| - make sure imfile reads a partial message | |
| - wait so that at least one timeout occurs | |
| - add the message termination sequence | |
| This leads to a misaddressing, which may have no obvious effects potentially | |
| up to a segfault. | |
| closes https://github.com/rsyslog/rsyslog/issues/2661 | |
| - imfile bugfix: if freshStartTail is set some initial file lines missing | |
| When the option is set and a new file is created after rsyslog startup, | |
| freshStartTail is also applied to it. That is data written quickly to it | |
| (before rsyslog can process it) will potentially be discarded. If so, | |
| and how much, depends on the timing between rsyslog and the logging process. | |
| This problem is most likely to be seen in polling mode, where a relatively | |
| long time may be required for rsyslog to find the new file. | |
| This is changed so that now freshStartTail only applies to files that | |
| are already-existing during rsyslog's initial processing of the file | |
| monitors. HOWEVER, depending on the number and location (network?) of | |
| existing files, this initial startup processing may take some time as | |
| well. If another process creates a new file at exactly the time of | |
| startup processing and writes data to it, rsyslog might detect this | |
| file and it's data as prexisting and may skip it. This race is inevitable. | |
| So when freshStartTail is used, some risk of data loss exists. The same | |
| holds true if between the last shutdown of rsyslog and its restart log | |
| file content has been added. This is no rsyslog bug if it occurs. | |
| As such, the rsyslog team advises against activating the freshStartTail | |
| option. | |
| closes https://github.com/rsyslog/rsyslog/issues/2464 | |
| - core: fix undefined behavior (unsigned computation may lead to value < 0) | |
| This was detected by LLVM UBSAN. On some platforms re-setting the rawmsg | |
| inside the message object could lead to invalid computation due to the | |
| fact the the computation was carried out as unsigned and only then | |
| converted to integer. | |
| No known problem in practice. | |
| - CI/QA: | |
| - improved Elasticsearch tests so they can now be run without system- | |
| installed ES service; also enables us to specify specific ES versions | |
| and should now make the tests executable inside a container | |
| ------------------------------------------------------------------------------ | |
| Version 8.35.0 [v8-stable] 2018-05-15 | |
| - imptcp: add ability to configure socket backlog | |
| this can be useful under heavy load. | |
| For a detailed discussion see https://github.com/rsyslog/rsyslog/pull/2561 | |
| Thanks to Maxime Graff for implementing this. | |
| - omfile: do not permit filename that only consists of whitespace | |
| - fmhash: new hash function module | |
| implements hash32() and hash64() functions | |
| Thanks to Harshvardhan Shrivastava for implementing these | |
| - some better error messages | |
| - imklog: add ratelimiting capability | |
| On Linux kernel logs are ratelimited only for messages using | |
| printk_ratelimit(). Some logs do not use this facility, so | |
| we ratelimit kernel ourselves. | |
| Thanks to Berend De Schouwer for the patch. | |
| - omkafka: added impstats counters for librdkafka returned statistics | |
| Adds: | |
| * statscallback counters | |
| * librdkafka failure and error counters | |
| * acked message counter | |
| Thanks to Abdul Waheed for implementing this. | |
| - imudp | |
| * use rsyslog message rate-limiter instead of home-grown one | |
| imudp introduced it's own (feature-limited) rate-limiting capability for | |
| message on disallowed senders before we had central rate-limiters | |
| inside rsyslog. Also, that code evolved from running on a single | |
| thread to running on multiple threads, which introduced data races | |
| and so made unreliable. | |
| Now we removed the old rate-limiting capability and depend on the | |
| system rate limiter for internal rsyslog messages. | |
| closes https://github.com/rsyslog/rsyslog/issues/2467 | |
| * add stats counter "disallowed" | |
| counts the number of messages discarded due to being received from | |
| disallowed senders | |
| see also https://github.com/rsyslog/rsyslog/issues/2467 | |
| - imrelp: add parameter "oversizeMode" | |
| Permits to instruct librelp how to handle oversize messages. The new default | |
| is to truncate messages. Previously, the connection was aborted, what often | |
| lead to stuck messages at the sender side. Now, there are three options passed | |
| down to librelp: | |
| * abort - same behavior as previously, connection is aborted on error | |
| * truncate - do not abort but instead truncate oversize message to | |
| configured max size | |
| * accept - accept all oversize messages (note: this can cause security issues, | |
| see doc for details) | |
| see also https://github.com/rsyslog/rsyslog/pull/1525#issuecomment-384179393 | |
| see also https://github.com/rsyslog/rsyslog/issues/2190 | |
| closes https://github.com/rsyslog/rsyslog/issues/2633 | |
| closes https://github.com/rsyslog/rsyslog/issues/1741 | |
| closes https://github.com/rsyslog/rsyslog/issues/1782 | |
| closes https://github.com/rsyslog/rsyslog/issues/2496 | |
| - core: consistent handling of oversize input messages | |
| In the community we frequently discuss handling of oversize messages. | |
| David Lang rightfully suggested to create a central capability inside | |
| rsyslog core to handle them. | |
| We need to make a distinction between input and output messages. Also, | |
| input messages frequently need to have some size restrictions done at | |
| a lower layer (e.g. protocol layer) for security reasons. Nevertheless, | |
| we should have a central capability | |
| * for cases where it need not be handled at a lower level | |
| * as a safeguard when a module invalidly emits it (imfile is an example, | |
| see https://github.com/rsyslog/rsyslog/pull/2632 for a try to fix it | |
| on the module level - we will replace that with the new capability | |
| described here). | |
| The central capability works on message submission, and so cannot be | |
| circumvented. It has these capabilities: | |
| * oversize message handling modes: | |
| - truncate message | |
| - split message | |
| this is of questionable use, but also often requested. In that mode, | |
| the oversize message content is split into multiple messages. Usually, | |
| this ends up with message segments where all but the first is lost | |
| anyhow as the regular filter rules do not match the other fragments. | |
| As it is requested, we still implemented it. | |
| - accept message as is, even if oversize | |
| This may be required for some cases. Most importantly, it makes | |
| quite some sense when writing messages to file, where oversize | |
| does not matter (accept from a DoS PoV). | |
| * report message to a special "oversize message log file" (not via the | |
| regular engine, as that would obviously cause another oversize message) | |
| This commit, as the title says, handles oversize INPUT messages. | |
| see also https://github.com/rsyslog/rsyslog/issues/2190 | |
| closes https://github.com/rsyslog/rsyslog/issues/2681 | |
| closes https://github.com/rsyslog/rsyslog/issues/498 | |
| Note: this commit adds global parameters: | |
| * "oversizemsg.errorfile", | |
| is used to specify the location of the oversize message log file. | |
| * "oversizemsg.report", | |
| is used to control if an error shall be reported when an oversize | |
| message is seen. The default it "on". | |
| * add global parameter "oversizemsg.input.mode" | |
| is used to specify the mode with which oversized messages will | |
| be handled. | |
| - omfwd: add support for bind-to-address for UDP | |
| To allow the same source address to be used regardless of the egress | |
| interface taken, an option is added for an address to bind the datagram | |
| socket to. Similarly to imudp, it is necessary to add an ipfreebind | |
| option which is set by default, so as to avoid an excess of errors at | |
| startup before the network interface has come up. This enhancement | |
| allows a usecase on networking devices, by which a source interface | |
| that is typically a loopback is specified, on which an address to bind | |
| to is configured. This is so that the same source address is used for | |
| all packets from rsyslog. | |
| Thanks to Mike Manning for the patch. | |
| - template systemd service file proposes higher permitted file handle limit | |
| Especially on busy systems the default are too low. Please keep in mind | |
| that on a very busy system even the now-proposed setting may be too low. | |
| Thanks to github user jvymazal for the patch. | |
| - imuxsock: replace select() call by poll() | |
| While extremely unlikely, imuxsock could abort if a file descriptor | |
| > 1024 was received during the startup phase (never occured in | |
| practice, but theoretically could if imfile monitored a large number | |
| of files and were loaded before imuxsock - and maybe other | |
| strange cases). | |
| see also https://github.com/rsyslog/rsyslog/issues/2615 | |
| - nsdsel_ptcp: replace select() by poll() | |
| This removes us of problems with fds > 1024. The performance will | |
| probably also increase in most cases. | |
| Note this is not a replacement for the epoll drivers, but a general | |
| stability improvement when epoll() is not available for some reason. | |
| see also https://github.com/rsyslog/rsyslog/issues/2615 | |
| closes https://github.com/rsyslog/rsyslog/issues/1728 | |
| closes https://github.com/rsyslog/rsyslog/issues/1459 | |
| - omprog: refactor tests, fix child closing issues | |
| Refactor omprog tests. Fix sync issues in these tests by | |
| using the feedback mode (confirmMessages=on) to synchronize | |
| the test with the external program. Closes #2403 (I hope) | |
| Fix omprog not properly closing child process when | |
| signalOnClose=on. Needed for the new tests. Closes #2599 | |
| Fix omprog not waiting for the child process to terminate | |
| when signalOnClose=off. Needed for the new tests. Closes #2600 | |
| Close all fds before executing the child even when valgrind | |
| is enabled (--enable-valgrind). Needed for the new tests. | |
| Fix memory leak when the xxxTransactionMark parameters were | |
| used. | |
| Thanks to Joan Sala for the patch. | |
| - core: config optimizer did not handle call_indirect | |
| This also caused the emission of an "internal error" error message | |
| closes https://github.com/rsyslog/rsyslog/issues/2665 | |
| - debug support: add capability to print testbench-specific timeout reports | |
| done by setting RSYSLOG_DEBUG_TIMEOUTS_TO_STDERR to "on" | |
| this is by default activated inside the testbench | |
| - mmgrok: fix potential segfault | |
| The modules used strtok(), which is not thread-safe. So it will potentially | |
| segfault when multiple instances are spawned (what e.g. happens on busy | |
| systems). | |
| This patch replaces strtok() with its thread-safe counterpart | |
| strtok_r(). | |
| see also https://github.com/rsyslog/rsyslog/issues/1359 | |
| - imrelp bugfix: maxDataSize could be set lower than maxMessageSize | |
| maxDataSize specifies the length which will still be accepted | |
| It previously could be set to any value, including values lower than the | |
| configured rsyslog max message size, which makes no sense. Now this is | |
| checked an error message is emitted if the size is set too low. | |
| - build system bugfix: build broken if liblogging-stdlog installed in custom path | |
| Thanks to Dirk Hörner for the patch. | |
| - core bugfix: segfault on queue shutdown | |
| if a ruleset queue is in direct mode, a segfault can occur during | |
| rsyslog shutdown. The root cause is that a direct queue does not | |
| have an associated worker thread pool, but the ruleset destructor | |
| does not anticipate that and tries to destruct the worker thread | |
| pool. It needs to do this itself, as otherwise we get a race | |
| between rulesets on shutdown. | |
| This was a regression from | |
| https://github.com/rsyslog/rsyslog/commit/3fbd901b3e6300010 | |
| closes https://github.com/rsyslog/rsyslog/issues/2480 | |
| - imfile bugfix: statefiles contain invalid JSON | |
| When imfile rewrites state files, it does not truncate previous | |
| content. If the new content is smaller than the existing one, the | |
| existing part will not be overwritten, resulting in invalid json. | |
| That in turn can lead to some other failures. | |
| closes https://github.com/rsyslog/rsyslog/issues/2662 | |
| - omfile bugfix: segfault if empty filename was given | |
| closes https://github.com/rsyslog/rsyslog/issues/2417 | |
| - fix build issues when atomic operations are not present | |
| for details, see https://github.com/rsyslog/rsyslog/pull/2604 | |
| - lmsig_ksils12 bugfix: build and static analyzer issues | |
| The module had a couple of problems building as well as some potential | |
| errors detected by clang static analyzer. These have been fixed. | |
| Thanks to Allan Park for the patch. | |
| closes https://github.com/rsyslog/rsyslog/issues/2517 | |
| - impstats bugfix: segfault if bound to non-existing ruleset | |
| segfault happens during shutdown; up until unload of impstats, | |
| rsyslog works correctly, except that no pstats are emitted. This | |
| can be considered to be expected, because the error message | |
| indicates the default ruleset is used instead. This is what | |
| now actually happens. | |
| closes https://github.com/rsyslog/rsyslog/issues/2645 | |
| - mmjsonparse bugfix: invalid container name was not detected | |
| in debug builds, this will trigger an assertion. In production | |
| builds, an rsyslog internal error is logged, but rsyslog | |
| continues to run. | |
| closes https://github.com/rsyslog/rsyslog/issues/2584 | |
| - mmkubernetes bugfixes: fix lnrules, add defaults, add test | |
| - Fix lnrules for CONTAINER_NAME | |
| - Add pkg check for lognorm >= 2.0.3 so we can set the macro | |
| to enable ln_loadSamplesFromString | |
| - Add some reasonable default values for parameters, such as | |
| kubernetesurl https://kubernetes.default.svc.cluster.local:443 | |
| - Clean up sample.conf configuration file | |
| Thanks to Richard Megginson for the patch set. | |
| - build system bugfix: --enable-atomic-operations did not work | |
| closes https://github.com/rsyslog/rsyslog/pull/2604 | |
| - bugfix: rsyslog aborts on startup when specific config errors are made | |
| The following errors must be made in rsyslog.conf: | |
| * no action present | |
| * a call statement is used on an undefined ruleset | |
| In this case, rsyslog emits an error message on the missing actions and | |
| then segfaults. Depending on memory layout, it may also continue to run | |
| but do nothing except accepting messages as no action is configured. | |
| This patch make rsyslog properly terminate after the error message. It | |
| is a change in behavior, but there really is no reason why a defunct | |
| instance should be kept running. | |
| closes https://github.com/rsyslog/rsyslog/issues/2399 | |
| - build system: remove no longer needed --enable-libcurl configure switch | |
| The --enable-libcurl switch was added to be able to disable libcurl | |
| functionality inside the rsyslog core, see 46f4f43. As libcurl is no | |
| longer used inside the core (due to introducing function modules), | |
| --enable-libcurl needs to be removed. | |
| closes https://github.com/rsyslog/rsyslog/issues/2628 | |
| - QA/CI | |
| * fixed races in some tests; root cause was that default enq timeout was too | |
| low - we may also see in the future that other tests also need adjustment | |
| (note that this is not a code problem but rather slow CI environments, | |
| so increasing the timeout to get to a stable test state is the absolutely | |
| correct thing to do) | |
| * enabled some additional useful compiler warnings | |
| * new test for diskqueue hitting configured disk space limit | |
| * new tests for omfile | |
| * added tests for mmkubernetes | |
| * added tests for some script functions that were missing them | |
| * made far more test compatible with FreeBSD, so that we now have fuller | |
| coverage there | |
| ------------------------------------------------------------------------------ | |
| Version 8.34.0 [v8-stable] 2018-04-03 | |
| - mmkubernetes: new module | |
| Thanks to Richard Megginson and Peter Portante for contributing the module. | |
| - rsyslog script: introduce loadable function modules | |
| rsyslog scripting can now also be extended via loadable modules - they | |
| provides functions (just like loadable input, output, ... modules) | |
| see also http://jan.gerhards.net/2018/03/loadable-rainerscript-functions.html | |
| - imfile: large refactoring of complete module | |
| This commit greatly refactors imfile internal workings. It changes the | |
| handling of inotify, FEN, and polling modes. Mostly unchanged is the | |
| processing of the way a file is read and state files are kept. | |
| This is about a 50% rewrite of the module. | |
| Polling, inotify, and FEN modes now use greatly unified code. Some | |
| differences still exists and may be changed with further commits. The | |
| internal handling of wildcards and file detection has been completely | |
| re-written from scratch. For example, previously when multi-level | |
| wildcards were used these were not reliably detected. The code also | |
| now provides much of the same functionality in all modes, most importantly | |
| wildcards are now also supported in polling mode. | |
| The refactoring sets ground for further enhancements and smaller | |
| refactorings. This commit provides the same feature set that imfile | |
| had previously and all existing CI tests pass, as do some newly | |
| created tests. | |
| Some specific changes: | |
| - bugfix: module parameter "sortfiles" ignored | |
| This parameter only works in Solaris FEN mode, but is otherwise | |
| ignored. Most importantly it is ignored under Linux. | |
| fixes https://github.com/rsyslog/rsyslog/issues/2528 | |
| - bugfix: imfile did not pick up all files when not present | |
| at startup | |
| fixes https://github.com/rsyslog/rsyslog/issues/2241 | |
| fixes https://github.com/rsyslog/rsyslog/issues/2230 | |
| fixes https://github.com/rsyslog/rsyslog/issues/2354 | |
| fixes https://github.com/rsyslog/rsyslog/issues/1716 | |
| - bugfix: directories only support "*" wildcard, no others | |
| fixes https://github.com/rsyslog/rsyslog/issues/2303 | |
| - bugfix: parameter "sortfiles" did only work in FEN mode | |
| fixes https://github.com/rsyslog/rsyslog/issues/2528 | |
| - provides the ability to dynamically add and remove files via | |
| multi-level wildcards | |
| see also https://github.com/rsyslog/rsyslog/issues/1280 | |
| - the state file name currently has been changed to inode number | |
| This will further be worked on in upcoming PRs | |
| see also https://github.com/rsyslog/rsyslog/issues/2231 | |
| - some enhancements were also done to CI tests, most importantly | |
| they were made more compatible with BSD | |
| Note that most of the mentioned bug fixes cannot be applied to older | |
| versions, as they fix design issues which are solved by the refactoring. | |
| Thus there are not separate commits for them. | |
| There are probably also a number of different issues fixed, which have | |
| not yet been full confirmed. Especially anyone having troubles with imfile | |
| and wildcards will benefit from the refactoring. | |
| closes https://github.com/rsyslog/rsyslog/issues/2359 | |
| - omelasticsearch: add support for CA cert, client cert auth | |
| This allows omelasticsearch to perform client cert based authentication | |
| to Elasticsearch. | |
| adds parameters: tls.cacert, tls.mycert, tls.myprivkey | |
| Thanks to Richard Megginson for the patch. | |
| - omfile-hardening: new contributed module for "omfile hardened operations" | |
| This extends omfile with features considered useful for hardening. Comes | |
| at the expense of performance loss and changed semantics. | |
| Thanks to Mikko Kortelainen for contributing this work. | |
| - stream/bugfix: memory leak on stream open if filename as already generated | |
| this can happen if imfile reads a state file. On each open, memory for the | |
| file name can be lost. | |
| We detected this while working on imfile refactoring, so there is no related | |
| bug report. No specific test has been crafted, as the refactored imfile | |
| tests catch it (as soon as they are merged). | |
| - bugfix/omkafka: did not build on platforms without atomic operations | |
| Thanks to github user bruce87en for the patch | |
| - bugfix/core/ratelimiting: SystemLogRateLimitBurst was limited to 65535 | |
| rsyslog uses unsigned short for configuration setting SystemLogRateLimitBurst. | |
| Being just 16 bits, unsigned short cannot hold values bigger than 65535. in a | |
| practical setting rsyslog misbehaved with SystemLogRateLimitBurst being bigger | |
| than 65535. | |
| Thanks to github user KaleviKolttonen for the patch. | |
| - bugfix imfile: memory leak in readMode 0 | |
| closes https://github.com/rsyslog/rsyslog/issues/2421 | |
| - bugfix omfile: some error messages had parameters in wrong order | |
| which made the message look strange, but still readable | |
| Thanks to Hans Rakers for the patch. | |
| - bugfix omprog: file handle leak | |
| There was a fd leak in the feedback feature added in v8.31.0 (github PR #1753). | |
| The leak occurred when omprog was used with the confirmMessages=on setting | |
| and no output setting. One fd was leaked every time the external program was | |
| started. | |
| Thanks to Joan Sala for the patch. | |
| - bugfix imuxsock: data alignment problems | |
| gcc did rightly complain that the cred and ts pointers would cause | |
| alignment problems, so they were converted to structs and the necessary | |
| data was memcpy()'ed to them. | |
| the aux[] buffer was also potentially misaligned, so making a union | |
| out of it and struct cmsghdr insured it was properly aligned. | |
| The problems were especially visible on alpha and ia64 machines. | |
| Thanks to Jason Duerstock for the fix. | |
| - bugfix testbench: some test were accidentally not executed | |
| Thanks to Kasumi Hanazuki for the patch. | |
| - bugfix tcp subsystem: keepalive settings mixed up | |
| TCPKeepAliveIntvl and TPCKeepAliveTime were switched. This is now correct and | |
| thus causes a CHANGE OF BeHAVIOR of these settings. We applied this change only | |
| after careful consideration of the effect. The contributor Alexandre Pierret | |
| explained the situation as such: | |
| "From my side, I work with thousands of servers centralizing logs to rsyslog | |
| servers in tcp. All of them are running RHEL 6 and 7. The default rsyslog settings | |
| in RHEL is TCPKeepAlive off. Since there are thousands of connections on the | |
| rsyslog servers, I had to turn the TCPKeepAlive on to setup an aggressive policy | |
| regarding ghost connections (following firewall tcp-timeout issue). Basically, | |
| I set up: intvl=10 - probe=5 - time=2 If intvl and time are switched, it won't | |
| break anything. It will just send 5 more empty tcp packets for 10 seconds (5 | |
| probe x 2 seconds), which is painless (any professional grade 100€ router can | |
| send more than 1.000.000 PPS). For 3000 servers, it means 3000 pps for 5 | |
| seconds (3000 servers x 2 back-and-forth / 2 seconds). Let's take another worst | |
| example: intvl=3600 - probe=5 - time=1. It means one keepalive every hour and | |
| a 5 retry every 1s after a network issue. If the time and intvl values are | |
| switched, it will generate 1 keepalive every seconds, It's a LOT more, but | |
| after 5 probe or packet, it will stop. To summarize, I think it won't | |
| break anything: | |
| * Keepalive is off by default is many linux distribution | |
| * When we enable it, it's to set up an aggressive policy. And setting up a | |
| 10-5-1, 60-5-2, 2-5-60 or 1-20-1 policy is almost the same. | |
| Bonus: For people setting up their rsyslog from the documentation, it will | |
| now work as expected." | |
| This was convincing, and we actually think that the fast majority of users set | |
| up keepalive based on the doc and did never verify it actually worked as | |
| expected. So we think that in all those cases, rsyslog will finally work as | |
| intended. So we consider it justified to "change the behavior" here. | |
| full discussion in PR: https://github.com/rsyslog/rsyslog/pull/2367 | |
| Thanks to Alexandre Pierret for analyzing the situation and providing the | |
| patch. | |
| - fix some cosmetic issues found by lgtm.com static code analyzer | |
| e.g. header file guard not correctly set - if you really are interested in | |
| details, check git log | |
| - CI | |
| * add build test without atomic operations - now catches missing mutex macros | |
| * add lgtm.com static analyzer (automatically called via GitHub PR) | |
| * improved stability of global-umask.sh test, which unnecessarily used | |
| wildcards for test output file checking. | |
| Thanks to Kasumi Hanazuki for the patch. | |
| * added some test for omprog with transactional interface | |
| Thanks to Joan Sala for the new tests. | |
| * fixed some omjournal tests which did not properly check result | |
| ------------------------------------------------------------------------------ | |
| Version 8.33.1 [v8-stable] 2018-03-06 | |
| - 8.33.0 tarball release was actually pre-8.33.0 | |
| ... and so did not contain all features. This alone made a re-release | |
| necessary, which is what now happens with 8.33.1. | |
| Note: the git 8.33.0 label was correctly applied, "just" the tarball | |
| was wrong. | |
| - devcontainer: use some more sensible defaults | |
| and add ability to specify generic docker run options | |
| this makes integration into CI (and other scripting) easier | |
| - fix problems with make dist | |
| make dist did not package everything that was needed for CI, thus | |
| resulting in make check failures if build from tarball. | |
| Thanks to Thomas D. (whissi), and Michael Biebl for alerting us on the | |
| problem, providing advise and some of the patches. We also added addt'l | |
| patches ourselves. The problem occured as the CI check for tarball | |
| completeness was more or less disabled a couple of weeks ago, which | |
| unfortunately went unnoticed. We have also applied some more safeguards | |
| to detect such problems in the future. | |
| ------------------------------------------------------------------------------ | |
| Version 8.33.0 [v8-stable] 2018-02-20 | |
| - auto-detect if running inside a container (as pid 1) | |
| adjust defaults in this case to be more container-friendly | |
| - config: add include() script object | |
| This permits to include files (like legacy $IncludeConfig) via a | |
| script object. Needless to say, the script object offers more | |
| features: | |
| - include files can now be | |
| - required, with rsyslog aborting when not present | |
| - required, with rsyslog emitting an error message but otherwise | |
| continuing when not present | |
| - optional, which means non-present include files will be | |
| skipped without notice | |
| This is controlled by the "mode" parameter. | |
| - text can be included form e.g. an environment variable | |
| --> ex: include(text=`echo $ENVVAR`) | |
| This finally really obsoletes $IncludeConfig. | |
| closes https://github.com/rsyslog/rsyslog/issues/2151 | |
| - template: add option to generate json "container" | |
| This enables easy JSON generation via template. | |
| This commit also corrects an issue with the constant "jsonf" | |
| format. That was recently added, and the implementation problem | |
| only became visible when used inside a larger json object. No | |
| officially released code is affected, thought - so it really | |
| is just a side-note. | |
| closes https://github.com/rsyslog/rsyslog/issues/2347 | |
| - core/template: add format jsonf to constant template entries | |
| closes https://github.com/rsyslog/rsyslog/issues/2348 | |
| - config: add ability to disable config parameter ("config.enabled") | |
| For auto-generated configs, it is useful to have the ability to disable some | |
| config constructs even though they may be specified inside the config. This | |
| can now be done via the ```config.enabled``` parameter, applicable to all | |
| script objects. If set to ```on``` or not specified, the construct will be | |
| used, if set to any other value, it will be ignored. This can be used | |
| together with the backtick functionality to configure enable and disable | |
| from either a file or environment variable. | |
| closes https://github.com/rsyslog/rsyslog/issues/2431 | |
| - script: permit to use environment variables during configuration | |
| new constant string type "backticks", inspired by sh | |
| (sample: `echo $VARNAME`). | |
| - new global config parameter "shutdown.enable.ctlc" | |
| permits to shutdown rsyslog via ctl-c; useful e.g. in containers | |
| - config optimizer: detect totally empty "if" statements and optimize | |
| them out | |
| - template: constant entry can now also be formatted as json field | |
| This enhancements permits to craft clean templates that generate JSON, | |
| e.g. for ElasticSearch consumption (or any other REST API) | |
| - omstdout: support for new-style configuration parameters added | |
| - core: set TZ on startup if not already set | |
| In theory, TZ should be set by the OS. Unfortunately, this seems | |
| to be not the case any longer on many Linux distros. We now check | |
| it and set it appropriate if not already given. | |
| Thanks to github user JPvRiel for providing an excellent explanation | |
| of the reasoning for this and how to work around it. | |
| closes https://github.com/rsyslog/rsyslog/issues/2054 | |
| - imjournal bugfix: file handle leak during journal rotation | |
| Thanks to Peter Portante for the patch | |
| see also: https://github.com/rsyslog/rsyslog/pull/2437 | |
| - lmsig_ksils12 bugfix: dirOwner and dirGroup config was not respected | |
| - script bugfix: replace() function worked incorrectly in some cases | |
| If the end of the message was similar to the replacement string, parts | |
| of the string could (not always) be missing. | |
| Thanks to Yaroslav Bo for the patch. | |
| - build system bugfix: --disable-libcurl did not work | |
| Thanks to Dan Molik, Thomas D. (whissi), and Michael Biebl for the patches. | |
| - fixed build issues on Alpine Linux | |
| - core bugfix: misaddressing in external command parser | |
| This parser is used whenever a module (e.g. omprog) needs to process | |
| command lines. If command parameters were given, memory misaddressing | |
| occured. This could lead to a segfault. | |
| This is a regression in 8.32.0. | |
| closes https://github.com/rsyslog/rsyslog/issues/2408 | |
| - core bugfix: small memory leak in external command parser | |
| This parser is used whenever a module (e.g. omprog) needs to process | |
| command lines. On each action definitions for actions that use the | |
| parser a small amount of memory was leaked. It is an uncritical leak | |
| as it only occurs during config parsing. So it leaks a couple of | |
| KiB during startup but does not grow during actual message processing. | |
| This is a regression in 8.32.0. | |
| - core bugfix: string not properly terminated when RFC5424 MSGID is used | |
| This could lead to misaddressing when the jsonmesg property was used. | |
| closes https://github.com/rsyslog/rsyslog/issues/2396 | |
| - bugfix: strndup() compatibility layer func copies too much | |
| The function did not obey the upper limit, effectively becoming | |
| a strdup(). This was only noticed when the compatibility layer | |
| was required, most importantly on Solaris 10. | |
| - CI system | |
| - we now use well-defined containers for parts of the CI runs | |
| - now also build test under Alpine Linux | |
| - test added for omprog feedback feature | |
| ------------------------------------------------------------------------------ | |
| Version 8.32.0 [v8-stable] 2018-01-09 | |
| - NEW BUILD REQUIREMENTs: | |
| * libfastjson 0.99.8 is now required; older versions lead to bugs in rsyslog | |
| * libczmq >= 3.0.2 is now required for omczmq | |
| This was actually required for quite some while, but not properly checked | |
| during configure run. If the lib was older, build failed. Now configure | |
| detects this and emits the appropriate error message. | |
| * libcurl is now needed for rsyslog core | |
| due to new script function http_request(). This can be turned off by the | |
| new configure option --disable-libcurl. If so, http_request() is not | |
| available. | |
| - rsyslogd: add capability to specify that no pid file shall be written | |
| Command line option -iNONE provides this capability. This utilizes the | |
| pre-existing -i option, but uses the special name "NONE" to turn of the | |
| pid file check feature. Turning off is useful for systems where this no | |
| longer is needed (e.g. systemd based). | |
| closes https://github.com/rsyslog/rsyslog/issues/2143 | |
| - ompgsql: considerable enhancements | |
| The PostgreSQL output module was woefully out-of-date the following | |
| list is changes made to update the module to current Rsyslog standards. | |
| * allow for v6 configuration syntax | |
| * configurable ports | |
| * support transactional interface | |
| * push db connection into workers (libpq is threadsafe) | |
| * enable module testing on travis | |
| * ensure configuration syntax backwards compatibility | |
| * formatting around postgres core templating | |
| * use new test conventions | |
| * add new configuration syntax test | |
| * add valgrind tests for new and old syntax | |
| * add threading tests | |
| * add action queue long running tests | |
| * add action queue valgrind test | |
| Thanks to Dan Molik for contributing this great enhancement! | |
| - build system: removed --enable-rtinst configure option | |
| This was a historic debugging option which has been superseded by | |
| newer tools like valgrind, helgrind, ASAN, TSAN, etc... | |
| - pmrfc3164: support for headerless messages | |
| pmrfc3164 now detects if the first non-SP, non-HT character is either | |
| '{' or '[' and if so assume that no header (TAG, HOSTNAME, DATE) is | |
| given. If so, it uses defaults for these values. The full message is | |
| used as the MSG part in that case. Note that an initial PRI may still | |
| be specified. | |
| This follows the trend to send JSON messages via syslog without any | |
| header. We use default header values in this case. | |
| This right now is a bit experimental; we may roll it back if | |
| problems show up in practice. | |
| closes https://github.com/rsyslog/rsyslog/issues/2030 | |
| - omhiredis: add option to use RPUSH instead of LPUSH | |
| see also https://github.com/rsyslog/rsyslog/issues/1625 | |
| - mmexternal improvements | |
| * better error reporting if to-be-executed script cannot be executed | |
| * some general cleanup | |
| * better redirection of stdin/out/err for the executed script | |
| * bugfix: argv[0] of the script was missing if command line parameters | |
| were not specified | |
| - omprog: refactored, code shared with mmexternal moved to common object | |
| - logctl tool: refactor to support newer rsyslog standards | |
| * Made the logctl usertool ISO C90 compliant. | |
| * Made the logctl usertool use a homogeneous coding style. | |
| Thanks to Hugo Soszynski for contributing this work (as well as | |
| suggesting some workarounds related to libmongoc/libbson). | |
| - imfile: added support for Solaris File Event notification (FEN) | |
| also improves performance under Solaris, with FEN imfile provides | |
| features equivalent to inotify mode | |
| - core/action: new parameter "action.errorfile" | |
| permits to write failed messages to an "error file" so that they | |
| can be reviewed and potentially be reprocessed | |
| - imfile: added new module parameter "sortFiles" | |
| This permits to process newly created files in sorted order. | |
| - imuxsock: improved status reporting: socket name received from systemd | |
| Providing an indication of what we got from systemd facilitates problem | |
| analysis. | |
| - build system: added new testbench configure switches | |
| now --enable-testbench1 and --enable-testbench2 exists which permit | |
| to enable/disable parts of the testbench. By default, both are on | |
| when --enable-testbench is given. For full testbench coverage, both | |
| options must be given. These options are necessary because under | |
| Travis we hit the max runtime for tests and needed to split tests | |
| over multiple incarnations. | |
| - mmpstrucdata: new parameter "sd_name.lowercase" | |
| to permit preserving case for structured data identifiers | |
| Special thanks to github user alanrobson for the initial commit that | |
| preserves case (on which we based other work). | |
| - omfile: add module-global option "dynafile.donotsuspend" | |
| this permits to enable SUSPENDing dynafile actions. Traditionally, | |
| SUSPEND mode was never entered for dynafiles as it would have blocked | |
| overall processing flow. Default is not to suspend (and thus block). | |
| closes https://github.com/rsyslog/rsyslog/issues/2236 | |
| - testbench: add a capability to turn off libfaketime tests via configure | |
| Unfortunately, libfaketime becomes more and more problematic in newer | |
| versions and causes aborts on some platforms. This provides the ability | |
| to turn it off via --disable-libfaketime. | |
| In the longer term, we should consider writing our own replacement. | |
| - testbench: name valgrind tests consistently | |
| all valgrind tests now end in -vg.sh | |
| - RainerScript: add function parse_json() | |
| - RainerScript: add function substring() | |
| - RainerScript: add function http_request() | |
| - RainerScript: add function previous_is_suspended() | |
| This function returns a boolean indicating if the previous action is | |
| suspended (0 - no, 1 - yes). This is useful if arbitrary processing | |
| (other than calling actions) should be made depending on that state. | |
| A prime example for this is probably calling a ruleset. | |
| closes https://github.com/rsyslog/rsyslog/issues/1939 | |
| - Patches from BSD projects have been imported | |
| ... as far as they still apply. Some patches that patched BSD-specific | |
| code were broadened to be more generic. | |
| - script bugfix: invalid function names were silently ignored | |
| no error message was produced | |
| thanks to Matt Ezell for providing the patch. | |
| - rainerscript: add int2hex() function | |
| - rainerscript: add is_time() function | |
| Thanks to Stephen Workman for implementing this. | |
| - RainerScript: add function script_error() and error-reporting support | |
| This permits script functions that could fail to report errors back, so | |
| that the user can handle them if desired. We use an errno-style of | |
| interface. That means script_error() needs to be called after functions | |
| that supports it. It will return either 0 (all OK) or something else | |
| (an error condition). | |
| The commit also modifies the parse_time() function to use the new | |
| interface. First of all, this is useful for users, and secondly we | |
| needed a capability to craft a testbench test. | |
| closes https://github.com/rsyslog/rsyslog/issues/1978 | |
| - testbench: fixed build problem of testbench tools under Alpine Linux | |
| - added --enable-libsystemd configure option to enforce use of libsystemd | |
| so we can fail the build on platforms where this is required | |
| - core/glbl: remove long-unused option $optimizeforuniprocessor | |
| This was still available, but had no effect (for ~10 years or so). The | |
| plumbing is now removed. If someone tries to use the option, an | |
| error message is generated. | |
| closes https://github.com/rsyslog/rsyslog/issues/2280 | |
| - core/queue: emit better status messages at rsyslog shutdown | |
| this helps to diagnose issue - unfortunately we need more work to ensure | |
| that the messages always make it to the user. This is a start and | |
| hopefully useful at least for the testbench, possibly more. | |
| - fixed a couple of build issues with gcc-7 (in less frequently used modules) | |
| - fixed a couple of build issues on the arm platform (actually raspbian) | |
| - impstats: fix invalid counter definitions for getrusage() reporting | |
| some of the counters were defined as int (32 bit) vs. intctr_t (64 bit). | |
| On some platforms "long" seems to be 64bit, and getrusage() provides | |
| what we store as int via long. So this caused truncation and/or overflow. | |
| This had undefined effects. Most often, everything worked fine | |
| for values smaller than 2^31 but sometimes we got negative values. | |
| closes https://github.com/rsyslog/rsyslog/issues/1517 | |
| - imudp bugfix: potential segfault in ratelimiting | |
| The rate-limiter inside imudp was not set to be thread safe, but was | |
| used across multiple threads. This worked in default configuration, | |
| but failed when RepeatedMsgReduction was set to "on". | |
| Note that it in general is a bug to use a rate-limiter in | |
| non-threadsafe mode across multiple threads. This also causes invalid | |
| rate limiting counts in the default case. | |
| closes https://github.com/rsyslog/rsyslog/issues/441 | |
| fixes https://github.com/rsyslog/rsyslog/issues/2132 | |
| - imptcp bugfix: access to free'ed memory | |
| When notifyconnectionclose was on, a string buffer was accessed immediately | |
| after it was freed (as part of the connection close operation). | |
| Detected by LLVM ASAN. | |
| - mmanon bugfix: fix wrong ipv6 embedded recognition | |
| mmanon recognized IPv6 with embedded IPv4 that have too few (16 bit) fields. | |
| example: 13:abd:45:0.0.0.0 | |
| closes https://github.com/rsyslog/rsyslog/issues/2357 | |
| - imfile bugfix: not detecting files in directory when wildcards are used. | |
| When directories and files are created at the same time, | |
| imfile may missed subdirs or file if the machine is on high load. | |
| The handling has been enhanced to scan newly created directories ALWAYS for | |
| matching files. | |
| fixes https://github.com/rsyslog/rsyslog/issues/2271 | |
| However there still exist problems with multilevel directory configurations. | |
| Details are discussed in https://github.com/rsyslog/rsyslog/issues/2354 | |
| Fixes for the remaining issues are expected for 8.33.0. | |
| - script bugfix: improper string-to-number conversion for negative numbers | |
| - core/action bugfix: 100% CPU utilization on suspension of output module | |
| Triggering condition: | |
| * output module using the legacy transaction interface | |
| (e.g. omelasticsearch, omlibdbi) | |
| * output module needs to suspend itself | |
| In these cases, rsyslog enters a busy loop trying to resolve the | |
| suspend condition. The bug is rooted in rsyslog core action code. | |
| This patch fixes it by inserting a 1-second sleep during calls | |
| to the resume handler. | |
| Note: we cannot sleep exactly as long as tryResume needs. This | |
| would require larger refactoring, which probably is not worth for | |
| the legacy interface. The current solution is almost as good, as | |
| the one second sleep has very little overhead on a real system. | |
| Thus we have chosen that approach. | |
| This patch now also ensures that failed messages are properly | |
| handled and do not cause eternal hang. | |
| closes https://github.com/rsyslog/rsyslog/issues/2113 | |
| - core/variables bugfix: bare $! cannot be used in set statement | |
| fixes https://github.com/rsyslog/rsyslog/issues/326 | |
| - core bugfix: auto commit of actions improperly handled | |
| The internal state variable bHadAutoCommit was handled in thread-unsafe way | |
| and most probably caused (hard to see) issues when action instances were | |
| run on multiple worker threads. It looks like the state variable was | |
| forgotten to move over to worker state when action workers were introduced. | |
| closes https://github.com/rsyslog/rsyslog/issues/2046 | |
| - core bugfix: filename length limitation of 199 bytes | |
| file names (including path names) longer than 199 bytes could not be | |
| handled at many places. This has now been uplifted to 4KiB after careful | |
| review for the largest size supported among all relevant platforms. | |
| - core bugfix: undefined behavior due to integer overflow | |
| when searching strings, we may have an (unsigned) integer overflow | |
| which can lead to misaddressing. | |
| Detected by clang ubsan. | |
| - core bugfix: race on LocalHostIP property during startup | |
| The way the default was set could lead to a race if e.g. two internal | |
| messages were emitted at startup when the property was not yet set. This | |
| has been seen to happen in practice. It had no bad effect except a very | |
| small stationary memory leak, but made using thread analyzers unreliable | |
| (as it was rightfully detected as a problem). | |
| closes https://github.com/rsyslog/rsyslog/issues/2012 | |
| - bugfix: potential segfault on startup | |
| timezone info table was "sorted" even though it may be NULL. There is | |
| no practical case known where this lead to an actual abort, but in | |
| theory it would be possible. If so, it would happen immediately on | |
| startup. | |
| Detected by clang ubsan. | |
| - omhiredis bugfix: rsyslog segfault on startup if no template is specified | |
| - omprog bugfix: argv[0] not set when using binary without arguments | |
| When using the omprog plugin with a binary without arguments, argv[0] (the | |
| name of the binary) is not set, causing binaries that depend on this value | |
| being set to crash or misbehave. | |
| This patch also mildly refactors omprog argv[] creations, albeit some more | |
| refactoring would be useful. | |
| closes https://github.com/rsyslog/rsyslog/issues/1858 | |
| - core: refactoring of rsyslog's cstr "class" | |
| Function cstrGetSzStrNoNULL shall modified the string buffer on each call, | |
| albeit it is considered a "read-only" function. It currently adds a '\0' | |
| at the end. This is bad, especially when multiple threads access the same | |
| string. As always the same data is written, it was not causing real issues | |
| (except unnecessary cache writes), but it polluted the thread debugger and | |
| as such prevent more elaborate automated tests. | |
| - parent directory creation function refactored | |
| This should not cause any change of behavior, but is worth noting in case | |
| we see a regression not caught by the CI system. | |
| - mmsnmptrapd bugfix: potential misaddressing | |
| This only occured in situations when the system was totally out of memory. | |
| - imkafka: fix potential small resource leak | |
| If rdkafka handle cannot fully populated, cleanup is added. Previously, we | |
| could leak a handle if e.g. no brokers were available. Note that this was | |
| a cosmetic leak, as proper processing is not possible in any case and the | |
| leak is once per listener, so not growing. But we now also ensure that | |
| proper error reporting and handling is done in any case. Previously, we | |
| may have some misleading error messages if the defunct kafka handle was | |
| used. | |
| closes https://github.com/rsyslog/rsyslog/issues/2084 | |
| - imkafka bugfix: do not emit error message on regular state | |
| This was misunderstood as an error state and could spam the system | |
| log considerably. Regression from 8.31.0. | |
| - omkafka: expose operational status to user where useful | |
| omkafka emits many useful operational status messages only to the debug | |
| log. After careful review, we have exposed many of these as user error | |
| and warning message (ex: librdkafka queue full, so user knows why we | |
| suspend the plugin temporarily). This may have made the module too | |
| chatty. If so, one can always filter out messages via configuration. And | |
| if we really went overboard, we can undo those parts with the next | |
| release. So it's better to give a bit more than less, as this definitely | |
| eases troubleshooting for users. | |
| closes https://github.com/rsyslog/rsyslog/pull/2334 | |
| - omkafka bugfix: potential message duplication | |
| If a message that already failed in librdkafka was resubmitted and that | |
| resubmit also failed, it got duplicated. | |
| - omkafka: fix multithreading | |
| omkafka has several issue if multiple worker instances are used. This commit | |
| actually make the module use a single worker thread at max. Reasoning: | |
| Librdkafka creates background threads itself. So omkafka basically needs to move | |
| memory buffers over to librdkafka, which then does the heavy hauling. As such, we | |
| think that it is best to run max one wrkr instance of omkafka -- otherwise we just | |
| get additional locking (contention) overhead without any real gain. As such, | |
| we use a global mutex for doAction which ensures only one worker can be active | |
| at any given time. That mutex is also used to guard utility functions (like | |
| tryResume) which may also be accessed by multiple workers in parallel. | |
| Note: shall this method be changed, the kafka connection/suspension handling needs | |
| to be refactored. The current code assumes that all workers share state information | |
| including librdkafka handles. | |
| closes https://github.com/rsyslog/rsyslog/issues/2313 | |
| - omkafka bugfix: potential misaddressing | |
| < |