Permalink
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2104.0 (aka 2021.04) 2021-04-?? | |
- 2021-02-22: test bugfix: some tests did not work with newer TLS library versions | |
Newer versions provide TLS versions that cannot be disabled in older versions as they | |
are unknown there. This is solved by setting restrictions in multiple steps. For | |
older library versions, the final step will error out, but the other one be applied. | |
This permits to achieve proper test results. | |
closes: https://github.com/rsyslog/rsyslog/issues/4534 | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2102.0 (aka 2021.02) 2021-02-16 | |
- 2021-02-15: omfwd: add stats counter for sent bytes | |
Thanks to John Chivian for suggesting this feature. | |
- 2021-02-15: omfwd: add error reporting configuration option | |
RSyslog on a plain TCP cannot guarantee the message delivery | |
without using RELP protocol. Besides that the logs may be | |
flooded with connection errors making the rest of messages | |
difficult to find. To alleviate the problem (see issue 3910), | |
this patch adds a configuration option that enables to reduce | |
the number of network errors logged and reported. | |
For example, if each 10th network error message should be logged, | |
the rsyslog configuration has to be updated as follows. | |
action(type="omfwd" Target="<IP_ADDR>" Port="<PORT>" Protocol="tcp" ConErrSkip="10") | |
Thanks to Libor Bukata for the patch. | |
- 2021-02-15: action stats counter bugfix: failure count was not properly incremented | |
In some cases the counter was not incremented, most notably with transaction-enabled | |
actions. | |
Thanks to github user thinkst-marco for the patch. | |
- 2021-02-15: action stats counter bugfix: resume count was not incremented | |
And so it always stayed at zero. | |
Thanks to github user thinkst-marco for the patch. | |
- 2021-02-15: omfwd bugfix: segfault or error if port not given | |
If omfwd is configured via RainerScript config format and the "port" | |
parameter is not given, a segfault will most likely happen on | |
connection establishment for TCP connections. For UDP, this is | |
usually not the case. | |
Alternatively, in any case, errors may happen. | |
Note that the segfault will usually happen right on restart so this | |
was easy to detect. | |
We did not receive reports from practice. Instead, we found the bug | |
while conducting other work. | |
- 2021-01-29: lookup table bugfix: data race on lookup table reload | |
A data race could happen when a lookup table was reloaded. We found | |
this while moving to newer version of TSAN, but have no matching | |
report from practice. However, there is a potential for this to cause | |
a segfault under "bad circumstances". | |
- 2021-01-18: testbench modernization | |
Bump dependency versions, use newer distro versions for some tests. | |
Make kafka distcheck separate to help diagnose flaky kafka tests. | |
- 2021-01-16: testbench: fix invalid sequence of kafka tests runs | |
kafka tests can not run well in parallel (mostly due to ressource | |
constraints on CI machines). Accidentally, this was not enforced for | |
one of the tests. That could lead to random failures and false positives. | |
- 2021-01-14: testbench: fix kafkacat issues | |
The kafkacat tool has an upper limit of how many messages it can send | |
at once. Going over that limit causes messages loss. The exact limit | |
seems to depend on the environment. This causes testbench false positives. | |
This commit fixes two related issues: | |
- errors during kafkacat run were not detected - this has been added | |
- we now have a "max messages at once" setting, after which kafkacat | |
is restarted for the next batch of messages. It currently is set | |
to 25,000 msgs per incarnation. All tests loop now to send the | |
required number of messages. This has been fixed at the testbench | |
framework level, so no need to adjust individual tests. | |
- 2021-01-14: testbench: fix year-dependendt clickhouse test | |
A test had the year value hardcoded and as such failed whenever the | |
year changed. This patch corrects that. | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2012.0 (aka 2020.12) 2020-12-08 | |
- 2020-12-07: testbench bugfix: some tests did not work in make distcheck | |
- certificate file missing in dist tarball | |
- some test cases did not properly specify path to cert file | |
Thanks to Michael Biebl for alerting us and providing part of | |
the fix. | |
closes https://github.com/rsyslog/rsyslog/issues/4446 | |
- 2020-12-07: immark: rewrite with many improvements | |
- mark message text can now be specified | |
- support for rulesets | |
- support for using syslog API vs. regular internal interface | |
- support for output template system | |
- ability to specify is mark message flag can be set | |
- minor changes and improvements | |
- 2020-11-30: usability: re-phrase error message to help users better understand cause | |
see also https://github.com/rsyslog/rsyslog/issues/3910 | |
- 2020-11-10: add new system property $now-unixtimestamp | |
Among others, this may be used as a monotonic counter | |
for doing load-balancing and other things. | |
Thanks to Nicholas Brown for suggesting this feature. | |
- 2020-11-04: omfwd: add new rate limit option | |
Adding new rate limit option to omfwd for rate limiting | |
syslog messages sent to the remote server | |
ratelimit.interval: | |
Specifies the rate-limiting interval in seconds. | |
Default value is 0, which turns off rate limiting. | |
ratelimit.burst | |
Specifies the rate-limiting burst in number of messages. | |
closes https://github.com/rsyslog/rsyslog/issues/4423 | |
Thanks to Dinesh-Ramakrishnan for the patch. | |
- 2020-11-03: omfwd bug: param "StreamDriver.PermitExpiredCerts" is not "off" by default | |
The default behaviour of expired certificates of stream driver in TLS mode, should | |
have been that the see tcp transmission is closed due to expired certificates, and | |
error messages emited in rsyslog status. This was not the case. That in turn could | |
lead to permitting sessions which should not be permitted. | |
Thanks to Vincent Zhu for alerting us and providing a great problem analysis | |
closes: https://github.com/rsyslog/rsyslog/issues/4425 | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2010.0 (aka 2020.10) 2020-10-20 | |
- 2020-10-13: gnutls TLS subsystem bugfix: handshake error handling | |
If the tls handshake does not immediatelly finish, gnutls_handShake is called in | |
doRetry handler again. However the error handling was not | |
complete in the doRetry handler. A failed gnutls_handShake call | |
did not abort the connection and properly caused unexpected | |
problems like in issues: | |
https://github.com/rsyslog/rsyslog/issues/4270 | |
https://github.com/rsyslog/rsyslog/issues/4288 | |
- 2020-10-13: core/msg bugfix: memory leak | |
There is a missing call to json_object_put(json) if the call to | |
jsonPathFindParent() failed. It's leaking memory. Depending on workload and config, | |
this leak can potentially grow large (albeit we did not see reports from practice). | |
Thanks to Julien Thomas for the patch. | |
- 2020-10-13: core/msg bugfix: segfault in jsonPathFindNext() when <root> not an object | |
The segfault gets happens when <bCreate> is 1 and when the <root> | |
container where to insert the <namebuf> key is not an object. | |
Here is simple reproducible test case: | |
// ensure we start fresh | |
// unnecessary if there was no previous set | |
unset $!; | |
set $! = ""; | |
set $!event!created = 123; | |
Thanks to Julien Thomas for the patch. | |
- 2020-10-13: openssl TLS subsystem: improvments of error and status messages | |
Adding error logs at the ssl handshake failure scenarios. | |
Adding the header "nsd_ossl:" tag to these logs to identify | |
the origin module from which logs are generated. | |
Thanks to Anusha Pai G for the patch. | |
- 2020-10-06: add 'exists()' script function to check if variable exists | |
This implements a way to check if rsyslog variables (e.g. '$!path!var') is | |
currently set of not. | |
Sample: if exists($!somevar) then ... | |
closes https://github.com/rsyslog/rsyslog/issues/4385 | |
- 2020-10-03: core bugfix: do not create empty JSON objects on non-existent key access | |
Performing a condition (eg: check for an empty string) on a subtree key that do not | |
exists (depth > 1 from the root container), creates an empty "parent" object. | |
Depending on your context, you may end up with (kind of...) annoying garbage when | |
producing object documents (for instance to index in ES). | |
Also fixes a hypothetical hang condition with an almost (?) unused plugin parameter | |
passing mode, for details see | |
https://github.com/rsyslog/rsyslog/issues/4436 | |
closes https://github.com/rsyslog/rsyslog/issues/4430 | |
Thanks to Julien Thomas for the patch. | |
- 2020-09-28: gnutls subsysem bugfix: potential hang on session closure | |
Some TLS servers don't reply to graceful shutdown requests "for | |
optimization". This results in rsyslog's omfwd+gtls client to wait | |
forever for a reply of the TLS server which never comes, due to shutting | |
down the connection with gnutls_bye(GNUTLS_SHUT_RDWR). | |
On systemd systems, commands such as "systemctl restart rsyslog" just | |
hang for 1m30 and rsyslogd gets killed upon timeout by systemd. | |
This is fixed by replacing the call to gnutls_bye(GNUTLS_SHUT_RDWR) by calls to | |
gnutls_bye(GNUTLS_SHUT_WR) which is sufficient and doesn't wait for a | |
server reply. | |
As an example, Kiwi Syslog server is known to cause this issue. | |
Thanks to Renaud Métrich for the patch. | |
- 2020-09-23: core/network bugfix: obey net.enableDNS=off when querying local hostname | |
Local hostname resolution used DNS queries even if the enableDNS was set to off, and | |
this could cause unexpected delays in the HUP signal handling if the DNS server was | |
not responsive. | |
Thanks to Samu Nuutamo for the fix. | |
- 2020-09-14: core bugfix: potential segfault on query of PROGRAMNAME property | |
A data race can happen on variable iLenProgram as it is not guarded | |
by the message mutex at time of query. This can lead to it being | |
non -1 while the buffer has not yet properly set up. | |
Thanks to Leo Fang for alerting us and a related | |
patch proposal. | |
replaces https://github.com/rsyslog/rsyslog/pull/4300 | |
- 2020-09-14: imtcp bugfix: broken connection not necessariy detected | |
Due to an invalid return code check, broken TCP sessions could not | |
necessarily be detected "right in time". This can result is the loss | |
of one message. | |
closes https://github.com/rsyslog/rsyslog/issues/4227 | |
Thanks to Leo Fang for the patch. | |
- 2020-09-14: new module: imhttp - http input | |
permits to receive log data via HTTP. | |
uses http library to provide http input. | |
user would need to configure an 'endpoint' as input, along | |
with a ruleset, defining how the input should be routed in | |
rsyslog. | |
Thanks to Nelson Yen for contributing this module. | |
- 2020-09-11: mmdarwin bugfix: potential zero uuid when reusing existing one | |
- fix a use-after-free variable during darwin uuid message extraction | |
- improve debug/output by logging uuid parse errors | |
Thanks to github user frikilax for the patch. | |
- 2020-09-10: imdocker bugfix: build issue on some platforms | |
An invalid variable type was used, leading to compile errors at least on | |
all platform that use gcc 10 and above. Otherwise, however, it looks like the | |
issue caused no real harm. | |
- 2020-09-07: omudpspoof bugfix: make compatbile with Solaris build | |
Thanks to Dagobert Michelsen for the patch. | |
- 2020-09-03: testbench fix: python 3 incompatibility | |
- 2020-09-02: core bugfix: segfault if disk-queue file cannot be created | |
When using Disk Queue and a queue.filename that can not be created | |
by rsyslog, the service does not switch to another queue type as | |
supposed to and crashes at a later step. | |
closes: https://github.com/rsyslog/rsyslog/issues/4282 | |
- 2020-08-26: cosmetic: fix dummy module name in debug output | |
When we have optional components (like imjournal) a dummy module | |
is used. It's sole purpose is to emit "this module is not available". | |
During init, the module emitted an invalid module name into the debug | |
log. This has now been replaced by the generic term "dummy". | |
Note: it is highly unlikely that someone will ever see that message | |
at all, as it is unlikely for the dummy modules to be build. | |
see also: https://github.com/rsyslog/rsyslog/commit/84a7e3d80b80106dcc86c273ed8cf78a6c11c722#r41782830 | |
Thanks to Thomas D. (whissi) for the patch. | |
- 2020-08-26: config bugfix: intended warning emitted as error | |
When there are actions configured after a STOP, a warning should be | |
emitted. In fact, an error message is generated. This prevents the | |
construct, which may have some legit uses in exotic settings. It | |
may also break older configs, but as the message is an error | |
for so long now, this should be no longer of concern. | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2008.0 (aka 2020.08) 2020-08-25 | |
- 2020-08-25: imdocker bugfix: error reporting not always correct | |
A wrong function to obtain the error code was used. This | |
could lead to invalid error messages. | |
Thanks to Steve Grubb for the bug report and fix proposal. | |
closes https://github.com/rsyslog/rsyslog/issues/4381 | |
- 2020-08-25: imptcp: add max sessions config parameter | |
The max is per-instance, not global across all instances. | |
There is also a bugfix where if epoll failed I think we could leave a | |
session linked in the list of sessions, this code unlinks it. | |
Thank to Alfred Perlstein for the patch. | |
- 2020-08-24: omelasticsearch bugfix: reply buffer reset after health check | |
The issue happens when more than one server is defined on the | |
action. On that condition a health check is made through | |
checkConn() before sending the POST. The replyLen should be | |
set back to 0 after the health check, otherwise the response | |
data received from the POST gets appended to the end of the | |
last health check. | |
Thanks to Julien Thomas for the patch. | |
- 2020-08-14: omfile: do no longer limit dynafile cache size in legacy format | |
When using obsolete legacy config format, omfile had a hard limit of | |
1,000 dynafile cache entries. This does not play well with very | |
large installation. This limit is now removed and converted into | |
a warning if cache size > 25,000 is specified. | |
Note: the problem can easily be worked-around by using modern | |
config format (RainerScript). | |
closes: https://github.com/rsyslog/rsyslog/issues/4241 | |
- 2020-08-13: imudp: fix very small, static memory leak | |
When ruleset support was used, the ruleset name was not freed upon rsyslog | |
termination. While this has no consequences for regular runs, it generates | |
leak errors under memory debuggers and as such makes debugging harder than | |
necessary. | |
Thanks to github user frikilax for the patch. | |
- 2020-08-13: omelasticsearch: add parameter skipPipelineIfEmpty | |
When POST'ing a document, Elasticsearch does not allow an empty pipeline | |
parameter value. This patch introduces boolean option skipPipelineIfEmpty | |
to the omelasticsearch action. When set to true, the pipeline parameter | |
won't be posted. Default is false so we do not modify current behavior. | |
Thanks to Julien Thomas for the patch. | |
- 2020-08-12: systemd service file removed from project | |
This was done as distros nowadays have very different service files and it no | |
longer is useful to provide a "generic" (sic) example. | |
see also: https://github.com/rsyslog/rsyslog/issues/4333 | |
- 2020-08-11: gnutls TLS driver bugfix: EKU check not done properly | |
When the server accepted a new connection, it did not properly set the | |
dataTypeCheck field based on the listening socket. That resulted in | |
skipping ExtendedKeyUsage (EKU) check on the client. | |
Thanks to Daiki Ueno for the patch. | |
- 2020-08-06: MMDARWIN:: improve configuration flexibility and UUID fix | |
-t pu now able to get fields from local variables ($.) | |
- now able to configure a custom root container for mmdarwin fields | |
- now able to put nested keys ($!key1!key2) | |
- don't regenerate a UUID each time, but instead check if one exists before | |
creating it (allow successive calls without losing previous UUID) | |
Thanks to github user frikilax for the contribution. | |
- 2020-08-06: add --enable-imjournal=optional ./configure option | |
- 2020-08-06: IMPCAP::Fixes: segfault, memory and build corrections | |
* fix bug in ethernet packets parsing | |
* fix removes build error with gcc10: 'multiple definition of...' | |
* resolve memory leak during interface init failure (device not freed after post-create error) | |
* add test 'impcap_bug_ether' to prove ethernet parser fix is working | |
Thanks to github user frikilax for the contribution. | |
closes https://github.com/rsyslog/rsyslog/issues/4332 | |
- 2020-07-14: CI: add support for github actions | |
- 2020-07-14: imklog: add ruleset support | |
see also: https://github.com/rsyslog/rsyslog/issues/4344#issuecomment-658001854 | |
see also: https://github.com/rsyslog/rsyslog/issues/106 | |
- 2020-07-06: config system fix: ChkDisabled method to make config.enabled work | |
There was wrong negation in the method so it returned 0/1 in reverse | |
and also it did not mark the node to not be reported as unknown at all | |
times which is needed after all. | |
Thanks to Jiri Vymazal for the patch. | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2006.0 (aka 2020.06) 2020-06-23 | |
- 2020-06-22: queue: permit ability to double size at shutdown | |
This prevents message loss due to "queue full" when re-enqueueing data | |
under quite exotic settings. | |
see also https://github.com/rsyslog/rsyslog/issues/3941#issuecomment-549765813 | |
closes https://github.com/rsyslog/rsyslog/issues/4020 | |
- 2020-06-22:Fixing imfile segfaulting on selinux denial | |
If imfile is denied access to file watched trough symlink there is | |
unchecked condition resulting in access to not initialized memory. | |
- 2020-06-22: openssl: Fixed memory leak when tls handshake failed. | |
closes: https://github.com/rsyslog/rsyslog/issues/4319 | |
- 2020-06-22: change systemd service file to wait for network | |
now that rsyslog is usually only installed for real syslog servers, | |
we should assume that some network listening or forwarding happens | |
on start. As such we need to start a bit later, after the network. | |
This poses no problem as systemd nowadays comes with journal which | |
is in almost all cases configured to buffer log data while | |
rsyslog is not yet running. | |
see also https://github.com/rsyslog/rsyslog-pkg-rhel-centos/issues/72 | |
- 2020-06-22: NEW INPUT MODULE:: impcap, network packets input parser | |
Thanks to github user frikilax for the contribution. | |
- 2020-06-22: ksi bugfix: Optimized code in KSI module initialization fixed. | |
KSI module initialization will not stuck in infinite loop when code is | |
built with optimization -O2. | |
- 2020-06-05: operatingstatefile bugfix: month was given too low | |
The month was printed with the range 0 (January) to 11 (December). | |
This has now been corrected. | |
closes https://github.com/rsyslog/rsyslog/issues/4292 | |
- 2020-06-05: build system: add "optional" build functionality to some components | |
Nameley: | |
--enable-libdbi=optional | |
--enable-mmdblookup=optional | |
--enable-imkafka=optional | |
--enable-omkafka=optional | |
If used, builds a dummy module which just emits a "module not supported | |
on this platform" error message when loaded. | |
Primary use case for this system is Debian-ish builds on SUSE OBS, | |
where we prefer to have a single package definition for all versions | |
(else things get much more complicated). | |
- 2020-05-23: config system bugfix: backticks cat segfault if file cannot be opened | |
when a `cat <filename>` construct is used in rsyslog.conf and <filename> can not | |
be accessed (does not exist, no permissions, ...), rsyslog segfaults. | |
Thanks to Michael Skeffington for notifying us and providing root cause analysis. | |
closes https://github.com/rsyslog/rsyslog/issues/4290 | |
- 2020-05-15: imtcp bugfix: octet framing/stuffing problem with discardTruncatedMsg on | |
When "discardTruncatedMsg" was enabled in imtcp, messages were incorrectly | |
skipped if the last character before the truncation was the LFdelimiter. | |
Also adds two testbench tests for this case. | |
closes: https://github.com/rsyslog/rsyslog/issues/4281 | |
- 2020-05-12: ompipe bugfix: race during HUP | |
When HUP was received, the write mutex was not aquired. This could | |
lead to unexpected invalidation of the output file descriptor. | |
Thanks to Julien Thomas for alerting us on this issue. | |
see also https://github.com/rsyslog/rsyslog/pull/4136#issuecomment-578326278 | |
- 2020-05-12: ompipe: add action parameter tryResumeReopen | |
Sometimes we need to reopen a pipe after an ompipe action gets | |
suspended. Sending an HUP signal to rsyslog does the job but requires | |
an interraction with rsyslog. The patch adds support for a new boolean | |
option, tryResumeReopen, for the ompipe action. It mimics what an HUP | |
signal would do. | |
Thanks to Julien Thomas for the patch. | |
- 2020-05-12: imjournal: remove strcat call | |
Thanks to Jeff Marckel for the patch. | |
- 2020-05-12: build system: libzcmq version requirement needs to be bumped | |
Thanks to Thomas Deutschmann for pointing this out. | |
closes https://github.com/rsyslog/rsyslog/issues/3957 | |
- 2020-05-12: testbench: download ElasticSearch binaries from rsyslog.com | |
The official ElasticSearch download site sometimes denies the download. | |
- 2020-05-11: openssl netstream driver bugfix: context leak | |
The context object was not properly freed. | |
Thanks to Michael Zimmermann for the fix. | |
- 2020-05-11: omhttp: Add support for multiple http headers | |
Allows the inclusion of multiple http headers on the REST call. | |
Thanks to callmegar for the patch. | |
- 2020-04-29: core bugfix: group id could not be obtained for very large groups | |
Thanks to github user emilbart for the patch. | |
- 2020-04-29: testbench additions (relp broken connection test) | |
- 2020-04-29: omudpspoof bugfix: issues with oversized messages | |
First issue was an incorrect packet length in UDP Header. It has to be the FULL UDP Packet | |
regardless of the MTU Setting. As a result regardless of IP fragmentation, the MTU setting | |
also limited the siizmax size of the UDP message. | |
The second issue was incorrect calculation of the UDP Checksum with libnet if | |
IP fragmentation was used (Based on MTU Setting). As a result, the network packets were | |
dropped by the tcp stack before they even could reach there target. The workarround for this | |
problem is, that we set the UDP Checksum to 0x0000 which allows skipping of the checksum | |
test. Fixing the problem by calculating the correct UDP Checksum would require some | |
code changes in the libnet. | |
Also fixed the omudpspoof bigmsg test and increased the testing size to 16KB. | |
- 2020-04-29: omprog: fix assert failed on HUP with output flag | |
If the 'output' setting of omprog was used and rsyslog received a HUP | |
signal just after starting (and before the omprog action received the | |
first log to process), an internal assertion could fail, causing | |
rsyslog to terminate. The failure message was "rsyslogd: omprog.c:660: | |
closeOutputFile: Assertion `pCtx->bIsRunning' failed." | |
The failure could also occur if rsyslog received a HUP signal during | |
the shutdown sequence. | |
This bug was introduced in v8.2004 by PR https://github.com/rsyslog/rsyslog/pull/4255 | |
Although a test already existed that checked the interaction of HUPs | |
with the 'output' setting, it didn't always fail in this particular case | |
due to timing conditions. The test has been improved to cover this case | |
more reliably. | |
Thanks to Joan Sala Isern for the patch. | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2004.0 (aka 2020.04) 2020-04-28 | |
- 2020-04-28: ksi bugfix: When KSI module is suddenly closed, files are finalized | |
In async. mode all pending signature requests are closed immediately and | |
unsigned block marker is attached with message about sudden closure. | |
Similar approach is used for blocks that already contain some records. | |
Empty blocks are just closed without any metadata. | |
Thanks to Taavi Väljaots for the patch. | |
- 2020-04-28: ksi bugfix: Signer thread initialization is verified before usage. | |
When signer thread is created in rsksiInitModule thread successful | |
initialization is verified before returning the function. This will | |
prevent adding records to not initialized module and in case of an | |
error signature files opened will contain only magic bytes. | |
Thread flags replaced with thread state. | |
When init module fails, module is disabled. | |
Thanks to Taavi Väljaots for the patch. | |
- 2020-04-28: ksi bugfix: Hardcoded default hash algorithm replaced with 'default' | |
Instead of hardcoded SHA-256 KSI_getHashAlgorithmByName("default") | |
is used to get default hash function. | |
Function rsksiSetHashFunction and SetCnfParam updated. | |
Thanks to Taavi Väljaots for the patch. | |
- 2020-04-28: imfile bugfix: poential segfault in stream object on file read | |
- if cstrLen(pThis->prevMsgSegment) > maxMsgSize then len calculation | |
become negative if cstrLen(thisLine) < cstrLen(pThis->prevMsgSegment) | |
This causes illegal access to memory location and thus causing segfault. | |
- assigning len = 0 if cstrLen(pThis->prevMsgSegment) > maxMsgSize so that | |
it access the correct memory location. | |
Thanks to github user jaankit | |
- 2020-04-28: openssl TLS drivers: made more reliable for older openssl versions | |
OpenSSL can retry some failed operations, but older versions need an explicit | |
opt-in to do so. This is now done. | |
- 2020-04-28: omprog: fix bad fd errors in daemon mode | |
When omprog was used with the 'forceSingleInstance=on' option, and/or | |
the 'output' setting, "bad file descriptor" errors occurred, which | |
prevented the external program to be executed and/or the program output | |
to be correctly captured. The bug could also manifest as "resource | |
temporarily unavailable" errors, or other errors related to the use of | |
invalid/reassigned file descriptors. These errors only happened when | |
rsyslog ran in daemon mode (i.e. they didn't happen if rsyslogd was | |
run with the '-n' option). | |
The cause of the bug was that omprog opened the pipe fds needed by | |
these flags during the configuration load phase (in the 'newActInst' | |
module entrypoint). This is a bad place since the fork of the daemon | |
occurs after this phase, and all fds are closed when the daemon process | |
is started (see 'initAll' in rsyslogd.c), hence invalidating the | |
previously opened fds. | |
To correct this, the single child process and the output capture thread | |
are now started later, when the first log message is received by the | |
first worker thread. (Note: the 'activateCnf' module entrypoint, despite | |
being invoked after the fork, cannot be used for this purpose, since it | |
is invoked per module, not per action instance.) | |
Currently no automated test exists for this use case since the testbench | |
always runs rsyslog in non-daemon mode. | |
Affected versions: v8.38 and later | |
closes: https://github.com/rsyslog/rsyslog/issues/4247 | |
Thanks to Joan Sala Isern for the patch. | |
- 2020-04-28: omfile bugfix: $outchannel split log lines at rotation time | |
- 2020-04-17: openssl: add support for libreSSL | |
Disable use of "@SECLEVEL" in default cipher string and | |
avoid SSL_CONF_CTX_set_flags() API when LIBRESSL is used. | |
This means tlscommands will not work. | |
closes: https://github.com/rsyslog/rsyslog/issues/4210 | |
- 2020-03-04: imudp bugfix: build problems on some Linux kernel versions | |
Thanks to Wen Yang for the patch. | |
- 2020-03-02: conf output bugfix: -o produces missing space between call and rulename | |
Thanks to Tetiana Ohnieva for the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/3761 | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2002.0 (aka 2020.02) 2020-02-25 | |
- 2020-02-25: imfile: add per minute rate limiting | |
Add MaxBytesPerMinute and MaxLinesPerMinute options. | |
These take integer values and, respectively, limit the number | |
of bytes or lines that may be sent in a minute. | |
This can be used to put a limit on the count or volume of logs | |
that may be sent for an imfile. | |
Thanks to Greg Farrell for the patch. | |
- 2020-02-24: core: add global parameter "security.abortOnIDResolutionFail" | |
This parameter controls whether or not rsyslog aborts when a name ID | |
lookup fails (for user and group names). This is necessary as a security | |
measure, as otherwise the wrong permissions can be assigned or privileges | |
are not dropped. | |
CHANGE OF BEHAVIOR | |
The default for this parameter is "on". In previous versions, the default | |
was "off" (by virtue of this parameter not existing). As such, existing | |
configurations may now error out. | |
We have decided to accept this change of behavior because of the potential | |
security implications. | |
closes https://github.com/rsyslog/rsyslog/issues/4164 | |
- 2020-02-24: openssl TLS driver bugfix: chained certificates were not accepted | |
This was supported since always inside GnuTLS driver, but was missing for openssl one. | |
- 2020-02-24: core bugfix: too early parsing of incoming messages | |
In theory, rsyslog should call parsers on the queue worker threads whenever | |
possible. This enables the parsers to be executed in parallel. There are | |
some cases where parsers needs to be called earlier, namely when parsed | |
data is needed for rate-limiting. | |
The logic to do this previously did not work correctly and was fixed six | |
years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive: | |
it actually makes the early parser call now mandatory, effectively moving | |
parsing to the input side where there is no to little concurrency. | |
We still do not need to call the parser when all messages, regardless of | |
severity, need to be rate-limited. This is the default and very frequent | |
case. This patch introduces support for this and as such makes parsers | |
able to run in parallel in the frequent case again. | |
closes https://github.com/rsyslog/rsyslog/issues/4187 | |
- 2020-02-20: testbench bugfix: two minor issues in omkafkadynakey.sh test | |
lead to false positives during test runs (depending on circumstances) | |
closes: https://github.com/rsyslog/rsyslog/issues/4134 | |
- 2020-02-20: testbench: set max extra data length for tcpflood from 200 to 512KiB | |
Added a imrelp test for big messages (256KB). | |
closes: https://github.com/rsyslog/rsyslog/issues/4158 | |
- 2020-02-20: config system bugfix: 'config.enabled' directive oddities | |
Previously the directive was processed way too late which caused false | |
errors whenever it was set to 'off' and possibly other problems. | |
Thanks to Jiri Vymazal for the patch. | |
- 2020-02-09: imfile bugfix: timeout did not work on very busy system | |
The timeout feature was soley based on timeouts of the poll() | |
system call. On a very busy system, this would probably happen | |
very seldomly. Moreover, the timeout could occur later than | |
expected on any system with high load. | |
The issue was not reported from practice but discovered during | |
CI system improvements. | |
- 2020-01-30: build system: change --enable-imfile-tests default to "yes" | |
This was accidentally set to "no" some time ago (actual commit unknown). Tests for | |
imfile should by default run when imfile is enabled. | |
see also https://github.com/rsyslog/rsyslog/issues/4120 | |
- 2020-01-27: build system: add option --enable-gnutls-tests | |
This enables us to build GNUtls support but not necessarily | |
test it in CI. This is useful for some specialised subcomponent | |
test. The default is enabled if gnutls is enabled and disabled if not. | |
- 2020-01-26: testbench: new test for loadbalancing via global vars | |
This is a popular functionality which had not been routinely tested | |
in the past. | |
- 2020-01-26: mmdblookup bugfix: invalid data returned when no entry found | |
Since the upgrade of the package libmaxminddb on FreeBSD (1.3.2_2 -> 1.4.2), | |
the module mmdblookup returns the first entry of the mmdb database even if the entry | |
is not found. After some debug, I found the solution in the official maxminddb | |
repository : to check if the entry is in database, we must check the found_entry | |
attribute, otherwise the function MMDB_get_entry_data_list will return the first | |
entry of the database if the entry is not found in it. | |
Thanks to Kevin Guillemot for the patch. | |
- 2020-01-23: oversize message log bugfix: do not close fd -1 | |
The oversize message log fd is always closed on HUP, even if it never | |
was opened (and thus has -1 value). This patch corrects the issue. | |
The bug had no know-bad effect in practice other than getting an | |
(ignored) error status from close(). However, it introduced warnings | |
in test runs (e.g. when running under valgrind). | |
- 2020-01-22: imfile bugfix: saving of old file_id for statefiles | |
Previously we saved old file_id unconditionally, which led to not | |
deleting old statefiles if files changes without rsyslog running. | |
Now it should work correctly. | |
Thanks to Jiri Vymazal for the patch. | |
- 2020-01-22: imfile bugfix: misadressing and potential segfault | |
Commit 3f72e8c introduced an invalid memory allocation size. This lead to | |
too-short alloc and thus to overwrite of non-owned memory. That in turn | |
could lead to segfaults or other hard to find problems. | |
The issue was detected by our upgraded CI system. We did not receive | |
any problem reports in practice. Nevertheless, the problem is real and | |
people should update affected versions to patched ones. | |
The bug was present in scheduled stable release 8.1911.0 and 8.2001.0. | |
see also: https://github.com/rsyslog/rsyslog/issues/4120 | |
see also: https://github.com/rsyslog/rsyslog/pull/4141 | |
- 2020-01-20: core bugfix: potential race during HUP | |
when rsyslog is HUPed immediately after startup and before it is fully | |
initialized, there is a potential race with the list of loaded modules. | |
This patch ensures no bad things can happen in that case. | |
Detected by LLVM TSAN, not seen in practice. | |
- 2020-01-20: testbench improvements and fixes | |
modernize tests, reduce robustness against slow machines, provide some | |
test framework functional enhancements, and optimize some tests. | |
Also includes some code changes to C testing components. Among others, | |
tests have slightly been speeded up by reducing the wait time at queue | |
shutdown. This is possible because of better overall completion checks. | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.2001.0 (aka 2020.01) 2020-01-14 | |
- 2020-01-12: core bugfix: race condition related to libfastjson when using DA queue | |
Rsyslogd aborts when writing to disk queue from multiple workers simultaneously. | |
It is assumed that libfastjson is not thread-safe. | |
Resolve libfastjson race condition when writing to disk queue. | |
see also https://github.com/rsyslog/rsyslog/issues/4041 | |
Thanks to MIZUTA Takeshi for the fix. | |
- 2020-01-12: omfwd bugfix: parameter streamdriver.permitexpiredcerts did not work | |
closes https://github.com/rsyslog/rsyslog/issues/4098 | |
- 2020-01-11: Bugfix: KSI module + dynafile in asynchronous mode fixed | |
Thanks to Taavi Valjaots for the patch | |
- 2020-01-08: tls driver: add support to configure certificate verify depth | |
Support added in omfwd as instance parameter: | |
streamdriver.TlsVerifyDepth | |
Support added in imtcp as module parameter: | |
streamdriver.TlsVerifyDepth | |
Can be 2 or higher. | |
Support added into ossl driver | |
Support added into gtls driver | |
Added testcases for both drivers. | |
closes: https://github.com/rsyslog/rsyslog/issues/4035 | |
- 2020-01-08: modernization of testbench | |
moved some tests to newer standards, hardened them against slow testbench machines, | |
kafka component download improvements, and prevent dangling left-over test tool | |
instances from aborted tests | |
- 2020-01-07: tls subsystem bugfix: default for permitExpiredCerts was invalidly "on" | |
The problem occured with commit 3d9b8df in December 2018 and went into | |
scheduled stable 8.1901.0. Unfortunately, the change in default was not detected | |
until a year later. This commit re-enables the previous default ("off"), which is | |
also the only sensible default from a security PoV. Unfortunately, new 2019 | |
deployments may begin to see connection rejection when usin expired certs. As | |
expired certs should not be used, this hopefully will not cause problems in | |
practice. | |
Thanks to Jiri Vymazal for the patch. | |
- 2020-01-01: testbench: improve ElasticSearch test speed | |
We now support re-using suitable running ES instances, which reduces the | |
number of restarts. | |
- 2019-12-31: omelasticsearch: improve curl reply buffer handling | |
The curl reply buffer (pWrkrData->reply) was allocated, realloced and freed with | |
each request. This has now been reduced to once per module, slightly increasing | |
overall performance. | |
closes https://github.com/rsyslog/rsyslog/issues/1964 | |
- 2019-12-31: config system: emit proper error message on $ in double-quoted string | |
closes https://github.com/rsyslog/rsyslog/issues/2869 | |
- 2019-12-30: core bugfix: rsyslog aborts when config parse error is detected | |
In defaut settings, rsyslog tries to continue to run, but some data | |
structures are not properly initialized due to the config parsing error. | |
This causes a segfault. | |
In the following tracker, this is the root cause of the abort: | |
see also https://github.com/rsyslog/rsyslog/issues/2869 | |
- 2019-12-30: fix some alignment issues | |
So far, this worked everywhere (for years). But it may still have | |
caused issues on some platforms. | |
closes https://github.com/rsyslog/rsyslog/issues/2608 | |
- 2019-12-27: core bugfix: APP-NAME fields could become empty | |
RFC 5424 specifies that an empty APP-NAME needs to be indicated by | |
"-". Instead, the field could become empty under certain conditions. | |
If so, outgoing 5424 messages were invalidly formatted. | |
This happened under quite unusual conditions, but could be seen | |
in practice. | |
closes https://github.com/rsyslog/rsyslog/issues/4043 | |
- 2019-12-27: core bugfix: reopen /dev/urandom file descriptor after fork on Linux | |
This patch updates prepareBackground() in tools/rsyslogd.c to reopen any file | |
descriptors used for random number generation in the child process. This fixes | |
an issue on Linux systems where the file descriptor obtained for /dev/urandom | |
by seedRandomNumber() in runtime/srutils.c was left closed after the fork. This | |
could be observed in procfs, where /proc/fd/ would show no open descriptors to | |
/dev/urandom in the forked process. /dev/urandom is reopened as the child may be | |
be operating in a jail, and so should not continue to use file descriptors from | |
outside the jail (i.e. inherited from the parent process). | |
I found that this issue led to rsyslog intermittently hanging during seedIV() | |
in runtime/libgcry.c. After the fork, the closed file descriptor number tended | |
to get re-assigned. randomNumber() would then read from an incorrect (although | |
still valid) file descriptor, and could block (depending on the state of that | |
file descriptor). This gave rise to the intermittent hang that I observed. | |
Thanks to Simon Haggett for the patch. | |
- 2019-12-20: imdocker bugfix: did not compile without atomic operations | |
- 2019-12-20: omclickhouse: new parameter "timeout" | |
Thanks to Pavlo Bashynskiy for the patch. | |
- 2019-12-20: omhiredis: add 'set' mode plus some fixes | |
- new mode 'set' to send SET/SETEX commands | |
- new parameter 'expiration' to send SETEX instead of SET commands (only applicable to 'set' mode) | |
- fixes to missing frees | |
Thanks to github user frikilax for the patch. | |
- 2019-12-18: relp: Add support setting openssl configuration commands. | |
Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp. | |
(Using relpSrvSetTlsConfigCmd and relpCltSetTlsConfigCmd) | |
OpenSSL Version 1.0.2 or higher is required for this feature. | |
A list of possible commands and their valid values can be found in the | |
documentation: https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html | |
The setting can be single or multiline, each configuration command is | |
separated by linefeed (n). Command and value are separated by | |
equal sign (=). Here are a few samples: | |
tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.2" | |
tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 | |
MinProtocol=TLSv1.2" | |
Add to new testcases for librelp and tlscfgcmd. | |
closes https://github.com/rsyslog/rsyslog/issues/3959 | |
- 2019-12-18: bugfix core: potential segfault in template engine | |
under some circumstances (not entirely clear right now), memory | |
was freed but later re-used as state-tracking structures were not | |
properly maintained. Github issue mentioned below has full details. | |
Thanks to github user snaix for analyzing this issue and providing | |
a patch. I am committing as myself as snaix did not disclose his or | |
her identity. | |
closes https://github.com/rsyslog/rsyslog/issues/3019 | |
closes https://github.com/rsyslog/rsyslog/issues/4040 | |
- 2019-12-18: fixed some minor issues detected by clang static analyzer 9 | |
- 2019-12-10: core/config bugfix: false error msg when config.enabled="on" is used | |
When the 'config.enabled="on"' config parameter an invalid error message | |
was emitted that this parameter is not supported. However, it was still | |
applied properly. This commit removes the invalid error message. | |
closes https://github.com/rsyslog/rsyslog/issues/4011 | |
- 2019-12-03: omsnmp bugfix: "traptype" parameter invalidly rejected value 6 | |
"Traptype" needs to support values 0 to 6. | |
However, if value 6(ENTERPRISESPECIFIC) was set, an invalid error message | |
was emitted. Otherwise processing was correct. | |
This could lead to problems with automatic config deployment, | |
as valid configurations were invalidly reported as incorrect. | |
That in turn could make a deployment fail. | |
closes https://github.com/rsyslog/rsyslog/issues/3973 | |
- 2019-12-03: omsnmp: add new parameter "snmpv1dynsource" | |
If set, the source field from SNMPv1 trap can be overwritten | |
with a template, default is "%fromhost-ip%". The content should be a | |
valid IPv4 Address that can be passed to inet_addr(). If the content | |
is not a valid IPv4 Address, the source will not be set. | |
closes: https://github.com/rsyslog/rsyslog/issues/3991 | |
- 2019-12-02: imfile bugfix: state file renaming sometimes did not work properly | |
Now checking if file-id changes and renaming - cleaning state file | |
accordingly and always checking and cleaning old inode-only style | |
state files. | |
Thanks to Jiri Vymazal for the patch. | |
- 2019-12-02: ratelimit: increase rate limit interval parameter max value | |
The burst parameter in the ratelimit was increased to an unsigned int | |
but the interval remained an unsigned short. While it may be unusual, | |
there is possibly a chance to need to represent an interval longer than | |
about 3/4 of a day. | |
While here, go through and normalize all the various incarnations of | |
rate limiting to be explicitly unsigned int for the burst and interval. | |
Thanks to github user frikilax for the patch. | |
- 2019-12-02: ommongodb: Add other supported formats for 'time' and 'date' fields | |
Thanks to github user frikilax for the patch. | |
- 2019-12-02: imjournal bugfix: too many messages in error case | |
Under certain error conditions, `ignorePreviousMessages="on"` could be ignored | |
an existing messages be processed. | |
Thanks to github user 3chas3 for the patch. | |
- 2019-11-27: core bugfix: action on retry mangles messages | |
When a failed action goes into retry, template content is rendered | |
invalid if the action uses more than 1 template. | |
closes https://github.com/rsyslog/rsyslog/issues/3898 | |
Thanks to Mikko Kortelainen for the patch. | |
- 2019-11-27: testbench: improve mysql testing support | |
tests can now run in parallel and are hardened against several glitches | |
- 2019-11-22: omhttp: add basic support for Loki Rest | |
Loki is a new message indexer and querier from Grafana Labs. See | |
https://github.com/grafana/loki for details on Loki. | |
This change provides the initial message structure to send bulk message | |
payloads to the Loki Rest endpoint. omhttp, received a new bulk message | |
format called lokirest. Additionally, the plugin relies on the user to | |
provide the correct "stream" read message format. | |
A loki template must be json compatible and include a "stream" key of | |
key value tags, and a values key of an array of 2 element arrays, where | |
each 2 element array is the unix epoch in nanoseconds followed by an | |
unstructured message. | |
An example: | |
template(name="array_loki" type="string" string="{\"stream\":{\"host\":\"%HOSTNAME%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"syslogtag\":\"%syslogtag%\"},\"values\": [[ \"%timegenerated:::date-unixtimestamp%000000000\", \"%msg%\" ]]}") | |
- 2019-11-22: testbench: obtain python binary path via AM_PATH_PYTHON | |
see also https://github.com/rsyslog/rsyslog/issues/3853 | |
- 2019-11-22: omprog: detect violation of interface protocol | |
The spec for the omprog interaction with the program it calls specifies | |
that the program receives one message via one line. In other words: | |
it must be a string terminated by LF. | |
However, omprog does currently rely on a proper template to fulfill this | |
requirement, If the template does not provide for the LF, it is never | |
written. For the called program, this looks like it does not receive any | |
input at all. Even if it finally reads data (e.g. due to full buffer), | |
it will not properly be able to discern the messages. | |
This handling is improved with this commit. | |
We cannot just check the template, because at the end of the template | |
may by a non-constant value. As such, we do not know at config load | |
time if there is this problem or not. | |
So the correct approach is to, during runtime, check if each message | |
is properly terminated. For those that are not: | |
* we append a LF, because anything else makes matters worse | |
* log a warning message, at least for a sample of the messages | |
The warning is useful in the (expected most often) case that the template | |
is simply missing the LF. While appending works, it slows down processing. | |
As such the user should be given a chance to correct the config bug. | |
To avoid clutter, the warning is emitted at most once every 30 seconds. | |
This value is hardcoded as we do not envision a need to adjust it. Usually | |
users should quickly fix the template. | |
closes https://github.com/rsyslog/rsyslog/issues/3975 | |
- 2019-11-19: core queue: emit warning if parameters are set for direct queue | |
Direct queues do not apply queue parameters because they are actually | |
no physical queue. As such, any parameter set is ignored. This can | |
lead to unintentional results. | |
The new code detects this case and warns the user. | |
closes https://github.com/rsyslog/rsyslog/issues/77 | |
- 2019-11-19: imjournal bugfix: do not wait too long on recovery try | |
When trying to recover journal errors, imjournal waited a hardcoded | |
period of 10s between tries. This was pretty long and could lead to | |
loss of journal data. | |
This commit adjust it to 100ms, which should still be fully sufficient | |
to prevent the journal from "hammering" the CPU. | |
It may be worth considering to make this setting configurable - but | |
let's first see if there is real demand to actually do that. | |
closes https://github.com/rsyslog/rsyslog/issues/3969 | |
- 2019-11-19: mmutf8fix: enhance handling of incorrect UTF-8 sequences | |
1. Invalid utf8 detection didn't handle 3 and 4-byte overlong encodings (2 | |
byte overlong encodings were handled explicitly by rejection E0 and E1 | |
start bytes). Unified checks for overlong encodings. | |
2. Surrogates U+D800..U+DFFF are not valid codepoints (Unicode Standard, D92) | |
3. Replacement of characters in invalid 3 or 4-bytes encodings was too | |
eager. It must not replace bytes which are valid UTF-8 sequences. For | |
example, in [0xE0 0xC2 0xA7] sequence the 0xC2 is invalid as a continuation | |
byte, but it starts a valid UTF8 symbol [0xC2 0xA7]. That is, with current | |
code processing the sequence will result in "???" but the correct result is "?§" | |
(provided that the replacement character is "?"). | |
4. Various tests for UTF-8 invalid/valid sequences. | |
Thanks to Sergei Turchanov for the patch. | |
- 2019-11-14: imfile: add new input parameter escapeLF.replacement | |
The new parameter permits to specify a replacement to be configured | |
when "escapeLF" is set to "on". Previously, a fixed replacement string | |
was used ("#012"/"\n") depending on circumstances. If the parameter is | |
set to an empty string, the LF is simply discarded. | |
closes https://github.com/rsyslog/rsyslog/issues/3889 | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.1911.0 (aka 2019.11) 2019-11-12 | |
- 2019-11-12: core queue: add config param "queue.takeFlowCtlFromMsg" | |
This is a fine-tuning option which permits to control whether or not | |
rsyslog shall alays take the flow control setting from the message. If | |
so, non-primary queues may also block when reaching high water mark. | |
This permits to add some synchronous processing to rsyslog core engine. | |
However, it is dangerous, as improper use may make the core engine | |
stall. As such, enabling this option requires very careful planning | |
of the rsyslog configuration and deep understanding of the consequences. | |
Note that the option is applied to individual queues, so a configuration | |
with a large number of queues can (and must if use) be fine-tuned to | |
the exact use case. | |
The rsyslog team strongly recommends to let the option turned off, | |
which is the default setting. | |
see also https://github.com/rsyslog/rsyslog/issues/3941 | |
- 2019-11-12: imrelp: add new config parameter "flowcontrol" | |
This permits to fine-tune the flowControl parameter. Possible values are | |
"no", "light", and "full". With light being the default and previously | |
only value. | |
Changing the flow control setting may be useful for some rare applications, | |
but be sure to know exactly what you are doing when changing this setting. | |
Most importantly, whole rsyslog may block and become unresponsive if you | |
change flowcontrol to "full". While this may be a desired effect when | |
intentionally trying to make it most unlikely that rsyslog needs to | |
lose/discard messages, usually this is not what you want. | |
see also https://github.com/rsyslog/rsyslog/issues/3941 | |
- 2019-11-11: imrelp: remove unsafe debug instrumentation | |
dbgprintf, which is not signal safe, was called from a signal handler | |
to get better understanding during debugging. While this usually works, | |
it can occasionally (5%) lead to a hang during shutdown. We have now | |
removed that debug info as it is no longer vital. | |
Note: this could only happen during debug runs. Production mode was | |
not affected. As such, this fix is only relevant to developers. | |
However, it caused some confusion in the following issue tracker. | |
see also https://github.com/rsyslog/rsyslog/issues/3941 | |
- 2019-11-06: ossl driver bugfix: fix wrong OpenSSL Version check | |
Fix OpenSSL Version check in: | |
- SetGnutlsPriorityString function in nsd_ossl.c | |
- initTLS() function tcpflood.c | |
See https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_VERSION_NUMBER.html | |
for more. | |
This bug lead to not enabling some functionality correctly. | |
Removed "MinProtocol=TLSv1.1" from two testcases because MinProtocol | |
is only supported by OpenSSl 1.1.0 or higher and was not really | |
necessary for the testcases. | |
closes https://github.com/rsyslog/rsyslog/issues/3939 | |
- 2019-11-05: mmdarwin: Optimizations, new parameters, update to protocol header | |
- use permanent worker-dependent buffers to avoid malloc/free for each entry | |
- move socket structures to worker data, remove global mutex | |
- add log lines for parameters and general workflow | |
- don't send body if empty/incomplete (see new parameters) | |
- don't close/reopen socket every time -> let session open or create new every X | |
entry (see new parameters) | |
- clean up code | |
- added 'send_partial', to let mmdarwin send body if not all fields were | |
retrieved, or not; default false = only send complete bodies | |
- added 'socket_max_use' to open new session every X packet, useful for | |
some versions of Darwin (prior to 1.1) | |
default is 0 = do not open new session/keep only one | |
- added 'evt_id' to the darwin header (Darwin v1+ compatibility) | |
Note: mmdarwin is a contributed module | |
Thanks to github user frikilax for the patch. | |
- 2019-11-01: mmkubernetes bugfix: improper use of realloc() | |
could cause problems under extreme memory shortage - very unlikely | |
credits to LGTM.COM for detecting this | |
- 2019-10-31: imjournal: set the journal data threshold to MaxMessageSize | |
When data is read from the journal using sd_journal_get_data it may be | |
truncated to a certain threshold (64K by default). | |
If the rsyslog MaxMessageSize is larger than the threshold, there is a | |
chance rsyslog will receive incomplete messages from the journal. | |
Empirically, this appears to happen reliably when XZ compression is | |
used by journald. Systems where journald uses LZ4 compression do not | |
appear to suffer this issue reliably--if at all. | |
This change sets the threshold to the MaxMessageSize when the | |
journal is opened. | |
Thanks to Robert Winslow Dalpe for the patch. | |
- 2019-10-30: improg bugfix: allow improg to handle multi-line inputs | |
miscellaneous bug fixes in improg: | |
* properly truncate string after an input event is submitted | |
* set msgoffset to 0. | |
* tests added to check above fixes | |
Thanks to Nelson Yen for the fix. | |
- 2019-10-30: mmdblookup bugfix: missing space in city name | |
This fixes the issue that spaces in city names are dropped. However, the | |
fix is more or less a work-around. As it turns out, the libmaxminddb API | |
is not correctly used. In the somewhat longer term, we should fix this. | |
see also https://github.com/maxmind/libmaxminddb/issues/218 | |
closes https://github.com/rsyslog/rsyslog/issues/1650 | |
- 2019-10-30: core/queue: provide ability to run diskqueue on multiple threads | |
Up until this release, disk queues could only use a single thread, | |
what limited their performance with outputs like ElasticSearch. | |
Now disk queues can utilize multiple threads just like any other | |
queue type. Most importantly, the disk queue part of a DA queue | |
now inherits the max number of threads from its memory queue | |
counterpart. | |
NOTE: the new multi-threaded DA disk queue is actually a change of | |
behavior. We have not guarded it by a new config switch as we | |
assume the new behavior is most often exactly within user | |
expectations. In any case, we cannot see any harm from running | |
the disk queue on multiple threads. | |
see also https://github.com/rsyslog/rsyslog/issues/3543 | |
closes https://github.com/rsyslog/rsyslog/issues/3833 | |
- 2019-10-25: omfile bugfix: file handle leak | |
The stream class does not close re-opened file descriptors. | |
This lead to leaking file handles and ultimately to the inability | |
to open any files/sockets/etc as rsyslog ran out of handles. | |
The bug was depending on timing. This involved different OS | |
thread scheduler timing as well as workload. The bug was more | |
common under the following conditions: | |
- async writing of files | |
- dynafiles | |
- not committing file data at end of transaction | |
However it could be triggered under other conditions as well. | |
The refactoring done in 8.1908 increased the likelihood of | |
experiencing this bug. But it was not a real regression, the new | |
code was valid, but changed the timing so that the race was more | |
likely. | |
Thanks to Michael Biebl for reporting this bug and helping to | |
analyze it. | |
closes https://github.com/rsyslog/rsyslog/issues/3885 | |
- 2019-10-22: imfile bugfix: improper use of calloc() | |
could cause problems under extreme memory shortage - very unlikely | |
credits to LGTM.COM for detecting this | |
- 2019-10-22: TLS driver bugfix: improper use of calloc() | |
can cause problems under extreme memory shortage - very unlikely | |
credits to LGTM.COM for detecting this | |
- 2019-10-22: imuxsock bugfix: improper use of calloc() | |
can cause problems under extreme memory shortage - very unlikely | |
credits to LGTM.COM for detecting this | |
- 2019-10-17: build system bugfix: incorrect default in ./configure help text | |
closes https://github.com/rsyslog/rsyslog/issues/3904 | |
Thanks to Michael Biebl for pointing this out. | |
- 2019-10-17: mmkubernetes bugfix: improper use of calloc() | |
can cause problems under extreme memory shortage - very unlikely | |
credits to LGTM.COM for detecting this | |
- 2019-10-16: core queue bugfix: propagate batch size to DA queue | |
This was a long-standing bug where the DA queue always had a fixed small batch | |
size because the setting was not propagated from the memory queue. This also | |
removes a needless and counter-productive "debug aid" which seemed to be in | |
the code for quite some while. It did not cause harm because of the batch | |
size issue. | |
- 2019-10-16: testbench: fix unreliable gzipwrite test | |
The test was timing-sensitive as we did not properly check all data | |
was output to the output file - we just relied on sleep periods. | |
This has been changed. Also, we made some changes to the testing | |
framework to fully support sequence checking of multiple ZIP files. | |
- 2019-10-16: core queue bugfix: handle multi-queue-file delete correctly | |
Rsyslog may leave some dangling disk queue files under the following | |
conditions: | |
- batch sizes and/or messages are large | |
- queue files are comparatively small | |
- a batch spans more than two queue files (from n to n+m with m>1) | |
In this case, queue files n+1 to (n+m-1) are not deleted. This can | |
lead to problems when the queue is re-opened again. In extreme cases | |
this can also lead to stalled processing when the max disk space is | |
used up by such left-over queue files. | |
Using defaults this scenario is very unlikely, but it can happen, | |
especially when large messages are being processed. | |
- 2019-10-16: imjournal: fix regression from yesterday's patch | |
commit 78976a9bc059 introduced a regression that caused writing | |
the journal state file to fail. This happens when the state file | |
is given as relative file name and the working directory is also | |
a relative path. This situation is very uncommon. So most deployments | |
will never experience it. We discovered the issue during CI runs | |
where the trigger condition is given. Note that it also takes | |
multiple times of loading the journal to actually see the bug. | |
see also https://github.com/rsyslog/rsyslog/pull/3878 | |
- 2019-10-15: imjournal plugin code restructuring, added remote option | |
Decomposed ReadJournal() a bit, also now coupling journald | |
variables in one struct, added few warning messages and debug | |
prints to help with bug hunts in future, also got rid of two | |
needless journald calls. WorkAroundJournalBug now deprecated. | |
Added option to pull journald records from outside local machine. | |
Thanks to Jiri Vymazal for the patch. | |
- 2019-10-11: core bugfix: potential abort on very long action name | |
The action name is stored in modified form for the debug header and | |
some messages. If it is extremely long, a buffer can be overrun, | |
resulting in misaddressing and potential segfault for rsyslog. This | |
can also happen if the action is NOT named, but a custom path to | |
the output module is given and that path is very long. This triggers | |
the same issue because by default the module load path is included | |
in the action name. | |
This patch corrects the problem and truncates overly long names | |
when being used for name generation. | |
The problem was detected during testbench work. We did never receive | |
a bug report from practice. | |
- 2019-10-10: testbench: add test for mmpstrucdata with RFC5424 escape sequences | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.1910.0 (aka 2019.10) 2019-10-01 | |
- 2019-10-01: core bugfix: incorrect error message on duplicate module load | |
A Null-pointer was passed to printf instead of the module name. | |
On some platforms this may lead to a segfault. On most platforms | |
printf check's for NULL pointers and uses the string "(null)" | |
instead. In any case, the module name is missing from the error message. | |
- 2019-10-01: imczmq nitfix: potential NULL ptr in printf on out-of-memory condition | |
very unlikely to happen but if it does without any real issue on most platforms. | |
- 2019-10-01: work around some compiler warning messages induced by pthreads API | |
- 2019-10-01: core ratelimiting: more verbose message when rate-limiting happens | |
When messages are rate-limited, the error message now also contains the | |
rate limiter setting. This enables the user to more quickly understand what | |
the problem is (especially if default values apply). | |
Thanks to Jiri Vymazal for the patch. | |
- 2019-10-01: openssl TLS driver: do not emit unnecessary error message | |
On older openssl versions, an API was missing to set user-defined parameters. If we | |
had such an older version, rsyslog emitted an error message even if the user did | |
not configure such parameters. This has been corrected, so that a message is only | |
emitted if there really is a problem. Based on user feedback the severity has also | |
been downgraded to "warning". | |
- 2019-10-01: pmcisconames (contributed module) bugfix: potential misaddressing | |
- 2019-09-30: pmaixforwardedfrom (contributed module) bugfix: potential misaddressing | |
- 2019-09-30: pmdb2diag (contributed module) bugfix: Out of bounds issue | |
Add a new sanity check after determining the level len. | |
Thanks to Philippe Duveau for the patch. | |
see also: https://nvd.nist.gov/vuln/detail/CVE-2019-17040 | |
- 2019-09-02: ability to set stricter TLS operation modes | |
- checking of extendedKeyUsage certificate field | |
- stricter checking of certificate name/addresses | |
Thanks to Jiri Vymazal for the patch. | |
- 2019-08-21: testbench: add basic test for immark | |
- 2019-08-20: core: do not unnecessarily set hostname on each HUP | |
- 2019-08-20: build system: support cross-platform build for mysql/mariadb | |
rsyslog fails to cross build from source, because it uses mysql_config | |
and mysql_config is unfixably broken for cross compilation. It would be | |
better to use pkg-config. The attached patch makes rsyslog try | |
pkg-config first and fall back to mysql_config. | |
Thanks to Helmut Grohne for providing a base patch. | |
- 2019-08-20: core/tcpsrv: potential race on startup/shutdown | |
if the tcpsrv component is started and quickly terminated, it may hang | |
for a short period of time. Also a very small amount of memory is leaked | |
immediately before shutdown. While this leak is irrelevant in practice | |
(the OS clean up the process anyways), it leads to CI failures. The hang, | |
however, can lead to longer than expected shutdown times for rsyslog. | |
The problem can be experienced via imtcp, imgssapi and imdiag (users | |
of affected core component). | |
---------------------------------------------------------------------------------------- | |
Scheduled Release 8.1908.0 (aka 2019.08) 2019-08-20 | |
- 2019-08-19: testbench: add test for $allowedSender functionality | |
- 2019-08-19: testbench: harden some tests against very slow CI machines | |
- 2019-08-16: testbench: make most tests use a port file and assign listen port 0 | |
This makes the test much more robust against heavily loaded test systems. | |
- 2019-08-16: core/action: guard action.externalstate.file content against whitespace | |
remove trailing whitespace before checking the status string. This is | |
most important as a line usually ends with \n, which is considered | |
trailing whitespace. Accepting this increases usability. | |
- 2019-08-16: imtcp bugfix: multiple listenerPortFile parameter did not work | |
... because they were treated as module-global. If we had multiple imtcp | |
listeners with multiple port files, only the last filename was always used. | |
closes https://github.com/rsyslog/rsyslog/issues/3817 | |
- 2019-08-16: testbench: improve testbench plumbing for gzip and fail cases | |
We have added new capabilities to the testbench plumbing to automatically | |
deal with gzip-compressed files. This also permits to use the wait_seq_check | |
function to work for gzip tests as well. The known-timing-sensitive | |
gzipwr_large test now makes use of the new capabilities. This enables us | |
to more reliably detect when we can savely shutdown the tested instance. | |
This commit also adds an ability to "abort" the full testbench run on | |
first test failure. This is especially useful during CI. | |
- 2019-08-13: testbench: add test for imuxsock legacy format | |
This was never tested. Ensures we don't accidentally break existing | |
configurations. | |
- 2019-08-13: omelasticsearch bugfix: segfault on unknown retryRuleset | |
omelasticsearch does some "interesting tricks" for an output module. | |
This causes a segfault if the retryRuleset is now known. | |
The action module interface currently expects that all config errors | |
be detected during instance creation. Instead omelasticsearch defers | |
the retry ruleset check to a later state. The reason is that it wants | |
to support the use the same rulesetname it is defined in - and this | |
is not yet available at action parsing. | |
We fix this by ensuring that any deleted instance is properly unlinked | |
from the instance list. One may argue the module interface should get | |
upgrade for such cases, but this is a longer-term approach. | |
closes https://github.com/rsyslog/rsyslog/pull/3796 | |
- 2019-08-12: imptcp bugfix: port="0" parameter did not work as expected | |
when multiple interfaces and/or protocols could be bound, each of | |
them used a different listener ports were assigned. While this is | |
basically correct, it makes things unusable, especially as | |
listenPortFileName will only contain the port number used for | |
the latest listener. | |
This patch now follows the model of nsd_ptcp.c to assign only | |
the first port randomly and then use that port consistently. | |
- 2019-08-10: omelasticsearch bugfix: potential resource leak with "rebindinterval" | |
If the "rebindInterval" parameter was used connections could be linked. This | |
was especially the case with small intervals (such as "2"). This is fixed by | |
forcing libcurl to close the connection on rebind. | |
Thanks to Noriko Hosoi for providing the patch. | |
- 2019-08-10: imjournal bugfix: state file close with fsync() was incorrect | |
This lead to fsync() now always applied where expected. | |
Thanks to Jiri Vymazal for the patch. | |
- 2019-08-10: testbench: add addtl test for multithreading and HUP | |
- 2019-08-10: imptcp bugfix: received bytes counter improperly maintained | |
imptcp counts the number of bytes received. However, receives | |
happen on different worker thread. The access to the counter | |
was not synchronized, which can cause loss of updates. Also, | |
thread debuggers validly flag this as an error, which creates | |
problems under CI. | |
This commit fixes the situation via atomic operations and | |
falls back to mutex calls if they are not available. | |
Detected by LLVM thread sanitizer. | |
closes https://github.com/rsyslog/rsyslog/issues/3798 | |
- 2019-08-07: testbench: add basic tests for omusrmsg | |
- 2019-08-05: omhttp bugfix: enable checkpath configuration parameter | |
omhttp, 'checkpath' option, was not configurable in the past. | |
- add 'checkpath' to the cnfparamdescr table. | |
- fix issue with checkpath passing extra garbage characters in string. | |
- add 'checkpath' into unit test - omhttp-retry.sh | |
Thanks to Nelson Yen for the fix. | |
- 2019-08-05: testbench bugfix: some tests were executed when req module was missing | |
In actual case if --enable-impstats was not given some other tests failed. | |
- 2019-08-03: iminternal bugfix: race on termination | |
This could in theory lead to loss of shutdown messages, but was mostly a | |
cosmetic issues. We primarily fixed it to get TSAN-clean so that we can | |
utilize LLVM TSAN in CI. | |
- 2019-08-02: testbench: new test for omfile outchannel functionality | |
- 2019-08-02: core/janitor bugfix: properly maintain dynafile cache | |
When the janitor cleans out timed-out files, it does not | |
properly indicate the entry is gone. Especially when running | |
in async mode this can lead to use-after-free and thus | |
memory corruption or segfault. | |
see also https://github.com/rsyslog/rsyslog/issues/3756 | |
- 2019-08-01: omfile bugfix: race file when async writing is enabled | |
This seems to be a long-standing bug, introduced around 7 years ago. | |
It became more visible by properly closing files during HUP, which | |
was done in 8.1905.0 (and was another bugfix). Note that due to this | |
race a memory corruption can occur under bad circumstances. As such, | |
this may have also caused segfaults or system hangs (mutexes could | |
have been affected). | |
closes https://github.com/rsyslog/rsyslog/issues/3772 | |
- 2019-08-01: testbench: additional tests for HUP | |
- 2019-07-31: imrelp bugfix: hang after HUP | |
termination condition was not properly checked; this lead to | |
premature termination after patch 1c8712415b9 was applied. | |
It is open to debate if patch 1c8712415b9 changed the module | |
interface. Actually it looks like this was previously not | |
well thought out. | |
closes https://github.com/rsyslog/rsyslog/issues/3760 | |
- 2019-07-24: mmdarwin: add new module | |
This is a contributed module. For details see doc. | |
Thanks to the Advens team for contributing it. | |
- 2019-07-23 iminternal bugfix: suppress mutex double-unlock | |
If there is a burst of log messages during a time when rsyslog is unable | |
to output (either during log rotation, an out-of-space condition, or | |
some other similar condition), rsyslog can SEGFAULT due to a mutex | |
double-unlock. | |
- 2019-07-23 imtcp: enable listenPortFileName parameter | |
this parameter was added, but it had no effect as it was not | |
passed down to the driver layer. This has been fixed. That also | |
now enables us to use dynamically-assigned port, which are | |
very useful for further testbench stabilization. Quite some | |
false positives occurred because the pre-selected port was | |
already in use again when rsyslog started. | |
- 2019-07-19 imtcp: enable listenPortFileName parameter | |
this parameter was added, but it had no effect as it was not | |
passed down to the driver layer. This has been fixed. That also | |
now enables us to use dynamically-assigned port, which are | |
very useful for further testbench stabilization. Quite some | |
false positives occurred because the pre-selected port was | |
already in use again when rsyslog started. | |
- 2019-07-18 core/action: no error file written if act suspended on TX commit | |
when an action was already disabled while the action was tried to be | |
committed, no error file was written. Note that this state is highly | |
unlikely to happen. Most probably, it can only happen if parameter | |
action.externalstate.file is used. | |
---------------------------------------------------------------------------------------- | |
Version 8.1907.0 (aka 2019.07) 2019-07-09 | |
NOTE TO MAINTAINERS: libee is not used by rsyslog for quite some while. | |
However, we never included this info into the changelog. So if you still | |
make rsyslog depend on libee (some do this), you should stop doing so now. | |
Libee is dead and no longer been maintained nor hosted by us. Old versions | |
can still be found at github for those in need. | |
GENERAL NOTE: during 8.1907 scheduled release timeframe we changed the ChangeLog | |
format to include the date a change went into master branch. This is to provide | |
an easy way to identify which changes went into the respective daily stable. | |
- 2019-07-05 imuxsock: support FreeBSD 12 out of the box | |
FreeBSD 12 uses RFC5424 on the system log socket by default. This | |
format is not supported by the special parser used in imuxsock. | |
Thus for FreeBSD the default needs to be changed to use the | |
regular parser chain by default. That is all this commit does. | |
closes https://github.com/rsyslog/rsyslog/issues/3694 | |
- 2019-07-05 function bugfix: "ipv42num" misspelled as "ip42mum" (without "v") | |
To fix the issue but keep compatible with existing deployments | |
both function names are now supported. | |
closes https://github.com/rsyslog/rsyslog/issues/3676 | |
- 2019-07-04 fix leading double space in rsyslog startup messages | |
see also https://github.com/rsyslog/rsyslog/issues/2979 | |
- omamqp1: port to latest api, add tests | |
This brings omamqp1 up-to-date with the latest qpid-proton-c | |
api version. This also adds a test for the plugin, to test | |
the basic functionality. The test requires the user to | |
install qdrouterd and the python qpid-proton library in order | |
to use the simple_recv.py test program. | |
Thanks to Richard Megginson for the patch. | |
- omclickhouse bugfix: potential segfault on omclickhouse batchmode | |
segfault happened when the template did not contain the string | |
"VALUES". | |
Thanks to github user wdjwxh for the fix. | |
- core bugfix: message duplication copied incorrect timestamp | |
MsgDup() placed timereported into timegenerated property, resulting | |
in invalid property values. Original timegenerated was lost. This | |
occured always when a message needed to be duplicated. Most | |
importantly this is the case when queues are used. | |
closes https://github.com/rsyslog/rsyslog/issues/3716 | |
- core bugfix: segfault on startup depending on queue file names | |
rsyslog will segfault on startup when a main queue file name has | |
been set and at least on other queue contains a file name. This | |
was cased by too-early freeing config error-detection data | |
structures. It is a regression caused by commit e22fb205a3. | |
Thanks to Wade Simmons for reporting this issue and providing | |
detailed analysis. That greatly helps fixing it quickly. | |
closes https://github.com/rsyslog/rsyslog/issues/3681 | |
- core "bugfix": alignment issue | |
This was not a hard error on current platforms, but a | |
to-be-considered compiler warning regarding invalid alignment. | |
While it works well on current platforms, alignment issues may | |
turn into real issues in future platforms. So we try to fix them | |
if possible. As not only a side-effect this resolves compiler | |
warnings even on current platforms. | |
This fix has some regression potential. If so, the problems | |
may occur during IP address resolution. | |
see also https://github.com/rsyslog/rsyslog/issues/2608 | |
- omfile bugfix: potential hang/segfault on HUP of dynafile action | |
when omfile was HUPed it did not sufficiently clear all dynafile | |
cache maintenance data structures. This usually lead to misaddressing | |
and could result in various issues, including a hang of rsyslog | |
processing or segfaults. It could also have "no effect" by pure | |
luck of not hitting anything important. This actually seems to | |
have been the most frequent case. | |
This seems to be a long-standing bug, but the likelihood of its | |
appearance seems to have been increased by commit 62fbef7 | |
introduced in 8.1905. Note: the commit itself has no regression, | |
just increases the likelihood to trigger the pre-existing bug. | |
special thanks to Alexandre Guédon for his help in analyzing | |
the issue - without him, we would probably still not know | |
what actually went wrong. | |
closes https://github.com/rsyslog/rsyslog/issues/3686 | |
- imjournal bugfix: potential message duplication | |
When journal was preloaded from previously saved cursor it was not advanced | |
to next entry so reading begun from last message which was therefore | |
duplicated. | |
Thanks to Jiri Vymazal for the patch. | |
- rfc5424 parser bugfix: leading space sometimes lost | |
if structured data is present a leading space in MSG field is lost | |
- queue subsystem bugfix: oversize queue warning message shown as error | |
The warning message was emitted as an error message, which is misleading | |
and may also break some automated procedures. | |
- core bugfix: HUP did not work reliable on all platforms | |
most notably not on FreeBSD, maybe others. The reason was obviously | |
different handling of signals in respect to multiple threads. | |
- build system bugfix: missing files in distribution tarball | |
- testbench | |
* fixed "make distcheck" settings which were missing some modules | |
This lead to incomplete "make distcheck" run; some errors were not | |
detected due to that. | |
* testbench framework: use ip tool instead of outdated ifconfig | |
The framework now first checks if "ip" is available and falls back | |
to "ifconfig" only if this is not the case. | |
Thanks to Michael Biebl for the suggestion. | |
closes https://github.com/rsyslog/rsyslog/issues/3682 | |
------------------------------------------------------------------------------ | |
Version 8.1905.0 (aka 2019.05) 2019-05-28 | |
- templates: add datatype template option for JSON generation | |
The new "datatype" and "onEmpty" template options permits to | |
generate non-string data rather easily. It works together with | |
jsonf formatting, which is what people should use nowadays. | |
closes https://github.com/rsyslog/rsyslog/issues/2827 | |
- config processing: check disk queue file is unique | |
If the same name is specified for multiple queues, the queue files | |
will become corrupted. This commit adds a check during config parsing. | |
If duplicate names are detected the config parser errors out and the | |
related object is not created. | |
Note: this may look to a change-of-behavior to some users. However, | |
this never worked and it was pure luck that these users did not run | |
into big problems (e.g. DA queues were never going to disk at the | |
same time). So it is acceptable to error out in this hard error case. | |
closes https://github.com/rsyslog/rsyslog/issues/1385 | |
- global config: new parameters for ruleset queue defaults | |
specifically: | |
* default.ruleset.queue.timeoutshutdown | |
* default.ruleset.queue.timeoutactioncompletion | |
* default.ruleset.queue.timeoutenqueue | |
* default.ruleset.queue.timeoutworkerthreadshutdown | |
closes https://github.com/rsyslog/rsyslog/issues/3656 | |
- add capability to write full config file (-o cmdline option) | |
Introduces the capability to create an output config file that explodes | |
all "includes" into a single file. This provides a much better overview | |
of how exactly the configuration is crafted. That could often be a great | |
troubleshooting aid. | |
This commit also contains some slight not-really-related cleanup. | |
closes https://github.com/rsyslog/rsyslog/issues/3634 | |
- queue subsystem: permit to disable "light delay mark" | |
New semantic: if lightDelayMark is 0, it is set to the max queue | |
size, effectively disabling the "light delay" functionality. | |
Thanks to Yury Bushmelev to mentioning issues related to light | |
delay mark and proposing the solution (which actually is what | |
this commit does). | |
closes https://github.com/rsyslog/rsyslog/issues/1778 | |
- queue subsystem: provide better user status messages | |
The queue subsystem now provides additional information messages which | |
may help a regular user to maintain system health. Most importantly, | |
DA queues now output when they persist queue data at end of run and | |
when they restart the queue based on persisted data. | |
- core: emit a warning message for ultra-large queue size definitions | |
We see error reports from users who have configured excessively large queues | |
and receive an OOM condition or other problems. | |
With that patch we generate a warning message if a queue is configured very | |
large. "Very large" is defined to be in excess of 500000 messages. | |
see also https://github.com/rsyslog/rsyslog/issues/3314 | |
closes https://github.com/rsyslog/rsyslog/issues/3334 | |
- new global config parameter "internalmsg.severity" | |
permits to specify a severity filter for internal message. Only | |
messages with this severity level or more severe are logged. | |
Originally this was done in rsyslog.conf as usual: you can filter | |
rsyslog messages on severity, just like any other. But with systemd, | |
we now emit primarily to the journal, and this is outside of rsyslog's | |
rule engine and so regular filters do not apply (at least in regard | |
to the journal). Logging to journal is good, because finally | |
folks begin to see the messages (traditional distro configs discard | |
them, for whatever is the reason). | |
This commit implements a global setting for a severity-based filter | |
for internal messages, before submitted to journal. So it's not 100% | |
of what rsyslog can do, but at least some way to customize. | |
see also https://github.com/rsyslog/rsyslog/issues/3639 | |
- config processing bugfix: error messages if config.enabled="off" is used | |
Using config.enabled="off" could lead to error messages on | |
"parameter xxx not known", which were invalid. They occured | |
because the config handler expected them to be used, which | |
was not the case due to being disabled. | |
This commit fixes that issue. | |
closes https://github.com/rsyslog/rsyslog/issues/2520 | |
- core portability bugfix: harden shutdown processing on FreeBSD | |
On FreeBSD, rsyslog does not always terminate immediately on SIGTERM. | |
Root cause seems to be that SIGTERM is delivered differently under | |
FreeBSD. This causes the main thread to not be awaken, and so it | |
takes until the next janitor interval to come back to life - which | |
can be far too long. Fixed this bug explicitly awaking the main | |
thread. | |
- imtcp bugfix: oversize message truncation causes log to be garbled | |
The actual problem is in the tcpserver component. However, the prime user | |
is imtcp and so users will likely experience this as imtcp problem. | |
When a too-long message is truncated, the byte after the truncation | |
position becomes the first byte of the next message. This will garble | |
the next messages and in almost all cases render it is syslog-noncompliant. | |
The same problem does NOT occur when the message is split. | |
This commit fixes the issue. It also includes a testbench fix. | |
Unfortunately the test for exactly this feature was not properly | |
crafted and so could not detect the problem. | |
closes https://github.com/rsyslog/rsyslog/issues/3580 | |
- omfile bugfix: FlushOnTXEnd does not work reliably with dynafiles | |
The flush was only done to the last dynafile in use at end of | |
transactions. Dynafiles that were also modified during the | |
transaction were not flushed. | |
Special thanks to Duy Nguyen for pointing us to the bug and | |
suggesting a solution. | |
This commit also contains a bit of cosmetic cleanup inside | |
the file stream class. | |
closes https://github.com/rsyslog/rsyslog/issues/2502 | |
- lmcry_gcry build bugfix: was not always properly build | |
Due to an invalid definition in build system this seems to have not | |
been correctly build on at least some platforms (but it worked on | |
others as it passed CI testing). This has now been corrected. | |
Thanks to Remi Locherer for the patch. | |
- dnscache bugfix: very unlikely memory leak | |
This fixes a memory leak that can only occur under OOM conditions. | |
Detected by Coverity Scan, CID 203717 | |
- testbench bugfix: wrong parameter check in diag.sh (tcpflood()) | |
When first parameter is check_only, the tcpflood funtion shall not | |
abort the test itself (The fail is intended if this option is set). | |
closes issue #3625 | |
- testbench bugfix: imfile-symlink test failed w/ parallel test run | |
The test sometimes failed. It used a symlink to a hardcoded name | |
rsyslog-link.*.log. This symlink was created but then disappears. | |
The reason is that upon (every!) test exit, rsyslog-link.*.log is | |
deleted. So a parallel test running the exit procedure just at the | |
"right" time can removed that file. | |
The bug is that the file name should be created using the tests's | |
dynamic name. This is done now. | |
closes https://github.com/rsyslog/rsyslog/issues/3550 | |
------------------------------------------------------------------------------ | |
Version 8.1904.0 (aka 2019.04) 2019-04-16 | |
- omfile: provide more helpful error message on file write errors | |
now contains actual file name plus a link to probable causes for this type | |
of problem | |
- imfile: emit error on startup if no working directory is set | |
When the work directory has not been set or is invalid, state files | |
are created in the root of the file system. This is neither expected | |
nor desirable. We now complain loudly about this fact. For backwards | |
compatibility reasons, we still need to support running imfile in | |
this case. | |
closes https://github.com/rsyslog/rsyslog/issues/1296 | |
- dnscache: add global parameter dnscache.default.ttl | |
This permits to control default TTL for cache entries. If set | |
to 0, the DNS cache is effectively disabled. | |
closes https://github.com/rsyslog/rsyslog/issues/49 | |
closes https://github.com/rsyslog/rsyslog/issues/1487 | |
- omelasticsearch: new parameter rebindinterval | |
Thanks to Richard Megginson for the patch. | |
- omelasticsearch: new parameter skipverifyhost | |
Add ability to specify the libcurl CURLOPT_SSL_VERIFYHOST | |
option to skip verification of the hostname in the peer cert. | |
WARNING: This option is insecure, and should only be used | |
for testing. The default value is off, meaning, the hostname | |
will be verified by default. | |
Thanks to Richard Megginson for the patch. | |
- omelasticsearch: set rawmsg to data from original request | |
Previously, when constructing the message to submit for a retry | |
for an original request, if the original request did not contain | |
the field `message`, the system property `rawmsg` was set to | |
the entire metadata + data from the original request. This was | |
causing problems with Elasticsearch. This patch changes | |
the code so that the `rawmsg` will be set to only the data part | |
of the original request if there is no `message` field. | |
closes https://github.com/rsyslog/rsyslog/issues/3573 | |
Thanks to Richard Megginson for the patch. | |
- mmkubernetes - support for metadata cache expiration | |
New parameters for mmkubernetes (module and action): | |
* `cacheexpireinterval` | |
If `cacheexpireinterval` is -1, then do not check for cache expiration. | |
If `cacheexpireinterval` is 0, then check for cache expiration. | |
If `cacheexpireinterval` is greater than 0, check for cache expiration | |
if the last time we checked was more than this many seconds ago. | |
* `cacheentryttl` - maximum age in seconds for cache entries | |
New statistics counters: | |
* `podcachenumentries` - the number of entries in the pod metadata cache. | |
* `namespacecachenumentries` - the number of entries in the namespace | |
metadata cache. | |
* `podcachehits` - the number of times a requested entry was found in the | |
pod metadata cache. | |
* `namespacecachehits` - the number of times a requested entry was found | |
in the namespace metadata cache. | |
* `podcachemisses` - the number of times a requested entry was not found | |
in the pod metadata cache, and had to be requested from Kubernetes. | |
* `namespacecachemisses` - the number of times a requested entry was not | |
found in the namespace metadata cache, and had to be requested from | |
Kubernetes. | |
- imdocker: new contributed module | |
imdocker will get (docker) container logs from a host as well as filling | |
out some basic container metadata as id, name, image, labels. | |
Thanks to Nelson Yen for the contribution. | |
- mmtaghostname: new contributed module | |
This module allows to force hostname after parsing to the localhostname of | |
rsyslog and/or add a tag to messages received from input modules without | |
tag parameter. | |
Thanks to Philippe Duveau for the contribution. | |
- imbatchreport: new contributed input module | |
This input module manage batches' reports : complete file as a single log. | |
Thanks to Philippe Duveau for the contribution. | |
- imtuxedolog: new contributed input module for Tuxedo ULOG | |
Thanks to Philippe Duveau for the contribution. | |
- openssl network driver: Added support setting openssl configcommands | |
We are using the gnutlsPriorityString setting variable, to pass | |
configuration commands to openssl. | |
closes: https://github.com/rsyslog/rsyslog/issues/3605 | |
- omkafka: drop messages rejected due to being too large | |
Drop messages that were rejected due to | |
'RD_KAFKA_RESP_ERR_MSG_SIZE_TOO_LARGE' error | |
Thanks to Nelson Yen for the patch | |
- core/action: implement capability to resume/suspend via external file | |
It has been reported that some TCP receivers exists that accept syslog tcp | |
messages at any rate, even if they do not manage to actually process them. | |
Instead, they silently drop the message. This behavior is not configurable. | |
All in all, it can lead to considerate message loss. | |
To support such use cases, we need to provide an ability to externally | |
trigger actions suspension and resumption. | |
We do this via a configured file which contains the status of the action. | |
Rsyslog periodically reads the file and if it contains "SUSPEND", it | |
suspend the action (and likewise for resume). | |
closes https://github.com/rsyslog/rsyslog/issues/2924 | |
- improg bugfix: some memory leaks | |
Thanks to Philippe Duveau for the contribution. | |
- msg object bugfix: regression from 1255a67 | |
closes https://github.com/rsyslog/rsyslog/issues/3570 | |
- pmnormalize: fix memory leaks, improve tests | |
This patch fixes a set of problems plus provides more and enhanced | |
tests for the module. | |
Most important problem was a memory leak that occured when a message | |
could not be passed at all. For each message that could not be parsed | |
memory of at least the size the message is leaked. Depending on | |
traffic pattern this can quickly lead to OOM. Note, however, that | |
this leak was never reported - it was discovered as part of code | |
review. | |
closes https://github.com/rsyslog/rsyslog/issues/2007 | |
- omkafka bugfix: build failure due to inconsistent type | |
fails depending on platform and settings; was somehow undetected by CI | |
- imjournal bugfix: potential segfault on some API failure returns | |
In one case there was possibility of free()'d value of journal | |
cursor not being reset, causing double-free and crash later on. | |
closes https://github.com/rsyslog/rsyslog/issues/3537 | |
- openssl subsystem bugfix: better error handling | |
Handling of SSL_ERROR_SYSCALL has been hardened. | |
Handling for SSL_Shutdown errors has been corrected. | |
Also fixed SSL Shutdown handling in tcpflood (openssl code). | |
If SSL_Shutdown returns error, we call SSL_read as described in | |
the documentation to do a bidirectional shutdown. | |
Closes https://github.com/rsyslog/rsyslog/issues/3561 | |
- imjournal bugfix: Fetching journal cursor only for valid journal | |
The sd_journal_get_cursor() got called regardless of previous | |
retcodes from other journal calls which flooded logs with journald | |
errors. Now skipping the call in case of previous journal call | |
non-zero result. Fixed success checking of get_cursor() call | |
to eliminate double-free possibility. | |
Also, making WorkAroundJournalBug true by default, as there were no | |
confirmed performance regressions for a quite long time. | |
Thanks to Jiri Vymazal for the patch. | |
- omamqp: fix build errors | |
They occur on some, newer, platforms. We do not really fix them, but rather | |
make the compiler ignore them. This is not really good, but the module is | |
contributed and so that's for now the best thing we can do. | |
- testbench: change manytcp.sh to use a larger connection count again | |
not sure why it was reduced, maybe related to | |
https://github.com/rsyslog/rsyslog/issues/1108 | |
also, modernize this and another test | |
- tcpflood bugfix: make soft connection limit work again | |
It looks like the soft limit became defunct when tcpflood was enhanced to | |
request more open file handles from OS. | |
closes https://github.com/rsyslog/rsyslog/issues/1108 | |
- testbench bugfix: omhttp tests were not run during "make distcheck" | |
- build system bugfix: omhttp test files were not included in dist tarball | |
Thanks to Thomas D. (whissi) for the patch. | |
------------------------------------------------------------------------------ | |
Version 8.1903.0 (aka 2019.03) 2019-03-05 | |
- omrabbitmq: add features (RabbitMQ HA management, templatize routing_key, | |
populate amqp message headers, delivery_mode and expiration parameters) | |
- improg: create input module to use external program as input datas | |
- imtuxedoulog: create input module to consume Tuxedo ULOG files | |
- omhttp: rewritten with large feature enhancements | |
Many thanks to Gabriel Intrator for this work. Gabriel also has adopted the | |
module and plans to support it in the future. | |
- pmdb2diag: create parser module for DB2 diag logs | |
- TLS subsystem: add support for certless communication | |
both openssl and GnuTLS drivers have been updated to support certless | |
communications. In this case e.g. Diffie-Helman is used. | |
NOTE: this is an insecure mode, as it does NOT guard against | |
man-in-the-middle attacks. We implemented it because of the large demand, | |
not because we think it makes sense to use this mode. We strongly recommend | |
against it. | |
closes https://github.com/rsyslog/rsyslog/issues/1068 | |
- imrelp/omrelp: add capability to specify tlslib for librelp | |
closes https://github.com/rsyslog/rsyslog/issues/3451 | |
- build system: introduce a better way to handle compiler pragmas | |
we now use macros and _Pragma(). This requires less code lines and is more | |
portable. | |
- omkafka: add support for dynamic keys | |
A new configuration property "dynaKey" is added that, when "on", changes the | |
value of property "key" to a template names instead of a constant value. | |
This is similar in approach to the DynaTopic implementation. | |
Thanks to Ludo Brands for the patch. | |
- AIX port: add AIX linking extensions on many plugins and contributions to | |
allow building them on this os. | |
- template: add Time-Related System Property $wday which is the day of week | |
This allow to get a week based rotation of log as AIX does. | |
- ksi subsystem: add high availability mode | |
Note: ksi subsystem now REQUIRES libksi 3.19.0 or above | |
Thanks to Allan Park for the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/3338 | |
- imfile bugfix: file reader could get stuck | |
State file handling was invalid. When a file was moved and re-created | |
rsyslog could use the file_id if the new file to write the old files' | |
state file. This could make the file reader stuck until it reached the | |
previous offset. Depending on file sizes this could never happen AND | |
would cause large message loss. This situation was timing dependent | |
(a race) and most frequently occurred under log rotation. In polling | |
mode the bug was less likely, but could also occur. | |
closes: https://github.com/rsyslog/rsyslog/issues/3465 | |
closes: https://github.com/rsyslog/rsyslog/issues/3468 | |
- imfile bugfix: potential segfault when working with directories or symlinks | |
see also https://github.com/rsyslog/rsyslog/pull/3496 | |
Thanks to Nelson Yen for the patch | |
- omhttp bugfix: header items could not have spaces in them | |
Thanks to Nathan Brown for the patch. | |
- core bugfix: enlarged msg offset types for bigger structured messages | |
using a large enough (dozens of kBs) structured message | |
it is possible to overflow the signed short type which leads | |
to rsyslog crash. (applies to msg.c, the message object) | |
Thanks to Jiri Vymazal for the patch. | |
- core bugfix for AIX: timeval2syslogTime now handle the bias according to | |
local time zone as documented by IBM. | |
- imfile feature: add configuration parameter to force parsing of read logs | |
- imczmq bugfix: | |
Release zframe following read from socket | |
Make the 0MQ frame pointer local to the receive loop and destroy the | |
frame as soon as the contents have been copied. This avoids: | |
* a memory leak should the receive loop execute more than once | |
* referencing an un-initialized value during cleanup (finalize_it) | |
Thanks to Mark Gillott for the patch. | |
- omclickhouse bugfix: default template unusable | |
STDSQL option added to the default template used in output module of clickhouse | |
Thanks to gagandeep trivedi for the patch. | |
- omclickhouse "bugfix": work-around failed error detection | |
omclickhouse uses a questionable method to check if a request generated | |
an error. We have seen the method to fail when we slightly upgraded clickhouse | |
server in CI testing. | |
This commit makes the method a bit more reliable without really fixing it. | |
But it's at least a short-term solution. | |
This should be changed to a proper status check. I assume such is possible. | |
see also https://github.com/rsyslog/rsyslog/issues/3485 | |
- imptcp bugfix: overly long socket bind path can lead to segfault | |
if the `path` input parameter is overly long (e.g. more than 108 | |
characters on some platforms) a non-terminated string is generated | |
and then passed to OS API. This can lead to all sorts of problems | |
including segfault. | |
We detected that based on gcc-8 warnings during code inspection. | |
No real-world problem case is known. | |
- ommongodb bugfix: improper stpncpy() calls | |
- testbench tcpflood: add new transport option relp-tls | |
Tcpflood can now send messages via relp with tls support. | |
closes https://github.com/rsyslog/rsyslog/issues/3448 | |
- testbench: mmdb valgrind tests failed is srcdir env was not set | |
- testbench: add omclickhouse tests | |
- testbench bugfix: some long-running tests had too low runtime allowance | |
closes https://github.com/rsyslog/rsyslog/issues/3493 | |
- testbench bugfix: daqueue-dirty-shutdown test | |
This test occasionally failed with left-over spool files. As far as we | |
have analyzed, this is due to the use of an invalid shutdown timeout | |
(very short) in the second phase of the test. It looks like this is | |
actually a copy&paste error from phase one. Behavior of rsyslog was | |
correct, but the test itself created a false positive. | |
We have corrected the timeout now and also modernized the test | |
a bit. | |
closes https://github.com/rsyslog/rsyslog/issues/2122 | |
- testbench bugfix: some omhttp tests had compatibility issues with Python 3 | |
Thanks to Thomas D. (whissi) for the patch. | |
------------------------------------------------------------------------------ | |
Version 8.1901.0 (aka 2019.01) 2019-01-22 | |
- new version scheme: 8.yymm.0 - version now depends on release date | |
see also https://rainer.gerhards.net/2018/12/rsyslog-version-numbering-change.html | |
- queue: add support for minimum batch sizes | |
closes https://github.com/rsyslog/rsyslog/issues/495 | |
- change queue.timeoutshutdown default to 10 for action queues | |
The previous default of 0 gave action queues no real chance to | |
shutdown - at the time they were applied, they were usually already | |
expired (computing the absolute timeout took a small amount of time). | |
So we change this now to 10ms, which still is very quick but gives | |
the queue at least a chance to shutdown itself. That in turn | |
smoothes the whole shutdown process. | |
If a very large number of action queues is used this may lead | |
to a very slightly longer shutdown time, albeit this is very | |
improbable. | |
- omclickhouse: new output module for clickhouse | |
This output module adds the possibility to send | |
INSERT querys to a Clickhouse database. See doc for details. | |
The messages are sent via a REST interface. | |
This commit also adds support of the testbench | |
for clickhouse tests, as well as various tests. | |
Closes https://github.com/rsyslog/rsyslog/issues/2272 | |
- omkafka: Add ability to dump librdkafka statistics to a file | |
Use statsFile to specify statistics output file; also requires | |
setting statistics.interval.ms confparam to a non-zero value. | |
Thanks to github user pcullen65 for the contribution. | |
- tls(ossl/gtls): add new Option "StreamDriver.PermitExpiredCerts" | |
The new Option can have one of the following values: | |
on = Expired certificates are allowed | |
off = Expired certificates are not allowed | |
warn = Expired certificates are allowed but warning will be logged (Default) | |
Includes necessary tests to validate new code. | |
closes https://github.com/rsyslog/rsyslog/issues/3364 | |
- action: add "action.resumeIntervalMax" parameter | |
This parameter permits to set an upper limit on the growth of the | |
retry interval. This is most useful when a target has extended | |
outage, in which case retries can happen very infrequently. | |
closes https://github.com/rsyslog/rsyslog/issues/3401 | |
- report child process exit status according to config parameter | |
Add new global setting 'reportChildProcessExits' with possible values | |
'none|errors|all' (default 'errors'), and new global function | |
'glblReportChildProcessExit' to report the exit status of a child | |
process according to the setting. | |
Invoke the report function whenever rsyslog reaps a child, namely in: | |
- rsyslogd.c (SIGCHLD signal handler) | |
- omprog | |
- mmexternal | |
- srutils.c (execProg function, invoked from stream.c and omshell) | |
Remove redundant "reaped by main loop" info log in omprog. | |
Promote debug message in mmexternal indicating that the child has | |
terminated prematurely to a warning log, like in omprog. | |
closes https://github.com/rsyslog/rsyslog/issues/3281 | |
Thanks to Joan Sala for contributing this. | |
- build system: add capability to turn off helgrind tests | |
we add configure switch --enable-helgrind. We need to turn helgrind off | |
when we use clang coverage instrumentation. The instrumentation injects | |
mt-unsafe counter updates which we seem to be unable to suppress. | |
Note: for gcc this was possible, because they all occured in a utility | |
function. For clang, they are inlined so we get many -and changing- violations. | |
see also https://github.com/rsyslog/rsyslog/issues/3361#issuecomment-450502569 | |
- imzmq3/omzmq3: remove modules | |
according to @brianknox (their author) these modules are outdated: | |
https://twitter.com/taotetek/status/931860786959540224 | |
They are replaced by imczmq/omczmq and are no longer maintained. We put a | |
depreciation notice into the modules a year ago, and now it finally is time | |
to remove them. They do NOT build in any case, except if very old versions | |
of the 0mq ecosystem are used. | |
see also https://github.com/rsyslog/rsyslog/issues/2100 | |
closes https://github.com/rsyslog/rsyslog/issues/2103 | |
- bugfix omusrmsg: don't overwrite previous set _PATH_DEV value | |
Since commit 56ace5e418d149af27586c7c1264fccfbc6badf1, omusrmsg was broken | |
because "memcpy()" is not a suitable substitute for "strncat()" in this | |
context, it is actually replacing the previous added content. | |
Bug: https://bugs.gentoo.org/673004 | |
Closes: https://github.com/rsyslog/rsyslog/issues/3346 | |
Thanks to Thomas D. (whissi) for the patch. | |
- bugfix ossl TLS driver: fixed authentication mode anon | |
authentication mode "anon" was not properly supported in ossl TLS | |
driver; if selected, did still require a full certificate. | |
closes: https://github.com/rsyslog/rsyslog/issues/3037 | |
- bugfix tls subsystem: Receiver hang due to insufficient TLS buffersize. | |
gtls and ossl driver used a default buffersize of 8KiB to store received | |
TLS packets. When tls read returned more than buffersize, the additional | |
buffer was not processed until new data arrived on the socket again. | |
TLS RFCs require up to 16KiB+1 buffer size for a single TLS record. | |
closes https://github.com/rsyslog/rsyslog/issues/3325 | |
- bugfix pmpanngfw: build issue due to non-matching data types in comparison | |
Thanks to Narasimha Datta for the patch. | |
- omfile: work-around for "Bad file descriptor" errors | |
This works-around an issue we can reproduce e.g. via the | |
imtcp-tls-ossl-x509fingerprint.sh test. Here, omfile gets a write | |
error with reason EBADF. So far, I was not able to see an actual | |
coding error. However I traced this down to a multithreaded race | |
on open and close calls. I am very surprised to see this type | |
of issue, as I think the kernel guarantees that it does not happen. | |
Here is what I see in strace -f: | |
openssl accepts a socket: | |
[pid 66386] accept(4, {sa_family=AF_INET, sin_port=htons(59054), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 10 | |
then, it works a bit with that socket, detects a failure and shuts it down. Sometimes, at the very same instant omfile on another thread tries to open on output file. Then the following happens: | |
[pid 66386] close(10) = 0 | |
[pid 66389] openat(AT_FDCWD, "./rstb_356100_31fa9d20.out.log", O_WRONLY|O_CREAT|O_NOCTTY|O_APPEND|O_CLOEXEC, 0644 <unfinished ...> | |
[pid 66386] close(10 <unfinished ...> | |
[pid 66389] <... openat resumed> ) = 10 | |
[pid 66386] <... close resumed> ) = 0 | |
[pid 66386] poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, -1 <unfinished ...> | |
[pid 66389] write(2, "file './rstb_356100_31fa9d20.out"..., 66file './rstb_356100_31fa9d20.out.log' opened as #10 with mode 420 | |
) = 66 | |
[pid 66389] ioctl(10, TCGETS, 0x7f59aeb89540) = -1 EBADF (Bad file descriptor) | |
This is **literally** from the log, without deleting or reordering | |
lines. I read it so that there is a race between `open` and `close` | |
where fd 10 is reused, but seemingly closed - resulting in the `EBADF` | |
While it smells like a kernel issue, it may be a well-hidden program | |
bug - if so, one I currently do not find. HOWEVER, this commit | |
works around the issue by reopening the file when we receive EBADF. | |
That's the best thing to do in that case, especially if it really is | |
a kernel bug. Data loss should not occur, as the previous writes | |
succeeded in that case. | |
The backdraw of this work-around is that it only "fixes" omfile. In | |
theory every part of rsyslog can be affected by this issues (queue | |
files, for example). So this is not to be considered a final solution | |
of the root issues (but a big step forward for known problem cases). | |
see also https://github.com/rsyslog/rsyslog/issues/3404 | |
- omhttp bugfix: segfault due to NULL pointer access | |
many thanks to Gerardo Puerta for the patch | |
- omkafka bugfix: segfault when running in debug mode using dynamic topics | |
This should only affect test environments, as debug mode is not | |
suitable for production (and really does not work when running for | |
extended period of time). | |
- testbench bugfix: TLS syslog tests for "anon" mode were broken | |
They did not detect when "anon" mode was not properly supported by the | |
drivers. | |
- test tooling bugfix: correct tcpflood error messages | |
it looks like tcpflood's openssl code stems partly back to tcpdump, at | |
least the error messages indicate this. Thankfully tcpdump is BSD licensed, | |
so this should not be a big issue. Nevertheless, the incorrect program name | |
in error messages needs to be corrected, and this is what this commit does. | |
- tcpflood bugfix: tool did not terminate on certificate error | |
when tcpflood detected a certificate error, it reported an | |
error message but did not abort. This could make errors undetectable | |
during CI runs. | |
also fix tests which did not properly provide CA cert (which than | |
caused the error). | |
- testbench: fix issues with journal testing | |
The configure/Makefile checks were not correct, leading to the | |
build of journal components when not necessary, even if not | |
supported by the platform. Thus lead to invalid build and test | |
failures. | |
- testbench: add tests for "certless" tcp/tls | |
This adds a test to ensure that a client without certificate can | |
connect to a server with certificates. So it is not exactly | |
"certless". | |
The prime intent of this test is to match config suggestions given | |
by log hosting companies (like loggly) and so ensure that we do | |
not accidentally break them. This is especially important as the | |
capability for certless clients was not properly documented and | |
also become forgotten by the rsyslog team. | |
see also https://github.com/rsyslog/rsyslog/issues/3413 | |
- CI | |
- further improve testbench robustness against slow machines | |
- testbench: add tests for parser.EscapeControlCharacterTab global option | |
- testbench: Updated all expired x.509 certs | |
Closes https://github.com/rsyslog/rsyslog/issues/3348 | |
- fix a potential race in CI debug mode which can lead to segfault | |
only when instructed to do so, rsyslog may emit a "final worker thread shutdown" | |
messages. This is usually only enabled in CI and/or other testing. If enabled, | |
the code has a race on the pWti object which can lead to segfault or abort. | |
Only system which explicitly enable this CI aid are affected (running in debug | |
mode alone is NOT sufficient). | |
This is a regression from 8.40.0. | |
- testbench: improve robustness against slow CI, gen. improvements | |
* add an overall timeout value for tests - if running longer, | |
testbench framework tries to FAIL and end test. Note that | |
this is not bullet-proof and not intended to be so. | |
* guard against hanging rsyslog instances via a new imdiag | |
feature to abort after n number of seconds; among others, | |
this guards as against timeout-cancel in CI, which is always | |
pretty hard to diagnose - now we see these errors in test-suite.log | |
* fix a bug in tcp zip test, which actually did not use zip mode | |
* experimentally add debug output to better understand | |
shutdown_when_empty operation; goal is to improve understanding | |
and then remove that code again. | |
* improve shutdown predicate for a couple of tests | |
* made travis run make check with two parallel threads, for which | |
we seem ready now. Nevertheless, it's still experimental and we | |
may roll this back if required. | |
* testbench: disable omprog tests that hang under coverage instrumentation | |
When gcc coverage instrumentation is used, these tests hang. They work | |
with clang coverage instrumentation, but for some reason clang does not | |
give us full reports (at least not when used together with CodeCov.io). | |
We have tried to troubleshoot this for hours and hours - now is time to | |
give up until someone comes up with a bright idea. So we make the affected | |
tests skip themselves when they detect gcc with coverage instrumentation. | |
* testbench: add new test for imfile and logrotate in copytruncate mode | |
* testbench: add new omkafka tests for dynamic topics | |
* travis: do no longer run 0mq tests | |
This often causes trouble when the packages are rebuild by the 0mq project | |
(which happens frequently). We already do intensive testing of the 0mq | |
components in the buildbot infrastructure, where we use dedicated containers. | |
This is reliable, as the containers already contain everything needed and so | |
do not need to reach out to the 0mq package archives. In the light of this, | |
let's save us the trouble of Travis failures. The only downside is that | |
users cannot pre-test with their local Travis when modifying 0mq modules, | |
which is quite acceptable. | |
------------------------------------------------------------------------------ | |
Version 8.40.0 [v8-stable] 2018-12-11 | |
- mmkubernetes: add support for sslpartialchain for openssl | |
If `"on"`, this will set the OpenSSL certificate store flag | |
`X509_V_FLAG_PARTIAL_CHAIN`. This will allow you to verify the Kubernetes API | |
server cert with only an intermediate CA cert in your local trust store, rather | |
than having to have the entire intermediate CA + root CA chain in your local | |
trust store. See also `man s_client` - the `-partial_chain` flag. | |
This option is only available if rsyslog was built with support for OpenSSL and | |
only if the `X509_V_FLAG_PARTIAL_CHAIN` flag is available. If you attempt to | |
set this parameter on other platforms, you will get an `INFO` level log | |
message. This was done so that you could use the same configuration on | |
different platforms. | |
- openssl driver: improved error messages | |
also fixes misleading wording of some error messages | |
closes https://github.com/rsyslog/rsyslog/issues/3238 | |
- imfile: disable file vs directory error on symlinks | |
The file/directory node-object alignment now ignores symlinks. Previously | |
it reported error on each directory symlink spamming user error logs. | |
Thanks to Jiri Vymazal for the patch. | |
- cleanup: remove no longer needed --enable-rtinst code | |
configure option --enable-rtinst is gone-away since a while, but there were | |
still some supporting code left. It required careful analysis what could | |
actually be removed. This is now done and the code fully cleaned up. This | |
greatly simplifies the code and also makes it better readable for | |
developers which are not deep inside the rsyslog code base. | |
As a positive side effect, we could eliminate mutex calls inside | |
the debug system. This means we are more likely to reproduce race | |
conditions in runs with debugging enabled. | |
closes https://github.com/rsyslog/rsyslog/issues/2211 | |
- bugfix imfile: rsyslog re-sends data for files larger 2GiB | |
This occurs always if and only if | |
- reopenOnTruncate="on" is set | |
- file grows over 2GiB in size | |
Then, the data is continuously re-sent until the file becomes smaller | |
2GiB (due to truncation) or is deleted. | |
It is a regression introduced by 2d15cbc8221e385c5aa821e4a851d7498ed81850 | |
closes https://github.com/rsyslog/rsyslog/issues/3249 | |
- config: fix segfault in backticks "echo" expansion of undefined variables | |
The bug was introduced in commit abe0434 (config: enhance backticks "echo" | |
capability). The getenv() result passed to strlen() and es_addBuf() may be | |
NULL if the environment variable does not exist, resulting in a segfault. | |
Thanks to Julien Thomas for the patch. | |
fixes https://github.com/rsyslog/rsyslog/issues/3006 | |
- bugfix imsolaris: message timestamps on Solaris | |
On Solaris messages don't have their time directly in the raw body but in | |
a separate log_ctl structure which is currently not used. | |
When message is logged and processed, rsyslogd gives it current time because | |
it ignores the actual one. That means that old messages (e.g. from system | |
reboot) get timestamp of processing instead of the reboot itself (it is | |
not a problem for live logging where now is used anyway). | |
Thanks to Jakub Kulik for the patch. | |
- bugfix build system: "make distcheck" did not work for mysql tests | |
- bugfix build system: don't link liblogging-stdlog when available but not enabled | |
When liblogging-stdlog was available but configure option "--disable-liblogging-stdlog" | |
was set, rsyslog was still linking against liblogging-stdlog. | |
This commit will ensure that rsyslog will only link against liblogging-stdlog when | |
"--enable-liblogging-stdlog" was set. | |
see also: https://bugs.gentoo.org/667836 | |
- bugfix RainerScript: abs() could return negative value, now in range [0..max] | |
Thanks to Harshvardhan Shrivastava for providing the patch | |
- bugfix debug output: date property options output wrongly | |
inside debug logging, the date property options were not all | |
properly converted into strings. Some of the newer ones were | |
invalidly flagged as "UNKNOWN". This is primarily a cosmetic | |
problem and has no effect other than puzzling folks looking at | |
the debug log. | |
- bugfix omhttp: did not compile on some platforms | |
- CI | |
* made mysql-based tests (ommysql and omlibdbi) work inside containers | |
* bugfix testbench: do not execute libgcrypt tests if disabled | |
closes https://github.com/rsyslog/rsyslog/issues/3228 | |
* testbench: grep failed when string starting with "-" was used | |
The search term was mistakenly interpreted as an option. | |
* testbench: support auto-start/-stop of mysqld | |
This is required to run mysql/mariadb tests inside containers. | |
closes https://github.com/rsyslog/rsyslog/issues/3223 | |
* improve bash coding style and fix a some bug in testbench | |
- duplicate diag.sh init call was not detected due to typo | |
- queue-persists test did not work correctly | |
- some general testbench framework improvements | |
issues found be shellcheck, fixes brought up other work to do | |
* testbench: improve journal tests and testbench framework | |
improving both style and reliability of journal tests; along that way | |
also improve testbench framework: | |
- do cleanup on error_exit and skip | |
- explicit skip handler (vs exit 77) | |
this permits us to do better cleanup | |
- new testbench functions for journal-specific functionality | |
reduce code duplication and make things easier to maintain in the | |
future | |
- provide a way to do valgrind and non-valgrind tests with a single | |
test file | |
see also https://github.com/rsyslog/rsyslog/issues/2564 | |
* testbench: improve framework, harden rscript http test | |
- the test now tries to detect unavailable http server, which | |
should not result in test failure | |
- equivalent valgrind test changed to new method, removing code | |
duplication | |
- testbench supports | |
* new exit code 177, which indicates environment error, makes | |
test SKIP but still reports the failure | |
* new exitcode, logurl stats reporting fields | |
* report buildbot builder (if provided) in failure report | |
* testbench: add test for mmjsonparse with unparsable data | |
* testbench: make es-bulk-retry test more reliable | |
We now no longer depend on a fixed 'sleep' command but rather | |
check the output file for what we expect. This is much more | |
robust on slow test machines. | |
We believe this closes the below-mentioned issue. If not, it | |
should be re-opened. | |
closes https://github.com/rsyslog/rsyslog/issues/3104 | |
* testbench: suppress valgrind error caused by pthreads lib | |
finally I give up and honestly think this is a problem in pthreads and | |
not in rsyslog code. See issue below and previous commit for more | |
information. | |
Unfortunately, this will also mask off cases where we do not properly | |
call pthread_join() albeit it is needed. Nevertheless, this bug is | |
causing so much CI grief that it is definitely worth it. | |
closes https://github.com/rsyslog/rsyslog/issues/2902 | |
* testbench: made a couple of (unnamed due to too many) test more robust | |
against slow (CI) machines | |
------------------------------------------------------------------------------ | |
Version 8.39.0 [v8-stable] 2018-10-30 | |
- imfile: improve truncation detection | |
previously, truncation was only detected at end of file. Especially with | |
busy files that could cause loss of data and possibly also stall imfile | |
reading. The new code now also checks during each read. Obviously, there | |
is some additional overhead associated with that, but this is unavoidable. | |
It still is highly recommended NOT to turn on "reopenOnTruncate" in imfile. | |
Note that there are also inherent reliability issues. There is no way to | |
"fix" these, as they are caused by races between the process(es) who truncate | |
and rsyslog reading the file. But with the new code, the "problem window" | |
should be much smaller and, more importantly, imfile should not stall. | |
see also https://github.com/rsyslog/rsyslog/issues/2659 | |
see also https://github.com/rsyslog/rsyslog/issues/1605 | |
- imjournal: work around journald excessive reloading behavior | |
This is workaround for possible imjournal interaction with systemd | |
where journal invalidate fix is not present. The code tries to | |
detect SD_JOURNAL_INVALIDATE loop and not reload after each call. | |
Thanks to Jiri Vymazal for the patch. | |
- errmsg: remove no longer needed code | |
refactored code (over a long time) so that object-ish style is no longer | |
needed and could now finally be removed; We also refactored the last | |
component (omhttp contrib module) that used the old interface. | |
closes https://github.com/rsyslog/rsyslog/issues/1684 | |
- queue bugfix: invalid error message on queue startup | |
due to some old regression (commit not exactly identified, but for | |
sure a regression, 9 years ago it was correct) an error message | |
is emitted when no .qi file exists on startup of the queue, which | |
is a normal condition. | |
Actually, the code should not have tried to open the .qi file in | |
the first place because it detected that it did not exist. That | |
(necessary) shortcut had been removed a while ago. | |
closes https://github.com/rsyslog/rsyslog/issues/3117 | |
- bugfix imrelp: regression with legacy configuration startup fail | |
Startup of a relp listener failed if legacy configuration was used. | |
caused by commit: 32b71daa8aadb8f16fe0ca2945e54d593f47a824 | |
closes https://github.com/rsyslog/rsyslog/issues/3106 | |
- bugfix imudp: stall of connection and/or potential segfault | |
There was a regression in 493279b790a8cdace8ccbc2c5136985e820dd2fa. | |
This regression may cause stop (or delay) of reception from some systems | |
and may also cause a segfault. Triggering condition is that at least | |
one listener could not be created. | |
Thanks to Jens Låås for the patch. | |
- bugfix gcry crypto driver: small memleak | |
If a crypto key is specified directly via the key="" parameter, | |
the storage for that key is not freed, causing a small memleak. | |
Note that the problem occurs only once per context, so this | |
should not cause real issues. Even more so, as specifying a | |
key directly is meant only for testing purposes and is strongly | |
discouraged for production use. | |
Detected by internal testing, no actual fail case known. | |
- fix potential misaddressing in encryption subsystem | |
could happen if e.g. disk queues were encrypted | |
not seen in practice but caught by testbench test | |
- ksi subsystem changes | |
* enhance debug logging | |
* disable unsafe SHA1 algorithm | |
Thanks to Allan Park for the patch. | |
- bugfix core: regex compile error messages could be incorrect | |
- bugfix core: potential hang on rsyslog termination | |
The root cause was a deadlock during worker startup. This could | |
happen for example when a DA queue needed to persist data during | |
shutdown. | |
Fail condition: | |
* startup request for a new worker | |
* initialization of that worker | |
* immediate detection that the worker can or must shutdown | |
* main thread waiting for worker running state, which it skips, | |
and so the main thread hangs inside a loop | |
closes https://github.com/rsyslog/rsyslog/issues/3094 | |
- bugfix imkafka: system hang when backgrounded | |
imkafka initializes librdkafka too early (before the fork). This leads | |
to hangs in various parts of the system - not only im imkafka but | |
other functions as well (e.g. getaddrinfo() calls). | |
closes https://github.com/rsyslog/rsyslog/issues/3180 | |
- bugfix imfile: file change was not reliably detected | |
A change in the inode was not detected under all circumstances, | |
most importantly not in some logrotate cases. | |
Includes new tests made by Andre Lorbach. They now use the | |
logrotate tool natively to reproduce the issue. | |
closes https://github.com/rsyslog/rsyslog/issues/2659 | |
closes https://github.com/rsyslog/rsyslog/issues/1605 | |
- bugfix imrelp: do not fail build if librelp does not have relpSrvSetLstnAddr | |
closes https://github.com/rsyslog/rsyslog/issues/2938 | |
- bugfix queue subsystem: DA queue did ignore encryption settings | |
closes https://github.com/rsyslog/rsyslog/issues/3066 | |
closes https://github.com/rsyslog/rsyslog/issues/2575 | |
- bugfix KSI: lmsig-ksils12 module skips signing the last block | |
Thanks to Allan Park for the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/3105 | |
- bugfix fmhash: function hash64mod sometimes returned wrong result | |
Thanks to Harshvardhan Shrivastava for providing the patch | |
closes https://github.com/rsyslog/rsyslog/issues/3025 | |
- bugfix core/debug: data written to random fd 2 under some debug settings | |
This happens only during auto-backgrounding, where we cannot any longer | |
access stderr. Whatever is opened with fd2 receives some debug messages. | |
Note that the specific feature is usually turned on only in CI runs. | |
- cleanup: removed no longer needed code | |
Code that was unused for quite a while or did not really belong to the | |
project identified and removed. | |
- overall code cleanup | |
e.g. remove unused code, replace bad bash constructs, etc... | |
- CI: | |
* some small improvements in testbench plumbing | |
e.g. (`cmd` replaced by $(cmd), removed useless use of cat, ...) | |
* testbench: improve plumbing for kafka tests | |
- Removed all sleeps where possible. | |
- Moved all kafka start/stop/download logic into functions. | |
- Moved kafka/zookeeper stop into error_exit and exit_test. | |
- Kafka/Zookeeper cleanup only done on success now. | |
- Kafka/Zookeeper logfiles automatically dumped on error_exit only now. | |
- Added cleanup for Kafka/Zookeeper instances into CI/buildbot_cleanup.sh | |
- added new tests | |
* testbench: fix incompatibility of one omprog test with Python3 | |
Python3 writes to stderr immediately, and this caused the | |
captured output to differ with respect to Python2. Simplified | |
the test to do a single write to stderr. Also a cast to int | |
was needed when calculating 'numRepeats'. | |
closes https://github.com/rsyslog/rsyslog/issues/3030 | |
* testbench: fixed imfile parallel issues | |
- Fixed timing issues in some imfile wildcard/regex tests | |
- Added touch command in imfile wildcard tests to make sure directories | |
exist before files are created in it if IO is under stress. | |
- changed content checking in some tests to use "content_check_with_count" | |
with check timeouts instead of using fixed sleeptimes. | |
* testbench: new basic tests | |
These ensure that for some modules that did not have any tests at all | |
we have at least a minimal coverage (module loads, activates, is able | |
to emit error messages). Of course, further improvements would make | |
much sense. Modules: | |
- ommail | |
- testbench: new tests for disk queue encryption | |
- testbench: improved auto-diagnostics for hanging instance | |
- testbench: hardened kafka test against failing kafka subsystem, | |
not in 100% of the cases, but at least in some that frequently occur | |
- failing tests now report failure status so that we can get stats | |
on unreliable tests | |
- testbench tooling: fix incorrect tcpflood TLS parameter check | |
could lead to segfault when started | |
- bugfix testbench tooling: tcpflood invalid type in calloc (openssl mode) | |
It is unlikely that this has caused a real issue, as long as pointers | |
are all of the same size (what is highly probable). | |
detected by cppcheck via Codacy.com | |
------------------------------------------------------------------------------ | |
Version 8.38.0 [v8-stable] 2018-09-18 | |
- AIX: make basic modules work again | |
- make rsyslog build on AIX again | |
... at least for a limited set of default modules | |
- imfile: support for endmsg.regex | |
This adds support for endmsg.regex. It is similar to | |
startmsg.regex except that it matches the line that denotes | |
the end of the message, rather than the start of the next message. | |
This is primarily for container log file use cases such as this: | |
date stdout P start of message | |
date stdout P middle of message | |
date stdout F end of message | |
The `F` means this is the line which contains the final part of | |
the message. The fully assembled message should be | |
`start of message middle of message end of message`. | |
`startmsg.regex="^[^ ]+ stdout F "` will match. | |
Thanks to Richard Megginson for the patch. | |
- imkafka: add parameter "parseHostName" | |
This enables imkafka to parse the hostname from log message. | |
Previously that was not possible. It was most likely a bug, but | |
one that users may count on. The new parameter "ParseHostName" | |
(default is off) controls this behavior. Default is to NOT | |
parse the hostname. | |
Thanks to github user snaix for the contribution. | |
- im[p]tcp: improve error message on connect failure | |
Now a message with the actual OS error is emitted, making things far | |
easier to troubleshoot. | |
- imkafka: implement multithreading support for kafka consumers. | |
Each consumer runs in it's own consumer thread now. New tests have also | |
been added for this. | |
- omelasticsearch: write all header metadata to $.omes for retries | |
Write all of the original request metadata fields to $.omes for | |
the retry, if present. This may include all of the following: | |
_index, _type, _id, _parent, pipeline | |
This is in addition to the fields from the response. If the same | |
field name exists in the request metadata and the response, the | |
field from the request will be used, in order to facilitate | |
retrying the exact same request. | |
Thanks to Richard Megginson for the patch. | |
- core: improve error message on module load fail | |
The error message now lists all dlopen() errors in depth. This is | |
especially useful if the error is due to missing symbols or file | |
format errors. | |
- core/queue: add error message if queue file cannot be accessed | |
When having a disk-assisted queue without permission to write to the specified | |
queue file an error will now be generated. | |
closes https://github.com/rsyslog/rsyslog/issues/323 | |
- imtcp/imudp: new option preservecase for managing the case of FROMHOST value | |
default is left at current behavior | |
see also https://github.com/rsyslog/rsyslog/pull/2774 | |
see also https://bugzilla.redhat.com/show_bug.cgi?id=1309698 | |
- omprog: add feedback timeout and keep-alive feature | |
- Restart the program if it does not respond within timeout. | |
- New setting 'confirmTimeout' (default 10 seconds). | |
- Allow the program to provide keep-alive feedback when a | |
message requires long-running processing. | |
- Improve efficiency when reading feedback line (use buffer). | |
Retry interrupted writes/reads to/from pipe. | |
- New setting 'reportFailures' for reporting error messages | |
from the program. | |
- Report child termination when writing to pipe. | |
- Minor refactor: renamed writePipe function to sendMessage, | |
renamed readPipe to readStatus. | |
Thanks to Joan Sala for contributing this. | |
- omprog: fix forceSingleInstance configuration option | |
The forceSingleInstance option did not work as intended. Even | |
if set multiple instances were spawned. This most probably | |
was a regression from 0453b1670fc34c96d31ee7c9a370f0f5ec24744a | |
The code was broken roughly 3.5yrs ago, so it looks like the | |
issue was little-noticed. This also means that potentially some users | |
may see the bugfix as change of behavior. If so, just remove | |
the option. | |
closes https://github.com/rsyslog/rsyslog/issues/2813 | |
closes https://github.com/rsyslog/rsyslog/issues/2468 | |
Thanks to Joan Sala for contributing this. | |
- imfile: implement file-id, used in state file | |
This ensures that files with the same inodes are not accidentally treated | |
as equal, at least within the limits of the file id hash (see doc for | |
details). | |
We use the siphash reference implementation to generate our non-cryptographic | |
hash. | |
closes https://github.com/rsyslog/rsyslog/issues/2530 | |
closes https://github.com/rsyslog/rsyslog/issues/2231 | |
- imfile: experimental input throttling feature | |
The new input parameter delay.message has been added. It specifies | |
a delay in microseconds after each line read. | |
closes https://github.com/rsyslog/rsyslog/issues/2960 | |
- core: emit TZ warning on startup not on Linux non-container | |
On Linux it seems common that the TZ variable is NOT properly set. | |
There are some concerns that the warning related to rsyslog correcting | |
this confuses users. It also seems that the corrective action rsyslog | |
takes is right, and so there is no hard need to inform users on that. | |
In Linux containers, however, the warning seems to be useful as the | |
timezone setup there seems to be frequently-enough different and | |
rsyslog's corrective action may not be correct. | |
So we now check if we are running under Linux and not within a container. | |
If so, we do not emit the warning. In all other case, we do. This is | |
based on the assumption that other unixoid systems still should have | |
TZ properly set. | |
closes https://github.com/rsyslog/rsyslog/issues/2994 | |
- omkafka: | |
* better debug information | |
* Fixed minor issue in omkafka producing wrong kafka timestamps when | |
msgTimestamp was NULL. | |
* Setting RD_KAFKA_V_KEY(NULL, 0) in rd_kafka_producev now when KEY is not | |
configured. | |
* Fixed minor issue when rsyslog is compiled with --enable-debug and | |
librdkafka is too old. | |
- omfile bugfix: errant error message when dynafile param needed | |
also fixes related message in contributed module omfile-hardened | |
closes https://github.com/rsyslog/rsyslog/issues/2975 | |
Thanks to Frank Bicknell for the patch | |
- omhttp: new contributed module | |
Thanks to Christian Tramnitz for contributing it. | |
Some more info at https://github.com/rsyslog/rsyslog/pull/2782 | |
- mmkubernetes: action fails preparation cycle if kubernetes API ... | |
... destroys resource during bootup sequence | |
The plugin was not handling 404 Not Found correctly when looking | |
up pods and namespaces. In this case, we assume the pod/namespace | |
was deleted, annotate the record with whatever metadata we have, | |
and cache the fact that the pod/namespace is missing so we don't | |
attempt to look it up again. | |
In addition, the plugin was not handling error 429 Busy correctly. | |
In this case, it should also annotate the record with whatever | |
metadata it has, and _not_ cache anything. By default the plugin | |
will retry every 5 seconds to connect to Kubernetes. This | |
behavior is controlled by the new config param `busyretryinterval`. | |
This commit also adds impstats counters so that admins can | |
view the state of the plugin to see if the lookups are working | |
or are returning errors. The stats are reported per-instance | |
or per-action to facilitate using multiple different actions | |
for different Kubernetes servers. | |
This commit also adds support for client cert auth to | |
Kubernetes via the two new config params `tls.mycert` and | |
`tls.myprivkey`. | |
Thanks to Richard Megginson for the patch. | |
- bugfix pmnormalize/core: several memory leaks, invld property handling | |
- major memory leak which occurred once per message processed | |
So this could lead to OOM. Caused by improper free of json | |
structure | |
- another two major leaks of similar magnitude could occur if | |
"fromhost-ip" and/or "fromhost" properties were set | |
- minor leaks upon termination. these were unproblematic as | |
static and only occured immediately before shutdown. | |
But they triggered memory debugger errors. | |
- fixed test which did not check for mem leaks albeit it should | |
- core invalid handling of the "fromhost" property, if set via | |
the MsgSetPropsViaJSON() call. This was primarily of concern | |
for pmnormalize and mmexternal, and only if these properties | |
were used by either the rulebase or the external program | |
response. | |
Actually, most of the leaks go back to rsyslog core, but that | |
core functionality was not used by other modules in the same | |
way. But if some other would have used it, the effects would | |
have been the same (so be aware if you wrote custom modules). | |
- bugfix imptcp: fixed pointers for session counting | |
imptcp open, failedopen, and closed pstats counters were assigned the wrong | |
name, thus pstats values did provide a totally wrong picture of what was | |
going on. | |
Thanks to github user jeverakes for the patch. | |
- bugfix omprog: invalid memory access on partial writes to pipe | |
When sending logs to the program, in case of a partial write to the pipe, | |
invalid data was sent, or an invalid memory access could occur. (A | |
partial write can occur if the syscall is interrupted or the pipe is full.) | |
Thanks to Joan Sala for contributing this. | |
- bugfix omprog: rsyslog's environment was not passed to script | |
closes https://github.com/rsyslog/rsyslog/issues/2921 | |
- bugfix omprog: severity of some log messages in waitForChild corrected | |
Log some messages related to child process termination as info/warn | |
instead of error. | |
- bugfix imfile: files which were loaded via symlink were not always followed | |
They were stopped watching after being rotated. | |
closes https://github.com/rsyslog/rsyslog/issues/2229 | |
Thanks to Jiri Vymazal for the patch. | |
- bugfix imfile: potential misaddressing when processing symlinks | |
Fixed parent name when processing symlinks. Detected during code review. | |
There was a garbage byte left before which could cause errors down the | |
road. | |
Thanks to Jiri Vymazal for the patch. | |
- bugfix ommongodb: build issue if mongo-c-driver is not compiled with TLS | |
Let ommongodb module works even if mongo-c-driver is not compiled with SSL support. | |
Thanks to Jérémie Jourdin for the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/2907 | |
- CI: | |
* many changes with the goal to support parallel test execution, e.g. | |
use dynamic ports and file names, changes to testing tools, etc. | |
* kafka tests re-enabled, as they should now no longer be racy. However, | |
this has yet to be proven in practice. | |
* upgrading kafka server version to current | |
* Fixed server configuration issues holding the kafka tests back from working | |
* Fixed some config issues in all sndrcv kafka tests. | |
* Generating dynamically kafka topics now for each kafka test. | |
* Reenabled kafka_multi test which runs a test on 3 kafka/zookeeper instances | |
------------------------------------------------------------------------------ | |
Version 8.37.0 [v8-stable] 2018-08-07 | |
- build system: add --enable-default-tests ./configure option | |
This permits to control the "default tests" in testbench runs. These | |
are those tests that do not need a special configure option. There are | |
some situations where we really want to turn them of so that we can | |
run tests only for a specific component (e.g. ElasticSearch). | |
This commit also removes the --enable-testbench[12] configure switches, | |
which were introduced just to work-around travis runtime restrictions. | |
With the new CI setup and new options we could reduce the Travis runtime | |
dramatically and so we do not need them any longer. | |
- overall adaptation to gcc 8 which emits new warnings | |
- fix some build warnings on 32bit systems, namely armhf architecture | |
- ommail change of behavior: "enable.body" default now "on" | |
This was always documented to be "on", but actually was "off". Usually, we | |
fix the doc, but after long discussion the agreement was that in this | |
specific case it was actually better to change the default. | |
see also: https://github.com/rsyslog/rsyslog/pull/2791 | |
- core/omfile: race in async writing mode | |
mutex was not properly locked at all times when the async writing buffer | |
was flushed | |
Thanks to Radovan Sroka for the patch. | |
- core: provide a somewhat better default action name | |
We now include the module name (e.g. "omelasticsearch" or "builtin:omfile") | |
as part of the name. This is still not perfect, but hopefully a bit | |
easier to grasp. | |
see also https://github.com/rsyslog/rsyslog/issues/342 | |
- new global() parameter "abortOnUncleanConfig" | |
This provides a new-style alternative to $AbortOnUncleanConfig. | |
closes https://github.com/rsyslog/rsyslog/issues/2744 | |
- tcpflood no longer links with -lgrcypt | |
as this is no longer necessary for GnuTLS | |
Thanks to Michael Biebl for the patch. | |
- imjournal: add journal-specific impstats counters | |
these provide some additional insight into journal operations | |
Thanks to Abdul Waheed for the patch. | |
- imjournal: fixed startup on missing state file | |
When starting rsyslog with imjournal for first time it outputs | |
an error and plugin does not run because no state file exists yet. | |
Now it skips the loading and creates state file on first persist. | |
Thanks to Jiri Vymazal for the patch. | |
- imjournal: fetching cursor on readJournal() and simplified pollJournal() | |
Fetching journal cursor in persistJournal could cause us to save | |
invalid cursor leading to duplicating messages further on, when new | |
WorkAroundJournalBug option is set we are saving it on each | |
readJournal() where we now that the state is good. | |
pollJournal() is now cleaner and faster, correctly handles INVALIDATE | |
status from journald and is able to continue polling after journal | |
flush. Also reduced POLL_TIMEOUT a bit as it caused rsyslog to exit | |
with error in corner cases for some ppc when left at full second. | |
re-factored imjournal CI tests with journal_print tool to have more | |
detailed error reporting. | |
Thanks to Jiri Vymazal for the patch. | |
- config: enhance backticks "echo" capability | |
This is now more along the lines of what bash does. We now support | |
multiple environment variable expansions as well as constant text | |
between them. | |
example: | |
env SOMEPATH is set to "/var/log/custompath" | |
config is: param=`echo $SOMEPATH/myfile` | |
param than is expanded to "/var/log/custompath/myfile" | |
among others, this is also needed inside the testbench to properly | |
support "make distcheck". | |
Note: testbench tests follows via separate commit. There will be | |
no special test, as the testbench itself requires the functionality | |
at several places, so the coverage will be very good even without | |
a dedicated test. | |
- imrelp: add support for setting address to bind to (#894) | |
This adds a new optional `address` parameter to `imrelp` inputs in order | |
to specify an address to bind to. | |
Based on support added by rsyslog/librelp@96eb5be | |
Thanks to Simon Wachter for the patch. | |
- omrelp: permit all authmodes; updated tests | |
omrelp for some time limited authentication modes to those | |
that were known. While this was OK, it prevented the easy | |
introduction of new auth modes into librel. | |
This has now been changed; omrelp now checks the validity of | |
the authmode directly via librelp by doing some librelp calls | |
upon processing the configuration. | |
Also, some tests have been updated to check this feature and | |
also ensure that the new librelp mode "certvalid" works | |
(if it is available). | |
- regexp.c: reduce lock contention when using glibc. | |
When using glibc, we enable per-thread regex to avoid lock contention. | |
See: | |
- https://github.com/rsyslog/rsyslog/issues/2759 | |
- https://github.com/rsyslog/rsyslog/pull/2786 | |
- https://sourceware.org/bugzilla/show_bug.cgi?id=11159 | |
This should not affect BSD as they don't seem to take a lock in regexec. | |
NOTE: it is assumed that we can craft an even better solution than | |
this patch, but it improves the situation and we do not have time to | |
craft more. So we decided to merge. For details see | |
https://github.com/rsyslog/rsyslog/pull/2786 | |
- mmpstrucdata: better error message, support $! in var names | |
see also https://github.com/rsyslog/rsyslog/issues/1262#issuecomment-404773495 | |
- more explicit error msg with message modification mod on queue | |
Message modification modules do not work if used with a non-direct queue. | |
We now make this more explicit in the config parsing error message. | |
closes https://github.com/rsyslog/rsyslog/issues/1323 | |
- omrabbitmq: improve high-load performance | |
A different pthread mutex is created for each connection (action) | |
instead of a single one shared by all connections. This will | |
improve performance when using multiple concurrent connections | |
to a single (or multiple) RabbitMQ instance(s) (e.g. for load balancing) | |
Thanks to github user micoq for contributing the patch. | |
- imudp: replace select() calls by poll() | |
This improves reliability in extreme cases (more than 1024 fds open when | |
imudp begins to listen) and potentially improves performance a little. | |
- ommysql: support mysql unix domain socket: | |
via action(.. socket="/tmp/mysqld.sock" ..) | |
Thanks to JoungKyun Kim for contributing this. | |
- impstats: emit warning if log.syslog="off" and ruleset name given | |
With this config, "ruleset" is silently ignored, what probably is | |
not obvious to a user. | |
closes https://github.com/rsyslog/rsyslog/issues/2821 | |
- build system cleanup: remove no longer needed --enable-memcheck | |
This was used for a very old testing capability, no longer functional but | |
causes build to fail if enabled. Replaced by ASAN/valgrind. | |
Issue detected while testing some other CI settings. | |
- tools: Updated python based statslog analyzer sample scripts | |
- developer tools: make devcontainer tool more developer friendly | |
slight improvement for easy interactive use | |
- enable better testing via "make distcheck" | |
Also a couple of changes to testbench worth mentioning: | |
* use cp -f to ensure files can be overwritten in VBUILD | |
* fix issue of missing include test file in EXTRA_DIST | |
* new suppressions | |
* testbench: try to use local system dependency cache | |
avoid going to Internet repos if not absolutely necessary. For | |
development containers, they should be pre-populated with the | |
important dependencies. | |
* do not enable libfaketime if ASAN is selected | |
unfortunately, libfaketime does not work in that case | |
Note: for modules with non-standard dependencies (e.g. databases), | |
"make distcheck" only enables what on the original ./configure line | |
was enabled. This is done in order to ensure that "distcheck" adapts | |
to what is actually available on the system in question. Rsyslog's | |
own CI system installs the maximum set of possible dependencies and | |
so tries the maximum set "make distcheck" can support on a platform. | |
see also https://github.com/rsyslog/rsyslog/issues/174 | |
- add new global config parameter "inputs.timeout.shutdown" | |
- omusrmsg: do not fall back to max username length of 8 | |
This happens if utmp.h and friends are not available and stems back to | |
the original syslogd. Nowadas, 32 is more appropriate and now being used | |
in that (now very unlikely) case. The detection logic for UT_NAMESIZE has | |
also been streamlined. | |
closes https://github.com/rsyslog/rsyslog/issues/2834 | |
- bugfix build system: fix race in parallel builds | |
If libgcry.la is built later than lmcry_gcry.la, there is a failure: | |
[snip] | |
|../aarch64-wrs-linux-libtool --tag=CC --mode=link aarch64-wrs-linux-gcc | |
-o lmcry_gcry.la lmcry_gcry_la-lmcry_gcry.lo libgcry.la -lgcrypt | |
|aarch64-wrs-linux-libtool: error: cannot find the library 'libgcry.la' | |
or unhandled argument 'libgcry.la' | |
|Makefile:1049: recipe for target 'lmcry_gcry.la' failed | |
|make[2]: *** [lmcry_gcry.la] Error 1 | |
[snip] | |
The LIBADD of lmcry_gcry.la contains libgcry.la, we should also add libgcry.la | |
to lmcry_gcry.la's DEPENDENCIES. | |
Thanks to Hongxu Jia for the patch. | |
- bugfix imfile: memory leak upon shutdown (cosmetic) | |
When rsyslog shuts down and imfile is inside a change polling loop, | |
it does not properly free memory returned by glob(). This is a cosmetic | |
bug as the process terminates within the next few milliseconds. However, | |
it causes memory analyzer reports and thus makes CI fail. | |
- bugfix core msg: potential deadlock (and rsyslog hang) | |
can happen e.g. with headerless messages when app-name | |
property is used | |
closes https://github.com/rsyslog/rsyslog/issues/3135 | |
- bugfix core: do not abort startup on problems setting scheduling policy | |
rsyslog creates a default scheduling policy on startup. This code | |
invalidly used CHKiRet (our exception handler) to check pthreads | |
return codes, what this macro cannot do. This lead to hard to | |
diagnose startup problems in cases where there were problems | |
setting the scheduling defaults (e.g. when rsyslog is set to run | |
at idle priority). Even more so, this blocked startup altogether, | |
which is not the right thing to do. Actually, this can be considered | |
a regression from commit 7742b21. That commit was 8 years ago, so | |
in general this cannot be a big issues ;-) | |
The code now emits proper error messages (to stderr, as at this point | |
no other output is available as it is during the initial state of | |
rsyslog initialization) and continues the startup. | |
closes https://github.com/rsyslog/rsyslog/issues/2855 | |
- bugfix core: input shutdown timeout not properly applied | |
The timeout could be reduced by mutex wait time, which was not the | |
intended behavior and could lead the the input thread being | |
cancelled while it would have been perfectly legal to shut it down | |
cleanly. | |
Noticed during working on the CI system. May explain some testbench | |
instability and may have caused trouble with state files (not) | |
properly being written by inputs. | |
- bugfix config optimizer: error in constant folding | |
did not work properly if a string and a number were to be folded. | |
Detected by gcc 8. | |
- build: fix improper function casts | |
no real issue, but generated warnings under gcc 8 and thus | |
broke CI | |
- bugfix omlibdbi: fix potential small memory leak | |
detected by clang static analyzer | |
- bugfix ommysql: unsafe use of strncpy() | |
also now reports oversize names as user error vs. silent truncation | |
overly long names only could affect config load phase | |
- bugfix omhttpfs: fix insecure usage if strncmp() | |
consequences not evaluated as this is a contributed module. | |
Detected by gcc 8. | |
- bugfix mmgrok: cosmetic build issue - compiler warnings | |
caused build under gcc 7 to fail with warning | |
- bugfix mmkubernetes: stops working with non-kubernetes container names | |
When mmkubernetes encounters a record with a CONTAINER_NAME field, | |
but the value does not match the rulebase, mmkubernetes returns | |
an error, and mmkubernetes does not do any further processing | |
of any records. | |
The fix is to check the return value of ln_normalize to see if | |
it is a "hard" error or a "does not match" error. | |
This also adds a test for pod names with dots in them. | |
Thanks to Richard Megginson for the patch. | |
- bugfix mmkubernetes: potential NULL pointer access | |
If token file could not be opened, fclose() was passed a NULL pointer. | |
Thanks to github user jvymazal for finding and Richard Megginson | |
for fixing the issue. | |
- bugfix omsnmp: invalid traptype was not detected | |
this could leave config errors unreported and cause unexpected | |
behavior | |
- bugfix mmkubernetes: default rules use container_name_and_id | |
also include rulebase files in dist and fix rule so that dot inside | |
pod name is supported. | |
Thanks to Richard Megginson for fixing the issue. | |
- bugfix omelasticsearch: build regression | |
Commit 6d4635efbb13907bf651b1a6e5a545effe84d9d9 introduced some compile | |
problems, which were only detected on CentOS6, which unfortunately did | |
not compile omelasticsearch during CI runs | |
- bugfix ommongodb: do not force MongoDB to use "PLAIN" auth mechanism | |
... which also seems not to be handled by current MongoDB. | |
Remove ?authMechanism=PLAIN URI part to let the mongo library chooses the | |
default mechanism. One can force a specific authentication mechanism by | |
adding ?authMechanism=XXX into the uristr argument of the module | |
Thanks to Jérémie Jourdin for the fix. | |
closes https://github.com/rsyslog/rsyslog/issues/2753 | |
- build system: do not disable tests via --disable-liblogging-stdlog | |
This setting controlled both the actual rsyslog functionality as well | |
as some testbench tests, which use liblogging-stdlog to provide some | |
specific functionality. This meant those tests were not run since | |
changing the default. Now untangling the dependency. | |
- CI: | |
* most test refactored to use newer testbench plumbing | |
while no functional change, this permits further enhancements | |
* ElasticSearch startup timeout in tests increased to care for | |
slower test systems | |
* imjournal: fixed tests to actually test plugin functionality | |
Thanks to Jiri Vymazal for the patch. | |
* new test for gnutls priority string in librelp | |
Thanks to github user jvymazal for the patch | |
* testbench: relax hanging instance detection | |
This does not work reliably if multiple instances of rsyslog | |
builds run on a single machine. We need to improve, but this | |
commit makes conflict less likely and provides some diagnostic | |
info to help guide us towards a final solution. | |
* testbench: fix tests that look awfully wrong | |
These tests indicated they terminate rsyslog forcefully without | |
draining the queues, but then checked if they were drained (all | |
messages processed). That does not make sense, and we cannot | |
envision why this was written the first place. So we assume some | |
copy&paste problem was the root of that. | |
* testbench: refactor tests which used "nettester" tool | |
Some old tests are carried out via the nettester tool. This was | |
our initial shot at a testbench a couple of years ago. While it | |
worked back then, the testbench framework has been much enhanced. | |
These old tests are nowadays very hard to handle, as they miss | |
debug support etc. So it is time to refactor them to new style. | |
As a side-activity, the testbench plumbing has been enhanced to | |
support some operations commonly needed by these tests. Contrary | |
to pre-existing plumbing, these new operations are now crafted | |
using bash functions, which we consider superior to the current | |
method. So this is also the start of converting the older-style | |
functionality into bash functions. We just did this now because | |
it was required and we entangled it into the test refactoring | |
because it was really needed. Else we had to write old-style | |
operations and convert them in another commit, which would | |
have been a waste of time. | |
Special thanks to Pascal Withopf for the initial step of taking | |
old tests and putting config as well as test data together into | |
the refactored tests, on which Rainer Gerhards than could build | |
to create the new tests and update testbench plumbing. | |
* testbench: ensure uxsock test leaves no dangling listener instances | |
..in case the test aborts. We utilize the timeout utility for now | |
to prevent this. | |
* testbench: make port for imdiag dynamic | |
This is prep work to support parallel test runs | |
------------------------------------------------------------------------------ | |
Version 8.36.0 [v8-stable] 2018-06-26 | |
- build system change: | |
Liblogging-stdlog was introduced to provide a broader ability to send rsyslog | |
internal logs to different sources. However, most distros did not pick up | |
that capability and so instead we do a regular syslog() call. We assume that | |
the actual functionality is never used in practice, so we plan to retire it. | |
That makes building rsyslog from source easier. | |
The plan is to disable use of liblogging-stdlog by default during | |
configure. So users (and distros!) can still opt-in to have it enabled if | |
they desire. | |
A couple of releases later, we want to completely remove the functionality, | |
except if there has desire been shown in the meantime which justifies to keep | |
liblogging-stdlog. | |
This version disabled liblogging-stdlog by default. We now also | |
emit a warning message ("liblogging-stdlog will go away") so that users | |
know what is going on and my react. | |
closes https://github.com/rsyslog/rsyslog/issues/2705 | |
see also https://github.com/rsyslog/rsyslog/issues/2706 | |
- add openssl driver alongside GnuTLS one for TLS communication | |
The openssl driver is currently experimental. It will become the new preferred | |
driver as it permits us to provide much better end-user error message than | |
we could provide with GnuTLS. It is also less picky with certificate files | |
and provides specific error messages if there are certificate problems. | |
closes: https://github.com/rsyslog/rsyslog/issues/1390 | |
closes: https://github.com/rsyslog/rsyslog/issues/1840 | |
closes: https://github.com/rsyslog/rsyslog/issues/1352 | |
closes: https://github.com/rsyslog/rsyslog/issues/1702 | |
closes: https://github.com/rsyslog/rsyslog/issues/2547 | |
- GnuTLS TLS driver: support intermediate certificates | |
this is necessary for certificate chains | |
Thanks to Arne Nordmark for providing the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/2762 | |
- omelasticsearch: write op types; bulk rejection retries | |
* Add support for a 'create' write operation type in addition to | |
the default 'index'. Using create allows specifying a unique id | |
for each record, and allows duplicate document detection. | |
* Add support for checking each record returned in a bulk index | |
request response. Allow specifying a ruleset to send each failed | |
record to. Add a local variable `omes` which contains the | |
information in the error response, so that users can control how | |
to handle responses e.g. retry, or send to an error file. | |
* Add support for response stats - count successes, duplicates, and | |
different types of failures. | |
* Add testing for bulk index rejections. | |
Thanks to Richard Megginson for the patch. | |
- lookup tables: reload message now with "info" severity (was "error") | |
thanks to Adam Chalkley for the patch | |
- imptcp: add support for regex-based framing | |
for complex multi-line messages (XML in particular), the multiLine method | |
does not work well. We now have a capability to specify via a regex when | |
a frame starts (and the previous thus ends). | |
adds imptcp input parameter "framing.delimiter.regex" | |
- imjournal: add statistics counter | |
following statistics counter are now supported by imjournal | |
- submitted = total number of messages submitted for processing | |
closes https://github.com/rsyslog/rsyslog/issues/2549 | |
- config: permit 4-digit file creation modes | |
permit 4-digit file creation modes (actually 5 with the leading zero) so | |
that the setgid bit can also be set (and anything else on that position. | |
closes https://github.com/rsyslog/rsyslog/issues/1092 | |
- ommongodb: add possibility to ignore some insertion error code | |
new config parameter "allowed_error_codes", which will be ignored if | |
they happen. For example, 11000 DuplicateKey in case of collection | |
containing a unique field. | |
Thanks to Hugo Soszynski for contributing this work | |
- omprog: simplify 'plugin-with-feedback.py' example | |
Make the skeleton easier to understand by removing transaction support. | |
Also, transaction failures did not work as explained in the skeleton, | |
because of issue #2420. In the future, a 'plugin-with-transactions.py' | |
example can be added, ideally once the issue is solved. | |
Thanks to Joan Sala for contributing this. | |
- core: misaddressing when writing disk queue files | |
when writing disk queue files during shutdown, access to freed | |
memory can occur under these circumstances: | |
- action A is processing data, but could not complete it | |
most importantly, the current in-process batch needs not to | |
be totally completed. Most probable cause for this scenario | |
is a suspended action in retry mode. | |
- action A is called from a ruleset RA which | |
- does not have a queue assigned | |
- where RA is called from a ruleset RO which is bound | |
to the input from which the message originated | |
- RO must be defined before RA inside the expanded config | |
- Disk queues (or the disk part of a DA queue) must be utilized by A | |
When re-injecting the unprocessed messages from A into the disk queue, the | |
name of ruleset RO is accessed (for persisting to disk). However, RO is | |
already destructed at this point in time. | |
The patch changes the shutdown processing of rulesets, so that all | |
shutdown processing is done before any ruleset data is destructed. This | |
ensures that all data items which potentially need to be accessed | |
remain valid as long as some part may potentially try to access them. | |
This follows a the approach used in | |
https://github.com/rsyslog/rsyslog/pull/1857 | |
where obviously that part of the problem was not noticed. | |
see also https://github.com/rsyslog/rsyslog/issues/1122 | |
closes https://github.com/rsyslog/rsyslog/issues/2742 | |
- core: fix message loss on target unavailability during shutdown | |
Triggering condition: | |
- action queue in disk mode (or DA) | |
- batch is being processed by failed action in retry mode | |
- rsyslog is shut down without resuming action | |
In these cases messages may be lost by not properly writing them | |
back to the disk queue. | |
closes https://github.com/rsyslog/rsyslog/issues/2760 | |
- imrelp bugfix: error message "librelp too old" is always emitted ... | |
... even if librelp is current. The condition check was actually missing. | |
This commit adds it. | |
closes https://github.com/rsyslog/rsyslog/issues/2712 | |
- imrelp: segfault on startup when cert without priv key is configured | |
closes https://github.com/rsyslog/rsyslog/issues/2747 | |
- omrelp bugfix: segfault on first message sent when authmode was wrong | |
A segfault could occur if the authmode was configured to an invalid value. | |
This is now caught during config processing and an error is reported. | |
closes https://github.com/rsyslog/rsyslog/issues/2743 | |
- imfile bugfix: double-free on module shutdown | |
detected by code review, not seen in practice | |
- imfile/core bugfix: potential misaddressing in string copy routine | |
This can be exposed via imfile, as follows: | |
- use a regex to process multiline messages | |
- configure timeouts | |
- make sure imfile reads a partial message | |
- wait so that at least one timeout occurs | |
- add the message termination sequence | |
This leads to a misaddressing, which may have no obvious effects potentially | |
up to a segfault. | |
closes https://github.com/rsyslog/rsyslog/issues/2661 | |
- imfile bugfix: if freshStartTail is set some initial file lines missing | |
When the option is set and a new file is created after rsyslog startup, | |
freshStartTail is also applied to it. That is data written quickly to it | |
(before rsyslog can process it) will potentially be discarded. If so, | |
and how much, depends on the timing between rsyslog and the logging process. | |
This problem is most likely to be seen in polling mode, where a relatively | |
long time may be required for rsyslog to find the new file. | |
This is changed so that now freshStartTail only applies to files that | |
are already-existing during rsyslog's initial processing of the file | |
monitors. HOWEVER, depending on the number and location (network?) of | |
existing files, this initial startup processing may take some time as | |
well. If another process creates a new file at exactly the time of | |
startup processing and writes data to it, rsyslog might detect this | |
file and it's data as prexisting and may skip it. This race is inevitable. | |
So when freshStartTail is used, some risk of data loss exists. The same | |
holds true if between the last shutdown of rsyslog and its restart log | |
file content has been added. This is no rsyslog bug if it occurs. | |
As such, the rsyslog team advises against activating the freshStartTail | |
option. | |
closes https://github.com/rsyslog/rsyslog/issues/2464 | |
- core: fix undefined behavior (unsigned computation may lead to value < 0) | |
This was detected by LLVM UBSAN. On some platforms re-setting the rawmsg | |
inside the message object could lead to invalid computation due to the | |
fact the the computation was carried out as unsigned and only then | |
converted to integer. | |
No known problem in practice. | |
- CI/QA: | |
- improved Elasticsearch tests so they can now be run without system- | |
installed ES service; also enables us to specify specific ES versions | |
and should now make the tests executable inside a container | |
------------------------------------------------------------------------------ | |
Version 8.35.0 [v8-stable] 2018-05-15 | |
- imptcp: add ability to configure socket backlog | |
this can be useful under heavy load. | |
For a detailed discussion see https://github.com/rsyslog/rsyslog/pull/2561 | |
Thanks to Maxime Graff for implementing this. | |
- omfile: do not permit filename that only consists of whitespace | |
- fmhash: new hash function module | |
implements hash32() and hash64() functions | |
Thanks to Harshvardhan Shrivastava for implementing these | |
- some better error messages | |
- imklog: add ratelimiting capability | |
On Linux kernel logs are ratelimited only for messages using | |
printk_ratelimit(). Some logs do not use this facility, so | |
we ratelimit kernel ourselves. | |
Thanks to Berend De Schouwer for the patch. | |
- omkafka: added impstats counters for librdkafka returned statistics | |
Adds: | |
* statscallback counters | |
* librdkafka failure and error counters | |
* acked message counter | |
Thanks to Abdul Waheed for implementing this. | |
- imudp | |
* use rsyslog message rate-limiter instead of home-grown one | |
imudp introduced it's own (feature-limited) rate-limiting capability for | |
message on disallowed senders before we had central rate-limiters | |
inside rsyslog. Also, that code evolved from running on a single | |
thread to running on multiple threads, which introduced data races | |
and so made unreliable. | |
Now we removed the old rate-limiting capability and depend on the | |
system rate limiter for internal rsyslog messages. | |
closes https://github.com/rsyslog/rsyslog/issues/2467 | |
* add stats counter "disallowed" | |
counts the number of messages discarded due to being received from | |
disallowed senders | |
see also https://github.com/rsyslog/rsyslog/issues/2467 | |
- imrelp: add parameter "oversizeMode" | |
Permits to instruct librelp how to handle oversize messages. The new default | |
is to truncate messages. Previously, the connection was aborted, what often | |
lead to stuck messages at the sender side. Now, there are three options passed | |
down to librelp: | |
* abort - same behavior as previously, connection is aborted on error | |
* truncate - do not abort but instead truncate oversize message to | |
configured max size | |
* accept - accept all oversize messages (note: this can cause security issues, | |
see doc for details) | |
see also https://github.com/rsyslog/rsyslog/pull/1525#issuecomment-384179393 | |
see also https://github.com/rsyslog/rsyslog/issues/2190 | |
closes https://github.com/rsyslog/rsyslog/issues/2633 | |
closes https://github.com/rsyslog/rsyslog/issues/1741 | |
closes https://github.com/rsyslog/rsyslog/issues/1782 | |
closes https://github.com/rsyslog/rsyslog/issues/2496 | |
- core: consistent handling of oversize input messages | |
In the community we frequently discuss handling of oversize messages. | |
David Lang rightfully suggested to create a central capability inside | |
rsyslog core to handle them. | |
We need to make a distinction between input and output messages. Also, | |
input messages frequently need to have some size restrictions done at | |
a lower layer (e.g. protocol layer) for security reasons. Nevertheless, | |
we should have a central capability | |
* for cases where it need not be handled at a lower level | |
* as a safeguard when a module invalidly emits it (imfile is an example, | |
see https://github.com/rsyslog/rsyslog/pull/2632 for a try to fix it | |
on the module level - we will replace that with the new capability | |
described here). | |
The central capability works on message submission, and so cannot be | |
circumvented. It has these capabilities: | |
* oversize message handling modes: | |
- truncate message | |
- split message | |
this is of questionable use, but also often requested. In that mode, | |
the oversize message content is split into multiple messages. Usually, | |
this ends up with message segments where all but the first is lost | |
anyhow as the regular filter rules do not match the other fragments. | |
As it is requested, we still implemented it. | |
- accept message as is, even if oversize | |
This may be required for some cases. Most importantly, it makes | |
quite some sense when writing messages to file, where oversize | |
does not matter (accept from a DoS PoV). | |
* report message to a special "oversize message log file" (not via the | |
regular engine, as that would obviously cause another oversize message) | |
This commit, as the title says, handles oversize INPUT messages. | |
see also https://github.com/rsyslog/rsyslog/issues/2190 | |
closes https://github.com/rsyslog/rsyslog/issues/2681 | |
closes https://github.com/rsyslog/rsyslog/issues/498 | |
Note: this commit adds global parameters: | |
* "oversizemsg.errorfile", | |
is used to specify the location of the oversize message log file. | |
* "oversizemsg.report", | |
is used to control if an error shall be reported when an oversize | |
message is seen. The default it "on". | |
* add global parameter "oversizemsg.input.mode" | |
is used to specify the mode with which oversized messages will | |
be handled. | |
- omfwd: add support for bind-to-address for UDP | |
To allow the same source address to be used regardless of the egress | |
interface taken, an option is added for an address to bind the datagram | |
socket to. Similarly to imudp, it is necessary to add an ipfreebind | |
option which is set by default, so as to avoid an excess of errors at | |
startup before the network interface has come up. This enhancement | |
allows a usecase on networking devices, by which a source interface | |
that is typically a loopback is specified, on which an address to bind | |
to is configured. This is so that the same source address is used for | |
all packets from rsyslog. | |
Thanks to Mike Manning for the patch. | |
- template systemd service file proposes higher permitted file handle limit | |
Especially on busy systems the default are too low. Please keep in mind | |
that on a very busy system even the now-proposed setting may be too low. | |
Thanks to github user jvymazal for the patch. | |
- imuxsock: replace select() call by poll() | |
While extremely unlikely, imuxsock could abort if a file descriptor | |
> 1024 was received during the startup phase (never occured in | |
practice, but theoretically could if imfile monitored a large number | |
of files and were loaded before imuxsock - and maybe other | |
strange cases). | |
see also https://github.com/rsyslog/rsyslog/issues/2615 | |
- nsdsel_ptcp: replace select() by poll() | |
This removes us of problems with fds > 1024. The performance will | |
probably also increase in most cases. | |
Note this is not a replacement for the epoll drivers, but a general | |
stability improvement when epoll() is not available for some reason. | |
see also https://github.com/rsyslog/rsyslog/issues/2615 | |
closes https://github.com/rsyslog/rsyslog/issues/1728 | |
closes https://github.com/rsyslog/rsyslog/issues/1459 | |
- omprog: refactor tests, fix child closing issues | |
Refactor omprog tests. Fix sync issues in these tests by | |
using the feedback mode (confirmMessages=on) to synchronize | |
the test with the external program. Closes #2403 (I hope) | |
Fix omprog not properly closing child process when | |
signalOnClose=on. Needed for the new tests. Closes #2599 | |
Fix omprog not waiting for the child process to terminate | |
when signalOnClose=off. Needed for the new tests. Closes #2600 | |
Close all fds before executing the child even when valgrind | |
is enabled (--enable-valgrind). Needed for the new tests. | |
Fix memory leak when the xxxTransactionMark parameters were | |
used. | |
Thanks to Joan Sala for the patch. | |
- core: config optimizer did not handle call_indirect | |
This also caused the emission of an "internal error" error message | |
closes https://github.com/rsyslog/rsyslog/issues/2665 | |
- debug support: add capability to print testbench-specific timeout reports | |
done by setting RSYSLOG_DEBUG_TIMEOUTS_TO_STDERR to "on" | |
this is by default activated inside the testbench | |
- mmgrok: fix potential segfault | |
The modules used strtok(), which is not thread-safe. So it will potentially | |
segfault when multiple instances are spawned (what e.g. happens on busy | |
systems). | |
This patch replaces strtok() with its thread-safe counterpart | |
strtok_r(). | |
see also https://github.com/rsyslog/rsyslog/issues/1359 | |
- imrelp bugfix: maxDataSize could be set lower than maxMessageSize | |
maxDataSize specifies the length which will still be accepted | |
It previously could be set to any value, including values lower than the | |
configured rsyslog max message size, which makes no sense. Now this is | |
checked an error message is emitted if the size is set too low. | |
- build system bugfix: build broken if liblogging-stdlog installed in custom path | |
Thanks to Dirk Hörner for the patch. | |
- core bugfix: segfault on queue shutdown | |
if a ruleset queue is in direct mode, a segfault can occur during | |
rsyslog shutdown. The root cause is that a direct queue does not | |
have an associated worker thread pool, but the ruleset destructor | |
does not anticipate that and tries to destruct the worker thread | |
pool. It needs to do this itself, as otherwise we get a race | |
between rulesets on shutdown. | |
This was a regression from | |
https://github.com/rsyslog/rsyslog/commit/3fbd901b3e6300010 | |
closes https://github.com/rsyslog/rsyslog/issues/2480 | |
- imfile bugfix: statefiles contain invalid JSON | |
When imfile rewrites state files, it does not truncate previous | |
content. If the new content is smaller than the existing one, the | |
existing part will not be overwritten, resulting in invalid json. | |
That in turn can lead to some other failures. | |
closes https://github.com/rsyslog/rsyslog/issues/2662 | |
- omfile bugfix: segfault if empty filename was given | |
closes https://github.com/rsyslog/rsyslog/issues/2417 | |
- fix build issues when atomic operations are not present | |
for details, see https://github.com/rsyslog/rsyslog/pull/2604 | |
- lmsig_ksils12 bugfix: build and static analyzer issues | |
The module had a couple of problems building as well as some potential | |
errors detected by clang static analyzer. These have been fixed. | |
Thanks to Allan Park for the patch. | |
closes https://github.com/rsyslog/rsyslog/issues/2517 | |
- impstats bugfix: segfault if bound to non-existing ruleset | |
segfault happens during shutdown; up until unload of impstats, | |
rsyslog works correctly, except that no pstats are emitted. This | |
can be considered to be expected, because the error message | |
indicates the default ruleset is used instead. This is what | |
now actually happens. | |
closes https://github.com/rsyslog/rsyslog/issues/2645 | |
- mmjsonparse bugfix: invalid container name was not detected | |
in debug builds, this will trigger an assertion. In production | |
builds, an rsyslog internal error is logged, but rsyslog | |
continues to run. | |
closes https://github.com/rsyslog/rsyslog/issues/2584 | |
- mmkubernetes bugfixes: fix lnrules, add defaults, add test | |
- Fix lnrules for CONTAINER_NAME | |
- Add pkg check for lognorm >= 2.0.3 so we can set the macro | |
to enable ln_loadSamplesFromString | |
- Add some reasonable default values for parameters, such as | |
kubernetesurl https://kubernetes.default.svc.cluster.local:443 | |
- Clean up sample.conf configuration file | |
Thanks to Richard Megginson for the patch set. | |
- build system bugfix: --enable-atomic-operations did not work | |
closes https://github.com/rsyslog/rsyslog/pull/2604 | |
- bugfix: rsyslog aborts on startup when specific config errors are made | |
The following errors must be made in rsyslog.conf: | |
* no action present | |
* a call statement is used on an undefined ruleset | |
In this case, rsyslog emits an error message on the missing actions and | |
then segfaults. Depending on memory layout, it may also continue to run | |
but do nothing except accepting messages as no action is configured. | |
This patch make rsyslog properly terminate after the error message. It | |
is a change in behavior, but there really is no reason why a defunct | |
instance should be kept running. | |
closes https://github.com/rsyslog/rsyslog/issues/2399 | |
- build system: remove no longer needed --enable-libcurl configure switch | |
The --enable-libcurl switch was added to be able to disable libcurl | |
functionality inside the rsyslog core, see 46f4f43. As libcurl is no | |
longer used inside the core (due to introducing function modules), | |
--enable-libcurl needs to be removed. | |
closes https://github.com/rsyslog/rsyslog/issues/2628 | |
- QA/CI | |
* fixed races in some tests; root cause was that default enq timeout was too | |
low - we may also see in the future that other tests also need adjustment | |
(note that this is not a code problem but rather slow CI environments, | |
so increasing the timeout to get to a stable test state is the absolutely | |
correct thing to do) | |
* enabled some additional useful compiler warnings | |
* new test for diskqueue hitting configured disk space limit | |
* new tests for omfile | |
* added tests for mmkubernetes | |
* added tests for some script functions that were missing them | |
* made far more test compatible with FreeBSD, so that we now have fuller | |
coverage there | |
------------------------------------------------------------------------------ | |
Version 8.34.0 [v8-stable] 2018-04-03 | |
- mmkubernetes: new module | |
Thanks to Richard Megginson and Peter Portante for contributing the module. | |
- rsyslog script: introduce loadable function modules | |
rsyslog scripting can now also be extended via loadable modules - they | |
provides functions (just like loadable input, output, ... modules) | |
see also http://jan.gerhards.net/2018/03/loadable-rainerscript-functions.html | |
- imfile: large refactoring of complete module | |
This commit greatly refactors imfile internal workings. It changes the | |
handling of inotify, FEN, and polling modes. Mostly unchanged is the | |
processing of the way a file is read and state files are kept. | |
This is about a 50% rewrite of the module. | |
Polling, inotify, and FEN modes now use greatly unified code. Some | |
differences still exists and may be changed with further commits. The | |
internal handling of wildcards and file detection has been completely | |
re-written from scratch. For example, previously when multi-level | |
wildcards were used these were not reliably detected. The code also | |
now provides much of the same functionality in all modes, most importantly | |
wildcards are now also supported in polling mode. | |
The refactoring sets ground for further enhancements and smaller | |
refactorings. This commit provides the same feature set that imfile | |
had previously and all existing CI tests pass, as do some newly | |
created tests. | |
Some specific changes: | |
- bugfix: module parameter "sortfiles" ignored | |
This parameter only works in Solaris FEN mode, but is otherwise | |
ignored. Most importantly it is ignored under Linux. | |
fixes https://github.com/rsyslog/rsyslog/issues/2528 | |
- bugfix: imfile did not pick up all files when not present | |
at startup | |
fixes https://github.com/rsyslog/rsyslog/issues/2241 | |
fixes https://github.com/rsyslog/rsyslog/issues/2230 | |
fixes https://github.com/rsyslog/rsyslog/issues/2354 | |
fixes https://github.com/rsyslog/rsyslog/issues/1716 | |
- bugfix: directories only support "*" wildcard, no others | |
fixes https://github.com/rsyslog/rsyslog/issues/2303 | |
- bugfix: parameter "sortfiles" did only work in FEN mode | |
fixes https://github.com/rsyslog/rsyslog/issues/2528 | |
- provides the ability to dynamically add and remove files via | |
multi-level wildcards | |
see also https://github.com/rsyslog/rsyslog/issues/1280 | |
- the state file name currently has been changed to inode number | |
This will further be worked on in upcoming PRs | |
see also https://github.com/rsyslog/rsyslog/issues/2231 | |
- some enhancements were also done to CI tests, most importantly | |
they were made more compatible with BSD | |
Note that most of the mentioned bug fixes cannot be applied to older | |
versions, as they fix design issues which are solved by the refactoring. | |
Thus there are not separate commits for them. | |
There are probably also a number of different issues fixed, which have | |
not yet been full confirmed. Especially anyone having troubles with imfile | |
and wildcards will benefit from the refactoring. | |
closes https://github.com/rsyslog/rsyslog/issues/2359 | |
- omelasticsearch: add support for CA cert, client cert auth | |
This allows omelasticsearch to perform client cert based authentication | |
to Elasticsearch. | |
adds parameters: tls.cacert, tls.mycert, tls.myprivkey | |
Thanks to Richard Megginson for the patch. | |
- omfile-hardening: new contributed module for "omfile hardened operations" | |
This extends omfile with features considered useful for hardening. Comes | |
at the expense of performance loss and changed semantics. | |
Thanks to Mikko Kortelainen for contributing this work. | |
- stream/bugfix: memory leak on stream open if filename as already generated | |
this can happen if imfile reads a state file. On each open, memory for the | |
file name can be lost. | |
We detected this while working on imfile refactoring, so there is no related | |
bug report. No specific test has been crafted, as the refactored imfile | |
tests catch it (as soon as they are merged). | |
- bugfix/omkafka: did not build on platforms without atomic operations | |
Thanks to github user bruce87en for the patch | |
- bugfix/core/ratelimiting: SystemLogRateLimitBurst was limited to 65535 | |
rsyslog uses unsigned short for configuration setting SystemLogRateLimitBurst. | |
Being just 16 bits, unsigned short cannot hold values bigger than 65535. in a | |
practical setting rsyslog misbehaved with SystemLogRateLimitBurst being bigger | |
than 65535. | |
Thanks to github user KaleviKolttonen for the patch. | |
- bugfix imfile: memory leak in readMode 0 | |
closes https://github.com/rsyslog/rsyslog/issues/2421 | |
- bugfix omfile: some error messages had parameters in wrong order | |
which made the message look strange, but still readable | |
Thanks to Hans Rakers for the patch. | |
- bugfix omprog: file handle leak | |
There was a fd leak in the feedback feature added in v8.31.0 (github PR #1753). | |
The leak occurred when omprog was used with the confirmMessages=on setting | |
and no output setting. One fd was leaked every time the external program was | |
started. | |
Thanks to Joan Sala for the patch. | |
- bugfix imuxsock: data alignment problems | |
gcc did rightly complain that the cred and ts pointers would cause | |
alignment problems, so they were converted to structs and the necessary | |
data was memcpy()'ed to them. | |
the aux[] buffer was also potentially misaligned, so making a union | |
out of it and struct cmsghdr insured it was properly aligned. | |
The problems were especially visible on alpha and ia64 machines. | |
Thanks to Jason Duerstock for the fix. | |
- bugfix testbench: some test were accidentally not executed | |
Thanks to Kasumi Hanazuki for the patch. | |
- bugfix tcp subsystem: keepalive settings mixed up | |
TCPKeepAliveIntvl and TPCKeepAliveTime were switched. This is now correct and | |
thus causes a CHANGE OF BeHAVIOR of these settings. We applied this change only | |
after careful consideration of the effect. The contributor Alexandre Pierret | |
explained the situation as such: | |
"From my side, I work with thousands of servers centralizing logs to rsyslog | |
servers in tcp. All of them are running RHEL 6 and 7. The default rsyslog settings | |
in RHEL is TCPKeepAlive off. Since there are thousands of connections on the | |
rsyslog servers, I had to turn the TCPKeepAlive on to setup an aggressive policy | |
regarding ghost connections (following firewall tcp-timeout issue). Basically, | |
I set up: intvl=10 - probe=5 - time=2 If intvl and time are switched, it won't | |
break anything. It will just send 5 more empty tcp packets for 10 seconds (5 | |
probe x 2 seconds), which is painless (any professional grade 100€ router can | |
send more than 1.000.000 PPS). For 3000 servers, it means 3000 pps for 5 | |
seconds (3000 servers x 2 back-and-forth / 2 seconds). Let's take another worst | |
example: intvl=3600 - probe=5 - time=1. It means one keepalive every hour and | |
a 5 retry every 1s after a network issue. If the time and intvl values are | |
switched, it will generate 1 keepalive every seconds, It's a LOT more, but | |
after 5 probe or packet, it will stop. To summarize, I think it won't | |
break anything: | |
* Keepalive is off by default is many linux distribution | |
* When we enable it, it's to set up an aggressive policy. And setting up a | |
10-5-1, 60-5-2, 2-5-60 or 1-20-1 policy is almost the same. | |
Bonus: For people setting up their rsyslog from the documentation, it will | |
now work as expected." | |
This was convincing, and we actually think that the fast majority of users set | |
up keepalive based on the doc and did never verify it actually worked as | |
expected. So we think that in all those cases, rsyslog will finally work as | |
intended. So we consider it justified to "change the behavior" here. | |
full discussion in PR: https://github.com/rsyslog/rsyslog/pull/2367 | |
Thanks to Alexandre Pierret for analyzing the situation and providing the | |
patch. | |
- fix some cosmetic issues found by lgtm.com static code analyzer | |
e.g. header file guard not correctly set - if you really are interested in | |
details, check git log | |
- CI | |
* add build test without atomic operations - now catches missing mutex macros | |
* add lgtm.com static analyzer (automatically called via GitHub PR) | |
* improved stability of global-umask.sh test, which unnecessarily used | |
wildcards for test output file checking. | |
Thanks to Kasumi Hanazuki for the patch. | |
* added some test for omprog with transactional interface | |
Thanks to Joan Sala for the new tests. | |
* fixed some omjournal tests which did not properly check result | |
------------------------------------------------------------------------------ | |
Version 8.33.1 [v8-stable] 2018-03-06 | |
- 8.33.0 tarball release was actually pre-8.33.0 | |
... and so did not contain all features. This alone made a re-release | |
necessary, which is what now happens with 8.33.1. | |
Note: the git 8.33.0 label was correctly applied, "just" the tarball | |
was wrong. | |
- devcontainer: use some more sensible defaults | |
and add ability to specify generic docker run options | |
this makes integration into CI (and other scripting) easier | |
- fix problems with make dist | |
make dist did not package everything that was needed for CI, thus | |
resulting in make check failures if build from tarball. | |
Thanks to Thomas D. (whissi), and Michael Biebl for alerting us on the | |
problem, providing advise and some of the patches. We also added addt'l | |
patches ourselves. The problem occured as the CI check for tarball | |
completeness was more or less disabled a couple of weeks ago, which | |
unfortunately went unnoticed. We have also applied some more safeguards | |
to detect such problems in the future. | |
------------------------------------------------------------------------------ | |
Version 8.33.0 [v8-stable] 2018-02-20 | |
- auto-detect if running inside a container (as pid 1) | |
adjust defaults in this case to be more container-friendly | |
- config: add include() script object | |
This permits to include files (like legacy $IncludeConfig) via a | |
script object. Needless to say, the script object offers more | |
features: | |
- include files can now be | |
- required, with rsyslog aborting when not present | |
- required, with rsyslog emitting an error message but otherwise | |
continuing when not present | |
- optional, which means non-present include files will be | |
skipped without notice | |
This is controlled by the "mode" parameter. | |
- text can be included form e.g. an environment variable | |
--> ex: include(text=`echo $ENVVAR`) | |
This finally really obsoletes $IncludeConfig. | |
closes https://github.com/rsyslog/rsyslog/issues/2151 | |
- template: add option to generate json "container" | |
This enables easy JSON generation via template. | |
This commit also corrects an issue with the constant "jsonf" | |
format. That was recently added, and the implementation problem | |
only became visible when used inside a larger json object. No | |
officially released code is affected, thought - so it really | |
is just a side-note. | |
closes https://github.com/rsyslog/rsyslog/issues/2347 | |
- core/template: add format jsonf to constant template entries | |
closes https://github.com/rsyslog/rsyslog/issues/2348 | |
- config: add ability to disable config parameter ("config.enabled") | |
For auto-generated configs, it is useful to have the ability to disable some | |
config constructs even though they may be specified inside the config. This | |
can now be done via the ```config.enabled``` parameter, applicable to all | |
script objects. If set to ```on``` or not specified, the construct will be | |
used, if set to any other value, it will be ignored. This can be used | |
together with the backtick functionality to configure enable and disable | |
from either a file or environment variable. | |
closes https://github.com/rsyslog/rsyslog/issues/2431 | |
- script: permit to use environment variables during configuration | |
new constant string type "backticks", inspired by sh | |
(sample: `echo $VARNAME`). | |
- new global config parameter "shutdown.enable.ctlc" | |
permits to shutdown rsyslog via ctl-c; useful e.g. in containers | |
- config optimizer: detect totally empty "if" statements and optimize | |
them out | |
- template: constant entry can now also be formatted as json field | |
This enhancements permits to craft clean templates that generate JSON, | |
e.g. for ElasticSearch consumption (or any other REST API) | |
- omstdout: support for new-style configuration parameters added | |
- core: set TZ on startup if not already set | |
In theory, TZ should be set by the OS. Unfortunately, this seems | |
to be not the case any longer on many Linux distros. We now check | |
it and set it appropriate if not already given. | |
Thanks to github user JPvRiel for providing an excellent explanation | |
of the reasoning for this and how to work around it. | |
closes https://github.com/rsyslog/rsyslog/issues/2054 | |
- imjournal bugfix: file handle leak during journal rotation | |
Thanks to Peter Portante for the patch | |
see also: https://github.com/rsyslog/rsyslog/pull/2437 | |
- lmsig_ksils12 bugfix: dirOwner and dirGroup config was not respected | |
- script bugfix: replace() function worked incorrectly in some cases | |
If the end of the message was similar to the replacement string, parts | |
of the string could (not always) be missing. | |
Thanks to Yaroslav Bo for the patch. | |
- build system bugfix: --disable-libcurl did not work | |
Thanks to Dan Molik, Thomas D. (whissi), and Michael Biebl for the patches. | |
- fixed build issues on Alpine Linux | |
- core bugfix: misaddressing in external command parser | |
This parser is used whenever a module (e.g. omprog) needs to process | |
command lines. If command parameters were given, memory misaddressing | |
occured. This could lead to a segfault. | |
This is a regression in 8.32.0. | |
closes https://github.com/rsyslog/rsyslog/issues/2408 | |
- core bugfix: small memory leak in external command parser | |
This parser is used whenever a module (e.g. omprog) needs to process | |
command lines. On each action definitions for actions that use the | |
parser a small amount of memory was leaked. It is an uncritical leak | |
as it only occurs during config parsing. So it leaks a couple of | |
KiB during startup but does not grow during actual message processing. | |
This is a regression in 8.32.0. | |
- core bugfix: string not properly terminated when RFC5424 MSGID is used | |
This could lead to misaddressing when the jsonmesg property was used. | |
closes https://github.com/rsyslog/rsyslog/issues/2396 | |
- bugfix: strndup() compatibility layer func copies too much | |
The function did not obey the upper limit, effectively becoming | |
a strdup(). This was only noticed when the compatibility layer | |
was required, most importantly on Solaris 10. | |
- CI system | |
- we now use well-defined containers for parts of the CI runs | |
- now also build test under Alpine Linux | |
- test added for omprog feedback feature | |
------------------------------------------------------------------------------ | |
Version 8.32.0 [v8-stable] 2018-01-09 | |
- NEW BUILD REQUIREMENTs: | |
* libfastjson 0.99.8 is now required; older versions lead to bugs in rsyslog | |
* libczmq >= 3.0.2 is now required for omczmq | |
This was actually required for quite some while, but not properly checked | |
during configure run. If the lib was older, build failed. Now configure | |
detects this and emits the appropriate error message. | |
* libcurl is now needed for rsyslog core | |
due to new script function http_request(). This can be turned off by the | |
new configure option --disable-libcurl. If so, http_request() is not | |
available. | |
- rsyslogd: add capability to specify that no pid file shall be written | |
Command line option -iNONE provides this capability. This utilizes the | |
pre-existing -i option, but uses the special name "NONE" to turn of the | |
pid file check feature. Turning off is useful for systems where this no | |
longer is needed (e.g. systemd based). | |
closes https://github.com/rsyslog/rsyslog/issues/2143 | |
- ompgsql: considerable enhancements | |
The PostgreSQL output module was woefully out-of-date the following | |
list is changes made to update the module to current Rsyslog standards. | |
* allow for v6 configuration syntax | |
* configurable ports | |
* support transactional interface | |
* push db connection into workers (libpq is threadsafe) | |
* enable module testing on travis | |
* ensure configuration syntax backwards compatibility | |
* formatting around postgres core templating | |
* use new test conventions | |
* add new configuration syntax test | |
* add valgrind tests for new and old syntax | |
* add threading tests | |
* add action queue long running tests | |
* add action queue valgrind test | |
Thanks to Dan Molik for contributing this great enhancement! | |
- build system: removed --enable-rtinst configure option | |
This was a historic debugging option which has been superseded by | |
newer tools like valgrind, helgrind, ASAN, TSAN, etc... | |
- pmrfc3164: support for headerless messages | |
pmrfc3164 now detects if the first non-SP, non-HT character is either | |
'{' or '[' and if so assume that no header (TAG, HOSTNAME, DATE) is | |
given. If so, it uses defaults for these values. The full message is | |
used as the MSG part in that case. Note that an initial PRI may still | |
be specified. | |
This follows the trend to send JSON messages via syslog without any | |
header. We use default header values in this case. | |
This right now is a bit experimental; we may roll it back if | |
problems show up in practice. | |
closes https://github.com/rsyslog/rsyslog/issues/2030 | |
- omhiredis: add option to use RPUSH instead of LPUSH | |
see also https://github.com/rsyslog/rsyslog/issues/1625 | |
- mmexternal improvements | |
* better error reporting if to-be-executed script cannot be executed | |
* some general cleanup | |
* better redirection of stdin/out/err for the executed script | |
* bugfix: argv[0] of the script was missing if command line parameters | |
were not specified | |
- omprog: refactored, code shared with mmexternal moved to common object | |
- logctl tool: refactor to support newer rsyslog standards | |
* Made the logctl usertool ISO C90 compliant. | |
* Made the logctl usertool use a homogeneous coding style. | |
Thanks to Hugo Soszynski for contributing this work (as well as | |
suggesting some workarounds related to libmongoc/libbson). | |
- imfile: added support for Solaris File Event notification (FEN) | |
also improves performance under Solaris, with FEN imfile provides | |
features equivalent to inotify mode | |
- core/action: new parameter "action.errorfile" | |
permits to write failed messages to an "error file" so that they | |
can be reviewed and potentially be reprocessed | |
- imfile: added new module parameter "sortFiles" | |
This permits to process newly created files in sorted order. | |
- imuxsock: improved status reporting: socket name received from systemd | |
Providing an indication of what we got from systemd facilitates problem | |
analysis. | |
- build system: added new testbench configure switches | |
now --enable-testbench1 and --enable-testbench2 exists which permit | |
to enable/disable parts of the testbench. By default, both are on | |
when --enable-testbench is given. For full testbench coverage, both | |
options must be given. These options are necessary because under | |
Travis we hit the max runtime for tests and needed to split tests | |
over multiple incarnations. | |
- mmpstrucdata: new parameter "sd_name.lowercase" | |
to permit preserving case for structured data identifiers | |
Special thanks to github user alanrobson for the initial commit that | |
preserves case (on which we based other work). | |
- omfile: add module-global option "dynafile.donotsuspend" | |
this permits to enable SUSPENDing dynafile actions. Traditionally, | |
SUSPEND mode was never entered for dynafiles as it would have blocked | |
overall processing flow. Default is not to suspend (and thus block). | |
closes https://github.com/rsyslog/rsyslog/issues/2236 | |
- testbench: add a capability to turn off libfaketime tests via configure | |
Unfortunately, libfaketime becomes more and more problematic in newer | |
versions and causes aborts on some platforms. This provides the ability | |
to turn it off via --disable-libfaketime. | |
In the longer term, we should consider writing our own replacement. | |
- testbench: name valgrind tests consistently | |
all valgrind tests now end in -vg.sh | |
- RainerScript: add function parse_json() | |
- RainerScript: add function substring() | |
- RainerScript: add function http_request() | |
- RainerScript: add function previous_is_suspended() | |
This function returns a boolean indicating if the previous action is | |
suspended (0 - no, 1 - yes). This is useful if arbitrary processing | |
(other than calling actions) should be made depending on that state. | |
A prime example for this is probably calling a ruleset. | |
closes https://github.com/rsyslog/rsyslog/issues/1939 | |
- Patches from BSD projects have been imported | |
... as far as they still apply. Some patches that patched BSD-specific | |
code were broadened to be more generic. | |
- script bugfix: invalid function names were silently ignored | |
no error message was produced | |
thanks to Matt Ezell for providing the patch. | |
- rainerscript: add int2hex() function | |
- rainerscript: add is_time() function | |
Thanks to Stephen Workman for implementing this. | |
- RainerScript: add function script_error() and error-reporting support | |
This permits script functions that could fail to report errors back, so | |
that the user can handle them if desired. We use an errno-style of | |
interface. That means script_error() needs to be called after functions | |
that supports it. It will return either 0 (all OK) or something else | |
(an error condition). | |
The commit also modifies the parse_time() function to use the new | |
interface. First of all, this is useful for users, and secondly we | |
needed a capability to craft a testbench test. | |
closes https://github.com/rsyslog/rsyslog/issues/1978 | |
- testbench: fixed build problem of testbench tools under Alpine Linux | |
- added --enable-libsystemd configure option to enforce use of libsystemd | |
so we can fail the build on platforms where this is required | |
- core/glbl: remove long-unused option $optimizeforuniprocessor | |
This was still available, but had no effect (for ~10 years or so). The | |
plumbing is now removed. If someone tries to use the option, an | |
error message is generated. | |
closes https://github.com/rsyslog/rsyslog/issues/2280 | |
- core/queue: emit better status messages at rsyslog shutdown | |
this helps to diagnose issue - unfortunately we need more work to ensure | |
that the messages always make it to the user. This is a start and | |
hopefully useful at least for the testbench, possibly more. | |
- fixed a couple of build issues with gcc-7 (in less frequently used modules) | |
- fixed a couple of build issues on the arm platform (actually raspbian) | |
- impstats: fix invalid counter definitions for getrusage() reporting | |
some of the counters were defined as int (32 bit) vs. intctr_t (64 bit). | |
On some platforms "long" seems to be 64bit, and getrusage() provides | |
what we store as int via long. So this caused truncation and/or overflow. | |
This had undefined effects. Most often, everything worked fine | |
for values smaller than 2^31 but sometimes we got negative values. | |
closes https://github.com/rsyslog/rsyslog/issues/1517 | |
- imudp bugfix: potential segfault in ratelimiting | |
The rate-limiter inside imudp was not set to be thread safe, but was | |
used across multiple threads. This worked in default configuration, | |
but failed when RepeatedMsgReduction was set to "on". | |
Note that it in general is a bug to use a rate-limiter in | |
non-threadsafe mode across multiple threads. This also causes invalid | |
rate limiting counts in the default case. | |
closes https://github.com/rsyslog/rsyslog/issues/441 | |
fixes https://github.com/rsyslog/rsyslog/issues/2132 | |
- imptcp bugfix: access to free'ed memory | |
When notifyconnectionclose was on, a string buffer was accessed immediately | |
after it was freed (as part of the connection close operation). | |
Detected by LLVM ASAN. | |
- mmanon bugfix: fix wrong ipv6 embedded recognition | |
mmanon recognized IPv6 with embedded IPv4 that have too few (16 bit) fields. | |
example: 13:abd:45:0.0.0.0 | |
closes https://github.com/rsyslog/rsyslog/issues/2357 | |
- imfile bugfix: not detecting files in directory when wildcards are used. | |
When directories and files are created at the same time, | |
imfile may missed subdirs or file if the machine is on high load. | |
The handling has been enhanced to scan newly created directories ALWAYS for | |
matching files. | |
fixes https://github.com/rsyslog/rsyslog/issues/2271 | |
However there still exist problems with multilevel directory configurations. | |
Details are discussed in https://github.com/rsyslog/rsyslog/issues/2354 | |
Fixes for the remaining issues are expected for 8.33.0. | |
- script bugfix: improper string-to-number conversion for negative numbers | |
- core/action bugfix: 100% CPU utilization on suspension of output module | |
Triggering condition: | |
* output module using the legacy transaction interface | |
(e.g. omelasticsearch, omlibdbi) | |
* output module needs to suspend itself | |
In these cases, rsyslog enters a busy loop trying to resolve the | |
suspend condition. The bug is rooted in rsyslog core action code. | |
This patch fixes it by inserting a 1-second sleep during calls | |
to the resume handler. | |
Note: we cannot sleep exactly as long as tryResume needs. This | |
would require larger refactoring, which probably is not worth for | |
the legacy interface. The current solution is almost as good, as | |
the one second sleep has very little overhead on a real system. | |
Thus we have chosen that approach. | |
This patch now also ensures that failed messages are properly | |
handled and do not cause eternal hang. | |
closes https://github.com/rsyslog/rsyslog/issues/2113 | |
- core/variables bugfix: bare $! cannot be used in set statement | |
fixes https://github.com/rsyslog/rsyslog/issues/326 | |
- core bugfix: auto commit of actions improperly handled | |
The internal state variable bHadAutoCommit was handled in thread-unsafe way | |
and most probably caused (hard to see) issues when action instances were | |
run on multiple worker threads. It looks like the state variable was | |
forgotten to move over to worker state when action workers were introduced. | |
closes https://github.com/rsyslog/rsyslog/issues/2046 | |
- core bugfix: filename length limitation of 199 bytes | |
file names (including path names) longer than 199 bytes could not be | |
handled at many places. This has now been uplifted to 4KiB after careful | |
review for the largest size supported among all relevant platforms. | |
- core bugfix: undefined behavior due to integer overflow | |
when searching strings, we may have an (unsigned) integer overflow | |
which can lead to misaddressing. | |
Detected by clang ubsan. | |
- core bugfix: race on LocalHostIP property during startup | |
The way the default was set could lead to a race if e.g. two internal | |
messages were emitted at startup when the property was not yet set. This | |
has been seen to happen in practice. It had no bad effect except a very | |
small stationary memory leak, but made using thread analyzers unreliable | |
(as it was rightfully detected as a problem). | |
closes https://github.com/rsyslog/rsyslog/issues/2012 | |
- bugfix: potential segfault on startup | |
timezone info table was "sorted" even though it may be NULL. There is | |
no practical case known where this lead to an actual abort, but in | |
theory it would be possible. If so, it would happen immediately on | |
startup. | |
Detected by clang ubsan. | |
- omhiredis bugfix: rsyslog segfault on startup if no template is specified | |
- omprog bugfix: argv[0] not set when using binary without arguments | |
When using the omprog plugin with a binary without arguments, argv[0] (the | |
name of the binary) is not set, causing binaries that depend on this value | |
being set to crash or misbehave. | |
This patch also mildly refactors omprog argv[] creations, albeit some more | |
refactoring would be useful. | |
closes https://github.com/rsyslog/rsyslog/issues/1858 | |
- core: refactoring of rsyslog's cstr "class" | |
Function cstrGetSzStrNoNULL shall modified the string buffer on each call, | |
albeit it is considered a "read-only" function. It currently adds a '\0' | |
at the end. This is bad, especially when multiple threads access the same | |
string. As always the same data is written, it was not causing real issues | |
(except unnecessary cache writes), but it polluted the thread debugger and | |
as such prevent more elaborate automated tests. | |
- parent directory creation function refactored | |
This should not cause any change of behavior, but is worth noting in case | |
we see a regression not caught by the CI system. | |
- mmsnmptrapd bugfix: potential misaddressing | |
This only occured in situations when the system was totally out of memory. | |
- imkafka: fix potential small resource leak | |
If rdkafka handle cannot fully populated, cleanup is added. Previously, we | |
could leak a handle if e.g. no brokers were available. Note that this was | |
a cosmetic leak, as proper processing is not possible in any case and the | |
leak is once per listener, so not growing. But we now also ensure that | |
proper error reporting and handling is done in any case. Previously, we | |
may have some misleading error messages if the defunct kafka handle was | |
used. | |
closes https://github.com/rsyslog/rsyslog/issues/2084 | |
- imkafka bugfix: do not emit error message on regular state | |
This was misunderstood as an error state and could spam the system | |
log considerably. Regression from 8.31.0. | |
- omkafka: expose operational status to user where useful | |
omkafka emits many useful operational status messages only to the debug | |
log. After careful review, we have exposed many of these as user error | |
and warning message (ex: librdkafka queue full, so user knows why we | |
suspend the plugin temporarily). This may have made the module too | |
chatty. If so, one can always filter out messages via configuration. And | |
if we really went overboard, we can undo those parts with the next | |
release. So it's better to give a bit more than less, as this definitely | |
eases troubleshooting for users. | |
closes https://github.com/rsyslog/rsyslog/pull/2334 | |
- omkafka bugfix: potential message duplication | |
If a message that already failed in librdkafka was resubmitted and that | |
resubmit also failed, it got duplicated. | |
- omkafka: fix multithreading | |
omkafka has several issue if multiple worker instances are used. This commit | |
actually make the module use a single worker thread at max. Reasoning: | |
Librdkafka creates background threads itself. So omkafka basically needs to move | |
memory buffers over to librdkafka, which then does the heavy hauling. As such, we | |
think that it is best to run max one wrkr instance of omkafka -- otherwise we just | |
get additional locking (contention) overhead without any real gain. As such, | |
we use a global mutex for doAction which ensures only one worker can be active | |
at any given time. That mutex is also used to guard utility functions (like | |
tryResume) which may also be accessed by multiple workers in parallel. | |
Note: shall this method be changed, the kafka connection/suspension handling needs | |
to be refactored. The current code assumes that all workers share state information | |
including librdkafka handles. | |
closes https://github.com/rsyslog/rsyslog/issues/2313 | |
- omkafka bugfix: potential misaddressing | |
The failed message list is improperly cleaned. This is a regression | |
from recent commit 4eae19e089b5a83da679fe29398c6b2c10003793, which | |
was introduced in 8.31.0. | |
This problem is more likely to happen under heavy load or bad | |
connectivity, when the local librdkafka queue overruns or message | |
delivery times out. | |
closes https://github.com/rsyslog/rsyslog/issues/2184 | |
closes https://github.com/rsyslog/rsyslog/issues/2067 | |
- omkafka bugfix: build fails with older versions of librdkafka | |
closes https://github.com/rsyslog/rsyslog/issues/2168 | |
- omgssapi bugfix: fix compiler warnings with gcc-7 | |
closes https://github.com/rsyslog/rsyslog/issues/2097 | |
- dnscache bugfix: entries were cached based on IP AND port number | |
That hash key which is used to find out already cached dns entry gets | |
incorrectly computed from the whole sockaddr_storage | |
(sockaddr_in/sockaddr_in6) structure including a sin_port (which doesn't | |
have a static value) instead of only an address, thus creating redundant | |
dns cache entries/consuming more space. This lead to potentially high memory | |
usage and ineffectiveness of the case. It could be considered a memory leak. | |
Thanks to Martin Osvald for the patch. | |
see also: https://github.com/rsyslog/rsyslog/pull/2160 | |
- omkafka bugfix: fixed memory leak | |
a memory leak occurred when librdkafka communicated error/status information | |
to omkafka. this seems to happen relatively frequently, so this leak | |
could be pretty problematic. | |
- mmdblookup bugfix: replace thread-unsafe strtok() by thread-safe counterpart | |
Many thanks to Will Storey (github user @horgh) for mentioning this and | |
his phantastic help in debugging this rsyslog problem! | |
- pmnormalize bugfix: remove unsave "strcat" implementation | |
- rainerscript bugfix: ltrim() and rtrim function misaddressing | |
This could lead to a segfault and was triggered by certain input data | |
schemes. For example, a ltrim(" a") could lead to the fault. | |
- imklog bugfix: local host IP was hardcoded to 127.0.0.1 | |
This is now taken from the global localHostIP setting, which is used | |
consistent across all modules. | |
Also, the removed (2012?) directive $klogLocalIPIF has been added | |
again but directly marked as removed. That way, an informative error | |
message is generated if someone tries to use it. | |
closes https://github.com/rsyslog/rsyslog/issues/2276 | |
- cleanup: remove obsolete pre-KSI GuardTime signature interface | |
this is no longer functional starting Jan 2018 as it reached end of life | |
closes https://github.com/rsyslog/rsyslog/issues/2128 | |
- cleanup: obsolete definition SOL_TCP replaced by newer IPPROTO_TCP | |
this should not have any effect at all except better portability, but is | |
worth mentioning in the ChangeLog nevertheless. | |
- lookup tables: fixed undefined behavior detected by UBSan | |
- CI testing | |
- ARM (via Raspberry PI) added to CI system | |
- Debian 9 added to CI system | |
- omgssapi and usertools components are now also tested in Travis | |
- test coverage on BSD has been enhanced | |
------------------------------------------------------------------------------ | |
Version 8.31.0 [v8-stable] 2017-11-28 | |
- NEW BUILD DEPENDENCY: ommongodb now requires libmongo-c | |
instead of deprecated libmongo-client. | |
- remove systemd embedded code, use libsystemd instead | |
Since the early days rsyslog used the original systemd embedded | |
interface code. This version now uses libsystemd where available. | |
If we do not find libsystemd, we assume the system does not use | |
systemd, which is a safe assumption nowadays. This ensures we use the | |
fresh interface code and also removes bloat from our project. | |
closes https://github.com/rsyslog/rsyslog/issues/1933 | |
- mmanon: add support for IPv6 addresses with embedded IPv4 address | |
While this format is uncommon, it may still be present in logs and as | |
such should be supported. It is configurable via individual settings, | |
though. Especially the number of bits to anonymize may be desired to | |
be different than in pure IPv6. | |
- ommongodb: big refactoring, more or less a feature-enhanced rewrite | |
New features are : | |
* Handle TLS connection to mongodb | |
* Handle MongoDB Replicasets | |
* Added the 'ssl_ca' and 'ssl_cert' directives to configure tls connection | |
* Added 'uristr' directive to configure the connection uri in the form | |
of 'mongodb://...' | |
Now uses the official mongo-c-driver library instead of the deprecated | |
mongo-client library | |
Special thanks to Hugo Soszynski and Jérémie Jourdin for there hard work | |
to make this a reality! | |
See also: https://github.com/rsyslog/rsyslog/pull/1917 | |
- rainerscript: add parse_time() function | |
Thanks to Stephen Workman for implementing this. | |
- omelasticsearch: add LF to every elastic response for error file | |
error file content was written without LF after each message, making | |
it hard to work with and monitor. | |
Thanks to Yaroslav Bo for the patch. | |
- omelasticsearch: add pipeline support | |
supports static and dynamic ElasticSearch pipeline parameter. | |
closes https://github.com/rsyslog/rsyslog/issues/1411 | |
Thanks to github users scibi and WaeCo for the implementation. | |
- lmsig_ksi_ls12: support asynchronous mode of libksi | |
- omprog: added error handling and transaction support for external plugins | |
This permits much better integration of external output modules. | |
Special thanks to Joan Sala for providing this work! | |
- imzmq3/omzmq3: marked as deprecated, modules will be remove in v8.41 | |
see also: https://github.com/rsyslog/rsyslog/issues/2100 | |
- imzmq3/omzmq3: fixed build issues with gcc-7 | |
- core: emit error message on abnormal input thread termination | |
This indicates a serious issue of which the user should be notified. | |
Was not done so far and caused grief when troubleshooting issues. | |
- core: refactored locking for json variable access | |
refactored the method so that it consistent in all functions and easier | |
to follow. Most importantly, now an as simple as possible locking paradigm | |
of lock ... unlock within the function. Hopefully easier to understand | |
both for humans and static code analyzers. | |
- core: refactored creation of UDP sockets | |
was very old, hard to follow code; streamlined that a bit | |
- core/dnscache: refactor locking | |
keep a simple lock ... unlock scheme within the function. That is | |
easier to follow for both humans as well as static analyzers. | |
Also removes Coverity scan CID 185419 | |
- rainerscript: use crypto-grade random number generator for random() function | |
We now use /dev/urandom if available. If not, we fall back to the weak PRNG. | |
- imkafka: improve error reporting and cleanup refactoring | |
- imkafka bugfix: segfault if "broker" parameter is not specified | |
Now emits warning message instead and uses hardcoded default | |
(which usually matches where the kafka broker can be found). | |
fixes https://github.com/rsyslog/rsyslog/issues/2085 | |
- omkafka: improve error reporting | |
- omkafka: slight speedup do to refactoring of LIST class | |
double-linked list was used for temporarily undeliverable message tracking | |
where singly-linked list was sufficient. Changed that. | |
- TCP syslog: support SNI when connecting as a client | |
This is done if a hostname is configured and turned off if an IP is used. | |
Thanks to Art O Cathain for the patch. | |
see also https://github.com/rsyslog/rsyslog/pull/1393 | |
- msg variable bugfix: potential segfault on variable access | |
A segfault is likely to happen if during processing a variable with | |
more than one path component is accessed (e.g. $!path!var) and NO | |
variables oft hat kind (local, message, global) are defined at all. | |
closes https://github.com/rsyslog/rsyslog/issues/1920 | |
- ratelimiting bugfix: data race in Linux-like ratelimiter | |
access to the Linux-like ratelimiter was not properly synchronized, and | |
the system rate-limiter was not using it in any case. | |
This could lead to the rate-limit settings not being properly | |
respected, but no "hard error". | |
- core/template bugfix: potential NULL pointer access at config load | |
could happen if the config was loaded and the parameters could not properly | |
be processed. If so, this occured only during the startup phase. | |
Detected by Coverity scan, CID 185318 | |
- core/json var subsystem bugfix: segfault in MsgSetPropsViaJSON | |
Invalid libfastjson API use lead to double-free. This was most importantly | |
triggered by mmexternal (but could potentially be triggered by other | |
uses as well) | |
closes https://github.com/rsyslog/rsyslog/issues/1822 | |
- core/wrkr threads bugfix: race condition | |
During e.g. shutdown it could happen that a worker thread was started | |
and immediately be requested to terminate. In such situations there was | |
a race the prevented proper initialization. This could lead to follow-on | |
problems. | |
We believe (but could not proof) that this also could lead to a hang of | |
the termination process. Thus we claim to close an issue tracker down | |
here below, but are not 100% certain it really closes it (we hope for | |
user feedback on this). In any case, the thread debugger showed invalid | |
operations and this has been fixed, so it is useful in any case. | |
closes https://github.com/rsyslog/rsyslog/issues/1959 | |
- core/wtp: potential hang during shutdown | |
when the wtp worker is cancelled in the final stage of shutting down | |
while the mutex is locked and there is one worker left, the system | |
will hang. The reason is that the cancelled thread could not free the | |
mutex that the other needs in order to shut down orderly. | |
Detected with clang thread sanitizer. | |
- omfwd bugfix: generate error message on connection failure | |
- imtcp bugfix: "streamdriver.mode" parameter could not be set to 0 | |
- imjournal bugfix: module was defunctional | |
The open function was broken by commit 92ac801, resulting in | |
no data being ever read from the journal. | |
patch bases on the idea of Radovan Sroka given here: | |
https://github.com/rsyslog/rsyslog/issues/1895#issuecomment-339017357 | |
but follows the current imjournal-paradigm of having the journal | |
handle inside a global variable. | |
see also https://github.com/rsyslog/rsyslog/issues/1895 | |
closes https://github.com/rsyslog/rsyslog/issues/1897 | |
- imjournal: refactor error handling, fix error messages | |
imjournal did not try to recover from errors and used the error state | |
returned by journal functions incorrectly, resulting in misleading | |
error messages. Fixed this and also increased the number of error | |
messages so that it now is easier to diagnose problems with this module. | |
Also a little bit of internal brush-up. | |
-mmdblookup bugfix: fix potential segfault due to threading issues | |
libmaxminddb seems to have issues when running under multiple threads. As | |
a first measure, we prevent mmdblookup now from running on more than one | |
thread concurrently. | |
see also: https://github.com/rsyslog/rsyslog/issues/1885#issuecomment-344882616 | |
- omelasticsearch bugfix: operational error messages not passed to user | |
This lead to sometimes hard to diagnose problem. Note that for obvious | |
reasons the amount of messages from omelasticsearch has increased; this | |
is not a sign of a problem in itself -- but it may reveal problems that | |
existed before and went unnoticed. Also note that too-verbose messages | |
can be filtered out via regular rsyslog methods (e.g. message discarding | |
based on content). | |
- omkafka bugfixes | |
* statistics counter maxoutqsize could not be reset | |
Thanks to github user flisky for the patch. | |
* potential hang condition | |
omkafka did not release a mutex under some error conditions (most | |
notably out of memory on specific alloc calls). This lead to a hang | |
during actively processing messages or during shutdown (at latest). | |
This could happen only if dynamic topics were configured. | |
Detected by Coverity Scan, CID 185781 (originally 185721, detected | |
as a different issue by Coverity after code refactoring done in regard | |
to 185721 -- then the problem became more obvious). | |
* file handle leak, which could occur when local buffering | |
of messages was needed | |
* invalid load of failedmsg file on startup if disabled | |
error "rsyslogd: omkafka: could not load failed messages from " | |
"file (null) error -2027 - failed messages will not be resend." | |
occurs because, despite `keepFailedMessages="off"` as a default, | |
omkafka still tries to check for and load a `(none)` file which | |
triggers an IO error of sorts according to the 2027 link above. | |
Obviously, it probably shouldn't try load the file if | |
`keepFailedMessages` is not enabled. | |
Thanks to github user JPvRiel for a very good error report and | |
analysis. | |
closes https://github.com/rsyslog/rsyslog/issues/1765 | |
* various config parameters did not work | |
These were not even recognized when used and lead to a config startup | |
error message: | |
~ closeTimeout | |
~ reopenOnHup | |
~ resubmitOnFailure | |
~ keepFailedMessages | |
~ failedMsgFile | |
closes https://github.com/rsyslog/rsyslog/issues/2052 | |
* considerable memory leak | |
Whenever a message could (temporarily) not be delivered to kafka, | |
a non-trivial amount of memory was leaked. This could sum up to | |
quite a big memory leak. | |
fixes https://github.com/rsyslog/rsyslog/issues/1991 | |
* some small memory leaks fixed | |
most of them cosmetic or a few bytes statically (not growing as | |
omkafka was used) -- thus we do not mention each one explicitly. | |
For more details, see git commit log or this pull request: | |
https://github.com/rsyslog/rsyslog/pull/2051 | |
- kafka bugfix: problem on invalid kafka configuration values | |
omkafka ended up in an endless loop and high cpu. | |
imkafka tried to subscribe to a not connected kafka server. | |
closes https://github.com/rsyslog/rsyslog/issues/1806 | |
- [io]mgssapi: fix build problems (regression from 8.30.0) | |
- [io]czmq: fix build problems on some platforms (namely gcc 7, clang 5) | |
- tcpsrv bugfix: potential hang during shutdown | |
- queue bugfix: potential hang during shutdown | |
- queue bugfix: NULL pointer dereference during config processing | |
If the queue parameters were incorrectly given, a NULL pointer dereference | |
could happen during config parsing. Once post that stage, no problem could | |
occur. | |
Detected by Coverity scan, CID 185339 | |
- imczmq bugfix: segfault | |
happened in a call to | |
371: zcert_destroy(&serverCert) called from rcvData(). | |
Thanks to ~achiketa Prachanda for the patch. | |
- imfile: some small performance enhancements | |
Thanks to Peter Portante for the patch | |
- omfile: handle file open error via SUSPEND mode | |
For a while, an open file error lead to suspension as the error was | |
not detected by the rule engine. This has changed with fixes | |
in 8.30.0. I asked users what they prefer (and expect) and | |
everyone meant it should still be handled via suspension. See | |
github tracker below for more details. | |
closes https://github.com/rsyslog/rsyslog/issues/1832 | |
- omfile bugfix: race during directory creation can lead to loop | |
There was a race where two threads were competing for directory creation | |
which could lead to none succeeding and a flood of error message like this | |
"error during config processing: omfile: creating parent directories for | |
file". This has been solved. | |
Thanks to Niko Kortström for the patch. | |
- imudp: improve error reporting | |
When udp listener cannot be created, an error message containing | |
the ip-address and port is put out. | |
closes https://github.com/rsyslog/rsyslog/issues/1899 | |
- omrelp bugfix: incorrect error handling | |
if librelp with TLS but without Authentication was included, librelp | |
did not emit the correct error message due to invalid error code | |
check. It also did not err-out but instead suspended itself. | |
Detected by Coverity scan, CID 185362 | |
- [io]mrelp bugfix: segfault on startup if configured cert not readable | |
When the certificate file specified in the omrelp/imrelp configuration | |
can't be accessed, e.g. because it doesn't exist or you don't have | |
permission to do so, a Segmentation Fault will appear when you start | |
Rsyslog. This commit fixes that problem. | |
closes https://github.com/rsyslog/rsyslog/issues/1869 | |
- mmanon fix: make build under gcc 7 | |
Thanks to William Dauchy for the patch | |
- mmpstrucdata bugfix: formatting error of ']' char | |
This was invalidly formatted as '"'. Thanks to github user | |
wu3396 for the error report including the patch idea. | |
closes https://github.com/rsyslog/rsyslog/issues/1826 | |
- mmexternalb bugfix: memory leak | |
- core/stats bugfix: memory leak if sender stats or tracking are enabled | |
- core bugfix: potential segfault during startup | |
A NULL pointer could be accessed if there was a problem with the | |
timezone parameters. Affects only startup, once started, no problem | |
existed. | |
Detected by Coverity scan; CID 185414 | |
- core bugfix: potential race in variable handling | |
Root of variable tree is accessed prior to locking access to it. | |
This introduces a race that may result in various kinds of | |
misaddressing. | |
Found while reviewing code, no bug report exists. | |
- core bugfix: potential segfault when shutting down rsyslog | |
when rulesets are nested a segfault can occur when shutting down | |
rsyslog. the reason is that rule sets are destructed in load order, | |
which means a "later" ruleset may still be active when an "earlier" | |
one was already destructed. In these cases, a "call" can invalidly | |
call into the earlier ruleset, which is destructed and so leads to | |
invalid memory access. If a segfault actually happens depends on the | |
OS, but it is highly probable. | |
The cure is to split the queue shutdown sequence. In a first step, | |
all worker threads are terminated and the queue set to enqOnly. | |
While some are terminated, it is still possible that the others | |
enqueue messages into the queue (which are then just placed into the | |
queue, not processed). After this happens, a call can no longer | |
be issued (as there are no more workers). So then we can destruct | |
the rulesets in any order. | |
closes https://github.com/rsyslog/rsyslog/issues/1122 | |
- core/action bugfix: potential misaddressing when processing hard errors | |
For batches that did fail in an output module, the rsyslog core | |
tries to find offending messages that generate hard (non-recoverable) | |
errors. During this process, the action templates for each message | |
are copied into a temporary buffer. That copy was invalidly sized, | |
actually copying only the first template string. As such, outputs | |
that requested more template strings AND had errors in batch submission | |
received uninitialized pointers. This could lead to all sorts of | |
problems. | |
see also https://github.com/rsyslog/rsyslog/issues/1885 | |
closes https://github.com/rsyslog/rsyslog/issues/1906 | |
- template object bugfix: NULL pointer access on invalid parameters | |
could happen only during startup | |
Detected by Coverity scan, CID 185376 | |
- omjournal bugfix: NULL pointer access on invalid parameters | |
could happen only during startup | |
- omelasticsearch bugfix: configured credentials not used during health check | |
Authentication credentials were not applied during health check, | |
permission to use unsigned CERTS was not applied to regular data post. | |
closes https://github.com/rsyslog/rsyslog/issues/1949 | |
- omelasticsearch bugfix: abort on unavailable ES server | |
Depending on the state of unavailability (libcurl return code), | |
omelasticsearch tries to process a NULL return message, what | |
leads to a segfault. | |
This fixes the problem and introduces better error handling and | |
better error messages. | |
see also https://github.com/rsyslog/rsyslog/issues/1885 | |
- omelasticsearch: fix memory leak and potential misaddressing | |
Commit 96b5fce introduced regressions, leading to potential misaddressing | |
and a very probable memory leak. This commit fixes the issues and | |
hardens the module to better detect some error cases in the | |
future. | |
It also adds valgrind-based testbench tests which ensure that no | |
pointer errors exist. If they had been in place, the regressions | |
would never have been undetected. | |
Note that commit 96b5fce was introduced in 8.23.0 on Nov, 15th 2016. | |
Many thanks to Yaroslav Bo for alerting me on the root problem and | |
providing a very good analysis and guidance. | |
see also https://github.com/rsyslog/rsyslog/issues/1906 | |
see also https://github.com/rsyslog/rsyslog/issues/1964 | |
closes https://github.com/rsyslog/rsyslog/issues/1962 | |
- omelasticsearch bugfix: output from libcurl to stdout | |
omelasticsearch made libcurl output messages to stdout. This | |
commit fixes that. It also automatically enables libcurl verbose | |
mode during debug runs - it needs to be seen if this is smart or | |
not (previously, code needed to be commented in). | |
closes https://github.com/rsyslog/rsyslog/issues/1909 | |
- iczmq bugfix: potential memory leak | |
- imptcp bugfix: potential misaddressing | |
When during a connection request the remote peer could not be | |
identified, imptcp could misaddress memory if it is configured | |
to emit messages on connection open. | |
Detected by clang 5.0 static analyzer. | |
- imptcp: potential buffer overflow | |
if the local hostname or IP is larger than NI_MAXHOST-1, an internal | |
buffer is overflowed. This is irrespective of any input data. | |
Detected by Coverity scan, CID 185390 | |
- core/nsd_gtls: fix potential uninitialized data access | |
could occur during certificate check | |
found by clang 5.0 static analyzer | |
- stats bugfix: potential program hang | |
due to missing unlock. This could only occur if pstats was set to | |
CEE-format logging (very uncommon) AND if the system runs out of | |
memory (in which case other things go pretty bad as well). | |
found by Coverity scan | |
- omfwd bugfix: memory leak if network namespaces are used | |
very seldom used feature, occurs only during error case | |
found by Coverity scan. | |
- core: potential misaddressing when accessing JSON properties | |
When a JSON property is accessed in template processing, memory may | |
have been misaddressed or a double-free may occur while obtaining the | |
property. | |
This was detected by a Coverity scan. | |
- gcry crypto provider bugfixes: potential misaddressing and memory leak | |
If the config parameters were invalid, a misaddressing could occur. If so, | |
this happens only during startup. | |
Also, a memory leak existed when the crypto provider errored out. This could | |
build up if it were used in frequently-changing dynafiles. This was | |
detected by Coverity scan, CID 185360. | |
- core/file stream object bugfix: memory leak | |
If a multiline read errored out, a line buffer could be leaked. | |
Detected by Coverity scan, CID 185328 | |
- imdiag bugfix: double mutex unlock when working with stats | |
Note: while this could potentially lead to a program hang, it affected | |
only testbench execution as imdiag is a testbench-only tool. | |
Detected by Coverity scan, CID 185348 and 185350 | |
- fixed several minor and cosmetic issues found by Coverity scan | |
including false positives. For details see "$ git log". All noteworthy | |
issues are separately mentioned in this ChangeLog. The ones not mentioned | |
are for example problems that can only occur during out of memory | |
conditions, under which it is extremely likely tha the rsyslog process | |
will be killed in any case | |
- testbench: | |
* added compile-check for [io]mgssapi, mmcount | |
* harden tests against hanging previous instances | |
* re-enable RELP tests on Solaris | |
* added basic test for imjournal | |
* added threading tests via valgrind's helgrind tool | |
* added valgrind test for kafka modules | |
* added capability to run elasticsearch tests with | |
a) different ElasticSearch versions | |
b) independently from OS-installed version | |
This also sets base to enable more elaborate ES tests | |