forked from rsyslog/rsyslog
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
14920 lines (14886 loc) · 838 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
----------------------------------------------------------------------------------------
Scheduled Release 8.2102.0 (aka 2021.02) 2021-02-??
- 2021-01-16: testbench: fix invalid sequence of kafka tests runs
kafka tests can not run well in parallel (mostly due to ressource
constraints on CI machines). Accidentally, this was not enforced for
one of the tests. That could lead to random failures and false positives.
- 2021-01-14: testbench: fix kafkacat issues
The kafkacat tool has an upper limit of how many messages it can send
at once. Going over that limit causes messages loss. The exact limit
seems to depend on the environment. This causes testbench false positives.
This commit fixes two related issues:
- errors during kafkacat run were not detected - this has been added
- we now have a "max messages at once" setting, after which kafkacat
is restarted for the next batch of messages. It currently is set
to 25,000 msgs per incarnation. All tests loop now to send the
required number of messages. This has been fixed at the testbench
framework level, so no need to adjust individual tests.
- 2021-01-14: testbench: fix year-dependendt clickhouse test
A test had the year value hardcoded and as such failed whenever the
year changed. This patch corrects that.
----------------------------------------------------------------------------------------
Scheduled Release 8.2012.0 (aka 2020.12) 2020-12-08
- 2020-12-07: testbench bugfix: some tests did not work in make distcheck
- certificate file missing in dist tarball
- some test cases did not properly specify path to cert file
Thanks to Michael Biebl for alerting us and providing part of
the fix.
closes https://github.com/rsyslog/rsyslog/issues/4446
- 2020-12-07: immark: rewrite with many improvements
- mark message text can now be specified
- support for rulesets
- support for using syslog API vs. regular internal interface
- support for output template system
- ability to specify is mark message flag can be set
- minor changes and improvements
- 2020-11-30: usability: re-phrase error message to help users better understand cause
see also https://github.com/rsyslog/rsyslog/issues/3910
- 2020-11-10: add new system property $now-unixtimestamp
Among others, this may be used as a monotonic counter
for doing load-balancing and other things.
Thanks to Nicholas Brown for suggesting this feature.
- 2020-11-04: omfwd: add new rate limit option
Adding new rate limit option to omfwd for rate limiting
syslog messages sent to the remote server
ratelimit.interval:
Specifies the rate-limiting interval in seconds.
Default value is 0, which turns off rate limiting.
ratelimit.burst
Specifies the rate-limiting burst in number of messages.
closes https://github.com/rsyslog/rsyslog/issues/4423
Thanks to Dinesh-Ramakrishnan for the patch.
- 2020-11-03: omfwd bug: param "StreamDriver.PermitExpiredCerts" is not "off" by default
The default behaviour of expired certificates of stream driver in TLS mode, should
have been that the see tcp transmission is closed due to expired certificates, and
error messages emited in rsyslog status. This was not the case. That in turn could
lead to permitting sessions which should not be permitted.
Thanks to Vincent Zhu for alerting us and providing a great problem analysis
closes: https://github.com/rsyslog/rsyslog/issues/4425
----------------------------------------------------------------------------------------
Scheduled Release 8.2010.0 (aka 2020.10) 2020-10-20
- 2020-10-13: gnutls TLS subsystem bugfix: handshake error handling
If the tls handshake does not immediatelly finish, gnutls_handShake is called in
doRetry handler again. However the error handling was not
complete in the doRetry handler. A failed gnutls_handShake call
did not abort the connection and properly caused unexpected
problems like in issues:
https://github.com/rsyslog/rsyslog/issues/4270
https://github.com/rsyslog/rsyslog/issues/4288
- 2020-10-13: core/msg bugfix: memory leak
There is a missing call to json_object_put(json) if the call to
jsonPathFindParent() failed. It's leaking memory. Depending on workload and config,
this leak can potentially grow large (albeit we did not see reports from practice).
Thanks to Julien Thomas for the patch.
- 2020-10-13: core/msg bugfix: segfault in jsonPathFindNext() when <root> not an object
The segfault gets happens when <bCreate> is 1 and when the <root>
container where to insert the <namebuf> key is not an object.
Here is simple reproducible test case:
// ensure we start fresh
// unnecessary if there was no previous set
unset $!;
set $! = "";
set $!event!created = 123;
Thanks to Julien Thomas for the patch.
- 2020-10-13: openssl TLS subsystem: improvments of error and status messages
Adding error logs at the ssl handshake failure scenarios.
Adding the header "nsd_ossl:" tag to these logs to identify
the origin module from which logs are generated.
Thanks to Anusha Pai G for the patch.
- 2020-10-06: add 'exists()' script function to check if variable exists
This implements a way to check if rsyslog variables (e.g. '$!path!var') is
currently set of not.
Sample: if exists($!somevar) then ...
closes https://github.com/rsyslog/rsyslog/issues/4385
- 2020-10-03: core bugfix: do not create empty JSON objects on non-existent key access
Performing a condition (eg: check for an empty string) on a subtree key that do not
exists (depth > 1 from the root container), creates an empty "parent" object.
Depending on your context, you may end up with (kind of...) annoying garbage when
producing object documents (for instance to index in ES).
Also fixes a hypothetical hang condition with an almost (?) unused plugin parameter
passing mode, for details see
https://github.com/rsyslog/rsyslog/issues/4436
closes https://github.com/rsyslog/rsyslog/issues/4430
Thanks to Julien Thomas for the patch.
- 2020-09-28: gnutls subsysem bugfix: potential hang on session closure
Some TLS servers don't reply to graceful shutdown requests "for
optimization". This results in rsyslog's omfwd+gtls client to wait
forever for a reply of the TLS server which never comes, due to shutting
down the connection with gnutls_bye(GNUTLS_SHUT_RDWR).
On systemd systems, commands such as "systemctl restart rsyslog" just
hang for 1m30 and rsyslogd gets killed upon timeout by systemd.
This is fixed by replacing the call to gnutls_bye(GNUTLS_SHUT_RDWR) by calls to
gnutls_bye(GNUTLS_SHUT_WR) which is sufficient and doesn't wait for a
server reply.
As an example, Kiwi Syslog server is known to cause this issue.
Thanks to Renaud Métrich for the patch.
- 2020-09-23: core/network bugfix: obey net.enableDNS=off when querying local hostname
Local hostname resolution used DNS queries even if the enableDNS was set to off, and
this could cause unexpected delays in the HUP signal handling if the DNS server was
not responsive.
Thanks to Samu Nuutamo for the fix.
- 2020-09-14: core bugfix: potential segfault on query of PROGRAMNAME property
A data race can happen on variable iLenProgram as it is not guarded
by the message mutex at time of query. This can lead to it being
non -1 while the buffer has not yet properly set up.
Thanks to Leo Fang for alerting us and a related
patch proposal.
replaces https://github.com/rsyslog/rsyslog/pull/4300
- 2020-09-14: imtcp bugfix: broken connection not necessariy detected
Due to an invalid return code check, broken TCP sessions could not
necessarily be detected "right in time". This can result is the loss
of one message.
closes https://github.com/rsyslog/rsyslog/issues/4227
Thanks to Leo Fang for the patch.
- 2020-09-14: new module: imhttp - http input
permits to receive log data via HTTP.
uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.
Thanks to Nelson Yen for contributing this module.
- 2020-09-11: mmdarwin bugfix: potential zero uuid when reusing existing one
- fix a use-after-free variable during darwin uuid message extraction
- improve debug/output by logging uuid parse errors
Thanks to github user frikilax for the patch.
- 2020-09-10: imdocker bugfix: build issue on some platforms
An invalid variable type was used, leading to compile errors at least on
all platform that use gcc 10 and above. Otherwise, however, it looks like the
issue caused no real harm.
- 2020-09-07: omudpspoof bugfix: make compatbile with Solaris build
Thanks to Dagobert Michelsen for the patch.
- 2020-09-03: testbench fix: python 3 incompatibility
- 2020-09-02: core bugfix: segfault if disk-queue file cannot be created
When using Disk Queue and a queue.filename that can not be created
by rsyslog, the service does not switch to another queue type as
supposed to and crashes at a later step.
closes: https://github.com/rsyslog/rsyslog/issues/4282
- 2020-08-26: cosmetic: fix dummy module name in debug output
When we have optional components (like imjournal) a dummy module
is used. It's sole purpose is to emit "this module is not available".
During init, the module emitted an invalid module name into the debug
log. This has now been replaced by the generic term "dummy".
Note: it is highly unlikely that someone will ever see that message
at all, as it is unlikely for the dummy modules to be build.
see also: https://github.com/rsyslog/rsyslog/commit/84a7e3d80b80106dcc86c273ed8cf78a6c11c722#r41782830
Thanks to Thomas D. (whissi) for the patch.
- 2020-08-26: config bugfix: intended warning emitted as error
When there are actions configured after a STOP, a warning should be
emitted. In fact, an error message is generated. This prevents the
construct, which may have some legit uses in exotic settings. It
may also break older configs, but as the message is an error
for so long now, this should be no longer of concern.
----------------------------------------------------------------------------------------
Scheduled Release 8.2008.0 (aka 2020.08) 2020-08-25
- 2020-08-25: imdocker bugfix: error reporting not always correct
A wrong function to obtain the error code was used. This
could lead to invalid error messages.
Thanks to Steve Grubb for the bug report and fix proposal.
closes https://github.com/rsyslog/rsyslog/issues/4381
- 2020-08-25: imptcp: add max sessions config parameter
The max is per-instance, not global across all instances.
There is also a bugfix where if epoll failed I think we could leave a
session linked in the list of sessions, this code unlinks it.
Thank to Alfred Perlstein for the patch.
- 2020-08-24: omelasticsearch bugfix: reply buffer reset after health check
The issue happens when more than one server is defined on the
action. On that condition a health check is made through
checkConn() before sending the POST. The replyLen should be
set back to 0 after the health check, otherwise the response
data received from the POST gets appended to the end of the
last health check.
Thanks to Julien Thomas for the patch.
- 2020-08-14: omfile: do no longer limit dynafile cache size in legacy format
When using obsolete legacy config format, omfile had a hard limit of
1,000 dynafile cache entries. This does not play well with very
large installation. This limit is now removed and converted into
a warning if cache size > 25,000 is specified.
Note: the problem can easily be worked-around by using modern
config format (RainerScript).
closes: https://github.com/rsyslog/rsyslog/issues/4241
- 2020-08-13: imudp: fix very small, static memory leak
When ruleset support was used, the ruleset name was not freed upon rsyslog
termination. While this has no consequences for regular runs, it generates
leak errors under memory debuggers and as such makes debugging harder than
necessary.
Thanks to github user frikilax for the patch.
- 2020-08-13: omelasticsearch: add parameter skipPipelineIfEmpty
When POST'ing a document, Elasticsearch does not allow an empty pipeline
parameter value. This patch introduces boolean option skipPipelineIfEmpty
to the omelasticsearch action. When set to true, the pipeline parameter
won't be posted. Default is false so we do not modify current behavior.
Thanks to Julien Thomas for the patch.
- 2020-08-12: systemd service file removed from project
This was done as distros nowadays have very different service files and it no
longer is useful to provide a "generic" (sic) example.
see also: https://github.com/rsyslog/rsyslog/issues/4333
- 2020-08-11: gnutls TLS driver bugfix: EKU check not done properly
When the server accepted a new connection, it did not properly set the
dataTypeCheck field based on the listening socket. That resulted in
skipping ExtendedKeyUsage (EKU) check on the client.
Thanks to Daiki Ueno for the patch.
- 2020-08-06: MMDARWIN:: improve configuration flexibility and UUID fix
-t pu now able to get fields from local variables ($.)
- now able to configure a custom root container for mmdarwin fields
- now able to put nested keys ($!key1!key2)
- don't regenerate a UUID each time, but instead check if one exists before
creating it (allow successive calls without losing previous UUID)
Thanks to github user frikilax for the contribution.
- 2020-08-06: add --enable-imjournal=optional ./configure option
- 2020-08-06: IMPCAP::Fixes: segfault, memory and build corrections
* fix bug in ethernet packets parsing
* fix removes build error with gcc10: 'multiple definition of...'
* resolve memory leak during interface init failure (device not freed after post-create error)
* add test 'impcap_bug_ether' to prove ethernet parser fix is working
Thanks to github user frikilax for the contribution.
closes https://github.com/rsyslog/rsyslog/issues/4332
- 2020-07-14: CI: add support for github actions
- 2020-07-14: imklog: add ruleset support
see also: https://github.com/rsyslog/rsyslog/issues/4344#issuecomment-658001854
see also: https://github.com/rsyslog/rsyslog/issues/106
- 2020-07-06: config system fix: ChkDisabled method to make config.enabled work
There was wrong negation in the method so it returned 0/1 in reverse
and also it did not mark the node to not be reported as unknown at all
times which is needed after all.
Thanks to Jiri Vymazal for the patch.
----------------------------------------------------------------------------------------
Scheduled Release 8.2006.0 (aka 2020.06) 2020-06-23
- 2020-06-22: queue: permit ability to double size at shutdown
This prevents message loss due to "queue full" when re-enqueueing data
under quite exotic settings.
see also https://github.com/rsyslog/rsyslog/issues/3941#issuecomment-549765813
closes https://github.com/rsyslog/rsyslog/issues/4020
- 2020-06-22:Fixing imfile segfaulting on selinux denial
If imfile is denied access to file watched trough symlink there is
unchecked condition resulting in access to not initialized memory.
- 2020-06-22: openssl: Fixed memory leak when tls handshake failed.
closes: https://github.com/rsyslog/rsyslog/issues/4319
- 2020-06-22: change systemd service file to wait for network
now that rsyslog is usually only installed for real syslog servers,
we should assume that some network listening or forwarding happens
on start. As such we need to start a bit later, after the network.
This poses no problem as systemd nowadays comes with journal which
is in almost all cases configured to buffer log data while
rsyslog is not yet running.
see also https://github.com/rsyslog/rsyslog-pkg-rhel-centos/issues/72
- 2020-06-22: NEW INPUT MODULE:: impcap, network packets input parser
Thanks to github user frikilax for the contribution.
- 2020-06-22: ksi bugfix: Optimized code in KSI module initialization fixed.
KSI module initialization will not stuck in infinite loop when code is
built with optimization -O2.
- 2020-06-05: operatingstatefile bugfix: month was given too low
The month was printed with the range 0 (January) to 11 (December).
This has now been corrected.
closes https://github.com/rsyslog/rsyslog/issues/4292
- 2020-06-05: build system: add "optional" build functionality to some components
Nameley:
--enable-libdbi=optional
--enable-mmdblookup=optional
--enable-imkafka=optional
--enable-omkafka=optional
If used, builds a dummy module which just emits a "module not supported
on this platform" error message when loaded.
Primary use case for this system is Debian-ish builds on SUSE OBS,
where we prefer to have a single package definition for all versions
(else things get much more complicated).
- 2020-05-23: config system bugfix: backticks cat segfault if file cannot be opened
when a `cat <filename>` construct is used in rsyslog.conf and <filename> can not
be accessed (does not exist, no permissions, ...), rsyslog segfaults.
Thanks to Michael Skeffington for notifying us and providing root cause analysis.
closes https://github.com/rsyslog/rsyslog/issues/4290
- 2020-05-15: imtcp bugfix: octet framing/stuffing problem with discardTruncatedMsg on
When "discardTruncatedMsg" was enabled in imtcp, messages were incorrectly
skipped if the last character before the truncation was the LFdelimiter.
Also adds two testbench tests for this case.
closes: https://github.com/rsyslog/rsyslog/issues/4281
- 2020-05-12: ompipe bugfix: race during HUP
When HUP was received, the write mutex was not aquired. This could
lead to unexpected invalidation of the output file descriptor.
Thanks to Julien Thomas for alerting us on this issue.
see also https://github.com/rsyslog/rsyslog/pull/4136#issuecomment-578326278
- 2020-05-12: ompipe: add action parameter tryResumeReopen
Sometimes we need to reopen a pipe after an ompipe action gets
suspended. Sending an HUP signal to rsyslog does the job but requires
an interraction with rsyslog. The patch adds support for a new boolean
option, tryResumeReopen, for the ompipe action. It mimics what an HUP
signal would do.
Thanks to Julien Thomas for the patch.
- 2020-05-12: imjournal: remove strcat call
Thanks to Jeff Marckel for the patch.
- 2020-05-12: build system: libzcmq version requirement needs to be bumped
Thanks to Thomas Deutschmann for pointing this out.
closes https://github.com/rsyslog/rsyslog/issues/3957
- 2020-05-12: testbench: download ElasticSearch binaries from rsyslog.com
The official ElasticSearch download site sometimes denies the download.
- 2020-05-11: openssl netstream driver bugfix: context leak
The context object was not properly freed.
Thanks to Michael Zimmermann for the fix.
- 2020-05-11: omhttp: Add support for multiple http headers
Allows the inclusion of multiple http headers on the REST call.
Thanks to callmegar for the patch.
- 2020-04-29: core bugfix: group id could not be obtained for very large groups
Thanks to github user emilbart for the patch.
- 2020-04-29: testbench additions (relp broken connection test)
- 2020-04-29: omudpspoof bugfix: issues with oversized messages
First issue was an incorrect packet length in UDP Header. It has to be the FULL UDP Packet
regardless of the MTU Setting. As a result regardless of IP fragmentation, the MTU setting
also limited the siizmax size of the UDP message.
The second issue was incorrect calculation of the UDP Checksum with libnet if
IP fragmentation was used (Based on MTU Setting). As a result, the network packets were
dropped by the tcp stack before they even could reach there target. The workarround for this
problem is, that we set the UDP Checksum to 0x0000 which allows skipping of the checksum
test. Fixing the problem by calculating the correct UDP Checksum would require some
code changes in the libnet.
Also fixed the omudpspoof bigmsg test and increased the testing size to 16KB.
- 2020-04-29: omprog: fix assert failed on HUP with output flag
If the 'output' setting of omprog was used and rsyslog received a HUP
signal just after starting (and before the omprog action received the
first log to process), an internal assertion could fail, causing
rsyslog to terminate. The failure message was "rsyslogd: omprog.c:660:
closeOutputFile: Assertion `pCtx->bIsRunning' failed."
The failure could also occur if rsyslog received a HUP signal during
the shutdown sequence.
This bug was introduced in v8.2004 by PR https://github.com/rsyslog/rsyslog/pull/4255
Although a test already existed that checked the interaction of HUPs
with the 'output' setting, it didn't always fail in this particular case
due to timing conditions. The test has been improved to cover this case
more reliably.
Thanks to Joan Sala Isern for the patch.
----------------------------------------------------------------------------------------
Scheduled Release 8.2004.0 (aka 2020.04) 2020-04-28
- 2020-04-28: ksi bugfix: When KSI module is suddenly closed, files are finalized
In async. mode all pending signature requests are closed immediately and
unsigned block marker is attached with message about sudden closure.
Similar approach is used for blocks that already contain some records.
Empty blocks are just closed without any metadata.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: ksi bugfix: Signer thread initialization is verified before usage.
When signer thread is created in rsksiInitModule thread successful
initialization is verified before returning the function. This will
prevent adding records to not initialized module and in case of an
error signature files opened will contain only magic bytes.
Thread flags replaced with thread state.
When init module fails, module is disabled.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: ksi bugfix: Hardcoded default hash algorithm replaced with 'default'
Instead of hardcoded SHA-256 KSI_getHashAlgorithmByName("default")
is used to get default hash function.
Function rsksiSetHashFunction and SetCnfParam updated.
Thanks to Taavi Väljaots for the patch.
- 2020-04-28: imfile bugfix: poential segfault in stream object on file read
- if cstrLen(pThis->prevMsgSegment) > maxMsgSize then len calculation
become negative if cstrLen(thisLine) < cstrLen(pThis->prevMsgSegment)
This causes illegal access to memory location and thus causing segfault.
- assigning len = 0 if cstrLen(pThis->prevMsgSegment) > maxMsgSize so that
it access the correct memory location.
Thanks to github user jaankit
- 2020-04-28: openssl TLS drivers: made more reliable for older openssl versions
OpenSSL can retry some failed operations, but older versions need an explicit
opt-in to do so. This is now done.
- 2020-04-28: omprog: fix bad fd errors in daemon mode
When omprog was used with the 'forceSingleInstance=on' option, and/or
the 'output' setting, "bad file descriptor" errors occurred, which
prevented the external program to be executed and/or the program output
to be correctly captured. The bug could also manifest as "resource
temporarily unavailable" errors, or other errors related to the use of
invalid/reassigned file descriptors. These errors only happened when
rsyslog ran in daemon mode (i.e. they didn't happen if rsyslogd was
run with the '-n' option).
The cause of the bug was that omprog opened the pipe fds needed by
these flags during the configuration load phase (in the 'newActInst'
module entrypoint). This is a bad place since the fork of the daemon
occurs after this phase, and all fds are closed when the daemon process
is started (see 'initAll' in rsyslogd.c), hence invalidating the
previously opened fds.
To correct this, the single child process and the output capture thread
are now started later, when the first log message is received by the
first worker thread. (Note: the 'activateCnf' module entrypoint, despite
being invoked after the fork, cannot be used for this purpose, since it
is invoked per module, not per action instance.)
Currently no automated test exists for this use case since the testbench
always runs rsyslog in non-daemon mode.
Affected versions: v8.38 and later
closes: https://github.com/rsyslog/rsyslog/issues/4247
Thanks to Joan Sala Isern for the patch.
- 2020-04-28: omfile bugfix: $outchannel split log lines at rotation time
- 2020-04-17: openssl: add support for libreSSL
Disable use of "@SECLEVEL" in default cipher string and
avoid SSL_CONF_CTX_set_flags() API when LIBRESSL is used.
This means tlscommands will not work.
closes: https://github.com/rsyslog/rsyslog/issues/4210
- 2020-03-04: imudp bugfix: build problems on some Linux kernel versions
Thanks to Wen Yang for the patch.
- 2020-03-02: conf output bugfix: -o produces missing space between call and rulename
Thanks to Tetiana Ohnieva for the patch.
closes https://github.com/rsyslog/rsyslog/issues/3761
----------------------------------------------------------------------------------------
Scheduled Release 8.2002.0 (aka 2020.02) 2020-02-25
- 2020-02-25: imfile: add per minute rate limiting
Add MaxBytesPerMinute and MaxLinesPerMinute options.
These take integer values and, respectively, limit the number
of bytes or lines that may be sent in a minute.
This can be used to put a limit on the count or volume of logs
that may be sent for an imfile.
Thanks to Greg Farrell for the patch.
- 2020-02-24: core: add global parameter "security.abortOnIDResolutionFail"
This parameter controls whether or not rsyslog aborts when a name ID
lookup fails (for user and group names). This is necessary as a security
measure, as otherwise the wrong permissions can be assigned or privileges
are not dropped.
CHANGE OF BEHAVIOR
The default for this parameter is "on". In previous versions, the default
was "off" (by virtue of this parameter not existing). As such, existing
configurations may now error out.
We have decided to accept this change of behavior because of the potential
security implications.
closes https://github.com/rsyslog/rsyslog/issues/4164
- 2020-02-24: openssl TLS driver bugfix: chained certificates were not accepted
This was supported since always inside GnuTLS driver, but was missing for openssl one.
- 2020-02-24: core bugfix: too early parsing of incoming messages
In theory, rsyslog should call parsers on the queue worker threads whenever
possible. This enables the parsers to be executed in parallel. There are
some cases where parsers needs to be called earlier, namely when parsed
data is needed for rate-limiting.
The logic to do this previously did not work correctly and was fixed six
years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive:
it actually makes the early parser call now mandatory, effectively moving
parsing to the input side where there is no to little concurrency.
We still do not need to call the parser when all messages, regardless of
severity, need to be rate-limited. This is the default and very frequent
case. This patch introduces support for this and as such makes parsers
able to run in parallel in the frequent case again.
closes https://github.com/rsyslog/rsyslog/issues/4187
- 2020-02-20: testbench bugfix: two minor issues in omkafkadynakey.sh test
lead to false positives during test runs (depending on circumstances)
closes: https://github.com/rsyslog/rsyslog/issues/4134
- 2020-02-20: testbench: set max extra data length for tcpflood from 200 to 512KiB
Added a imrelp test for big messages (256KB).
closes: https://github.com/rsyslog/rsyslog/issues/4158
- 2020-02-20: config system bugfix: 'config.enabled' directive oddities
Previously the directive was processed way too late which caused false
errors whenever it was set to 'off' and possibly other problems.
Thanks to Jiri Vymazal for the patch.
- 2020-02-09: imfile bugfix: timeout did not work on very busy system
The timeout feature was soley based on timeouts of the poll()
system call. On a very busy system, this would probably happen
very seldomly. Moreover, the timeout could occur later than
expected on any system with high load.
The issue was not reported from practice but discovered during
CI system improvements.
- 2020-01-30: build system: change --enable-imfile-tests default to "yes"
This was accidentally set to "no" some time ago (actual commit unknown). Tests for
imfile should by default run when imfile is enabled.
see also https://github.com/rsyslog/rsyslog/issues/4120
- 2020-01-27: build system: add option --enable-gnutls-tests
This enables us to build GNUtls support but not necessarily
test it in CI. This is useful for some specialised subcomponent
test. The default is enabled if gnutls is enabled and disabled if not.
- 2020-01-26: testbench: new test for loadbalancing via global vars
This is a popular functionality which had not been routinely tested
in the past.
- 2020-01-26: mmdblookup bugfix: invalid data returned when no entry found
Since the upgrade of the package libmaxminddb on FreeBSD (1.3.2_2 -> 1.4.2),
the module mmdblookup returns the first entry of the mmdb database even if the entry
is not found. After some debug, I found the solution in the official maxminddb
repository : to check if the entry is in database, we must check the found_entry
attribute, otherwise the function MMDB_get_entry_data_list will return the first
entry of the database if the entry is not found in it.
Thanks to Kevin Guillemot for the patch.
- 2020-01-23: oversize message log bugfix: do not close fd -1
The oversize message log fd is always closed on HUP, even if it never
was opened (and thus has -1 value). This patch corrects the issue.
The bug had no know-bad effect in practice other than getting an
(ignored) error status from close(). However, it introduced warnings
in test runs (e.g. when running under valgrind).
- 2020-01-22: imfile bugfix: saving of old file_id for statefiles
Previously we saved old file_id unconditionally, which led to not
deleting old statefiles if files changes without rsyslog running.
Now it should work correctly.
Thanks to Jiri Vymazal for the patch.
- 2020-01-22: imfile bugfix: misadressing and potential segfault
Commit 3f72e8c introduced an invalid memory allocation size. This lead to
too-short alloc and thus to overwrite of non-owned memory. That in turn
could lead to segfaults or other hard to find problems.
The issue was detected by our upgraded CI system. We did not receive
any problem reports in practice. Nevertheless, the problem is real and
people should update affected versions to patched ones.
The bug was present in scheduled stable release 8.1911.0 and 8.2001.0.
see also: https://github.com/rsyslog/rsyslog/issues/4120
see also: https://github.com/rsyslog/rsyslog/pull/4141
- 2020-01-20: core bugfix: potential race during HUP
when rsyslog is HUPed immediately after startup and before it is fully
initialized, there is a potential race with the list of loaded modules.
This patch ensures no bad things can happen in that case.
Detected by LLVM TSAN, not seen in practice.
- 2020-01-20: testbench improvements and fixes
modernize tests, reduce robustness against slow machines, provide some
test framework functional enhancements, and optimize some tests.
Also includes some code changes to C testing components. Among others,
tests have slightly been speeded up by reducing the wait time at queue
shutdown. This is possible because of better overall completion checks.
----------------------------------------------------------------------------------------
Scheduled Release 8.2001.0 (aka 2020.01) 2020-01-14
- 2020-01-12: core bugfix: race condition related to libfastjson when using DA queue
Rsyslogd aborts when writing to disk queue from multiple workers simultaneously.
It is assumed that libfastjson is not thread-safe.
Resolve libfastjson race condition when writing to disk queue.
see also https://github.com/rsyslog/rsyslog/issues/4041
Thanks to MIZUTA Takeshi for the fix.
- 2020-01-12: omfwd bugfix: parameter streamdriver.permitexpiredcerts did not work
closes https://github.com/rsyslog/rsyslog/issues/4098
- 2020-01-11: Bugfix: KSI module + dynafile in asynchronous mode fixed
Thanks to Taavi Valjaots for the patch
- 2020-01-08: tls driver: add support to configure certificate verify depth
Support added in omfwd as instance parameter:
streamdriver.TlsVerifyDepth
Support added in imtcp as module parameter:
streamdriver.TlsVerifyDepth
Can be 2 or higher.
Support added into ossl driver
Support added into gtls driver
Added testcases for both drivers.
closes: https://github.com/rsyslog/rsyslog/issues/4035
- 2020-01-08: modernization of testbench
moved some tests to newer standards, hardened them against slow testbench machines,
kafka component download improvements, and prevent dangling left-over test tool
instances from aborted tests
- 2020-01-07: tls subsystem bugfix: default for permitExpiredCerts was invalidly "on"
The problem occured with commit 3d9b8df in December 2018 and went into
scheduled stable 8.1901.0. Unfortunately, the change in default was not detected
until a year later. This commit re-enables the previous default ("off"), which is
also the only sensible default from a security PoV. Unfortunately, new 2019
deployments may begin to see connection rejection when usin expired certs. As
expired certs should not be used, this hopefully will not cause problems in
practice.
Thanks to Jiri Vymazal for the patch.
- 2020-01-01: testbench: improve ElasticSearch test speed
We now support re-using suitable running ES instances, which reduces the
number of restarts.
- 2019-12-31: omelasticsearch: improve curl reply buffer handling
The curl reply buffer (pWrkrData->reply) was allocated, realloced and freed with
each request. This has now been reduced to once per module, slightly increasing
overall performance.
closes https://github.com/rsyslog/rsyslog/issues/1964
- 2019-12-31: config system: emit proper error message on $ in double-quoted string
closes https://github.com/rsyslog/rsyslog/issues/2869
- 2019-12-30: core bugfix: rsyslog aborts when config parse error is detected
In defaut settings, rsyslog tries to continue to run, but some data
structures are not properly initialized due to the config parsing error.
This causes a segfault.
In the following tracker, this is the root cause of the abort:
see also https://github.com/rsyslog/rsyslog/issues/2869
- 2019-12-30: fix some alignment issues
So far, this worked everywhere (for years). But it may still have
caused issues on some platforms.
closes https://github.com/rsyslog/rsyslog/issues/2608
- 2019-12-27: core bugfix: APP-NAME fields could become empty
RFC 5424 specifies that an empty APP-NAME needs to be indicated by
"-". Instead, the field could become empty under certain conditions.
If so, outgoing 5424 messages were invalidly formatted.
This happened under quite unusual conditions, but could be seen
in practice.
closes https://github.com/rsyslog/rsyslog/issues/4043
- 2019-12-27: core bugfix: reopen /dev/urandom file descriptor after fork on Linux
This patch updates prepareBackground() in tools/rsyslogd.c to reopen any file
descriptors used for random number generation in the child process. This fixes
an issue on Linux systems where the file descriptor obtained for /dev/urandom
by seedRandomNumber() in runtime/srutils.c was left closed after the fork. This
could be observed in procfs, where /proc/fd/ would show no open descriptors to
/dev/urandom in the forked process. /dev/urandom is reopened as the child may be
be operating in a jail, and so should not continue to use file descriptors from
outside the jail (i.e. inherited from the parent process).
I found that this issue led to rsyslog intermittently hanging during seedIV()
in runtime/libgcry.c. After the fork, the closed file descriptor number tended
to get re-assigned. randomNumber() would then read from an incorrect (although
still valid) file descriptor, and could block (depending on the state of that
file descriptor). This gave rise to the intermittent hang that I observed.
Thanks to Simon Haggett for the patch.
- 2019-12-20: imdocker bugfix: did not compile without atomic operations
- 2019-12-20: omclickhouse: new parameter "timeout"
Thanks to Pavlo Bashynskiy for the patch.
- 2019-12-20: omhiredis: add 'set' mode plus some fixes
- new mode 'set' to send SET/SETEX commands
- new parameter 'expiration' to send SETEX instead of SET commands (only applicable to 'set' mode)
- fixes to missing frees
Thanks to github user frikilax for the patch.
- 2019-12-18: relp: Add support setting openssl configuration commands.
Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp.
(Using relpSrvSetTlsConfigCmd and relpCltSetTlsConfigCmd)
OpenSSL Version 1.0.2 or higher is required for this feature.
A list of possible commands and their valid values can be found in the
documentation: https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html
The setting can be single or multiline, each configuration command is
separated by linefeed (n). Command and value are separated by
equal sign (=). Here are a few samples:
tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.2"
tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
MinProtocol=TLSv1.2"
Add to new testcases for librelp and tlscfgcmd.
closes https://github.com/rsyslog/rsyslog/issues/3959
- 2019-12-18: bugfix core: potential segfault in template engine
under some circumstances (not entirely clear right now), memory
was freed but later re-used as state-tracking structures were not
properly maintained. Github issue mentioned below has full details.
Thanks to github user snaix for analyzing this issue and providing
a patch. I am committing as myself as snaix did not disclose his or
her identity.
closes https://github.com/rsyslog/rsyslog/issues/3019
closes https://github.com/rsyslog/rsyslog/issues/4040
- 2019-12-18: fixed some minor issues detected by clang static analyzer 9
- 2019-12-10: core/config bugfix: false error msg when config.enabled="on" is used
When the 'config.enabled="on"' config parameter an invalid error message
was emitted that this parameter is not supported. However, it was still
applied properly. This commit removes the invalid error message.
closes https://github.com/rsyslog/rsyslog/issues/4011
- 2019-12-03: omsnmp bugfix: "traptype" parameter invalidly rejected value 6
"Traptype" needs to support values 0 to 6.
However, if value 6(ENTERPRISESPECIFIC) was set, an invalid error message
was emitted. Otherwise processing was correct.
This could lead to problems with automatic config deployment,
as valid configurations were invalidly reported as incorrect.
That in turn could make a deployment fail.
closes https://github.com/rsyslog/rsyslog/issues/3973
- 2019-12-03: omsnmp: add new parameter "snmpv1dynsource"
If set, the source field from SNMPv1 trap can be overwritten
with a template, default is "%fromhost-ip%". The content should be a
valid IPv4 Address that can be passed to inet_addr(). If the content
is not a valid IPv4 Address, the source will not be set.
closes: https://github.com/rsyslog/rsyslog/issues/3991
- 2019-12-02: imfile bugfix: state file renaming sometimes did not work properly
Now checking if file-id changes and renaming - cleaning state file
accordingly and always checking and cleaning old inode-only style
state files.
Thanks to Jiri Vymazal for the patch.
- 2019-12-02: ratelimit: increase rate limit interval parameter max value
The burst parameter in the ratelimit was increased to an unsigned int
but the interval remained an unsigned short. While it may be unusual,
there is possibly a chance to need to represent an interval longer than
about 3/4 of a day.
While here, go through and normalize all the various incarnations of
rate limiting to be explicitly unsigned int for the burst and interval.
Thanks to github user frikilax for the patch.
- 2019-12-02: ommongodb: Add other supported formats for 'time' and 'date' fields
Thanks to github user frikilax for the patch.
- 2019-12-02: imjournal bugfix: too many messages in error case
Under certain error conditions, `ignorePreviousMessages="on"` could be ignored
an existing messages be processed.
Thanks to github user 3chas3 for the patch.
- 2019-11-27: core bugfix: action on retry mangles messages
When a failed action goes into retry, template content is rendered
invalid if the action uses more than 1 template.
closes https://github.com/rsyslog/rsyslog/issues/3898
Thanks to Mikko Kortelainen for the patch.
- 2019-11-27: testbench: improve mysql testing support
tests can now run in parallel and are hardened against several glitches
- 2019-11-22: omhttp: add basic support for Loki Rest
Loki is a new message indexer and querier from Grafana Labs. See
https://github.com/grafana/loki for details on Loki.
This change provides the initial message structure to send bulk message
payloads to the Loki Rest endpoint. omhttp, received a new bulk message
format called lokirest. Additionally, the plugin relies on the user to
provide the correct "stream" read message format.
A loki template must be json compatible and include a "stream" key of
key value tags, and a values key of an array of 2 element arrays, where
each 2 element array is the unix epoch in nanoseconds followed by an
unstructured message.
An example:
template(name="array_loki" type="string" string="{\"stream\":{\"host\":\"%HOSTNAME%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"syslogtag\":\"%syslogtag%\"},\"values\": [[ \"%timegenerated:::date-unixtimestamp%000000000\", \"%msg%\" ]]}")
- 2019-11-22: testbench: obtain python binary path via AM_PATH_PYTHON
see also https://github.com/rsyslog/rsyslog/issues/3853
- 2019-11-22: omprog: detect violation of interface protocol
The spec for the omprog interaction with the program it calls specifies
that the program receives one message via one line. In other words:
it must be a string terminated by LF.
However, omprog does currently rely on a proper template to fulfill this
requirement, If the template does not provide for the LF, it is never
written. For the called program, this looks like it does not receive any
input at all. Even if it finally reads data (e.g. due to full buffer),
it will not properly be able to discern the messages.
This handling is improved with this commit.
We cannot just check the template, because at the end of the template
may by a non-constant value. As such, we do not know at config load
time if there is this problem or not.
So the correct approach is to, during runtime, check if each message
is properly terminated. For those that are not:
* we append a LF, because anything else makes matters worse
* log a warning message, at least for a sample of the messages
The warning is useful in the (expected most often) case that the template
is simply missing the LF. While appending works, it slows down processing.
As such the user should be given a chance to correct the config bug.
To avoid clutter, the warning is emitted at most once every 30 seconds.
This value is hardcoded as we do not envision a need to adjust it. Usually
users should quickly fix the template.
closes https://github.com/rsyslog/rsyslog/issues/3975
- 2019-11-19: core queue: emit warning if parameters are set for direct queue
Direct queues do not apply queue parameters because they are actually
no physical queue. As such, any parameter set is ignored. This can
lead to unintentional results.
The new code detects this case and warns the user.
closes https://github.com/rsyslog/rsyslog/issues/77
- 2019-11-19: imjournal bugfix: do not wait too long on recovery try
When trying to recover journal errors, imjournal waited a hardcoded
period of 10s between tries. This was pretty long and could lead to
loss of journal data.
This commit adjust it to 100ms, which should still be fully sufficient
to prevent the journal from "hammering" the CPU.
It may be worth considering to make this setting configurable - but
let's first see if there is real demand to actually do that.
closes https://github.com/rsyslog/rsyslog/issues/3969
- 2019-11-19: mmutf8fix: enhance handling of incorrect UTF-8 sequences
1. Invalid utf8 detection didn't handle 3 and 4-byte overlong encodings (2
byte overlong encodings were handled explicitly by rejection E0 and E1
start bytes). Unified checks for overlong encodings.
2. Surrogates U+D800..U+DFFF are not valid codepoints (Unicode Standard, D92)
3. Replacement of characters in invalid 3 or 4-bytes encodings was too
eager. It must not replace bytes which are valid UTF-8 sequences. For
example, in [0xE0 0xC2 0xA7] sequence the 0xC2 is invalid as a continuation
byte, but it starts a valid UTF8 symbol [0xC2 0xA7]. That is, with current
code processing the sequence will result in "???" but the correct result is "?§"
(provided that the replacement character is "?").
4. Various tests for UTF-8 invalid/valid sequences.
Thanks to Sergei Turchanov for the patch.
- 2019-11-14: imfile: add new input parameter escapeLF.replacement
The new parameter permits to specify a replacement to be configured
when "escapeLF" is set to "on". Previously, a fixed replacement string
was used ("#012"/"\n") depending on circumstances. If the parameter is
set to an empty string, the LF is simply discarded.
closes https://github.com/rsyslog/rsyslog/issues/3889
----------------------------------------------------------------------------------------
Scheduled Release 8.1911.0 (aka 2019.11) 2019-11-12
- 2019-11-12: core queue: add config param "queue.takeFlowCtlFromMsg"
This is a fine-tuning option which permits to control whether or not
rsyslog shall alays take the flow control setting from the message. If
so, non-primary queues may also block when reaching high water mark.
This permits to add some synchronous processing to rsyslog core engine.
However, it is dangerous, as improper use may make the core engine
stall. As such, enabling this option requires very careful planning
of the rsyslog configuration and deep understanding of the consequences.
Note that the option is applied to individual queues, so a configuration
with a large number of queues can (and must if use) be fine-tuned to
the exact use case.
The rsyslog team strongly recommends to let the option turned off,
which is the default setting.
see also https://github.com/rsyslog/rsyslog/issues/3941
- 2019-11-12: imrelp: add new config parameter "flowcontrol"
This permits to fine-tune the flowControl parameter. Possible values are
"no", "light", and "full". With light being the default and previously
only value.
Changing the flow control setting may be useful for some rare applications,
but be sure to know exactly what you are doing when changing this setting.
Most importantly, whole rsyslog may block and become unresponsive if you
change flowcontrol to "full". While this may be a desired effect when
intentionally trying to make it most unlikely that rsyslog needs to
lose/discard messages, usually this is not what you want.
see also https://github.com/rsyslog/rsyslog/issues/3941
- 2019-11-11: imrelp: remove unsafe debug instrumentation
dbgprintf, which is not signal safe, was called from a signal handler
to get better understanding during debugging. While this usually works,
it can occasionally (5%) lead to a hang during shutdown. We have now
removed that debug info as it is no longer vital.
Note: this could only happen during debug runs. Production mode was
not affected. As such, this fix is only relevant to developers.
However, it caused some confusion in the following issue tracker.
see also https://github.com/rsyslog/rsyslog/issues/3941
- 2019-11-06: ossl driver bugfix: fix wrong OpenSSL Version check
Fix OpenSSL Version check in:
- SetGnutlsPriorityString function in nsd_ossl.c
- initTLS() function tcpflood.c
See https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_VERSION_NUMBER.html
for more.
This bug lead to not enabling some functionality correctly.
Removed "MinProtocol=TLSv1.1" from two testcases because MinProtocol
is only supported by OpenSSl 1.1.0 or higher and was not really
necessary for the testcases.
closes https://github.com/rsyslog/rsyslog/issues/3939
- 2019-11-05: mmdarwin: Optimizations, new parameters, update to protocol header
- use permanent worker-dependent buffers to avoid malloc/free for each entry
- move socket structures to worker data, remove global mutex
- add log lines for parameters and general workflow
- don't send body if empty/incomplete (see new parameters)
- don't close/reopen socket every time -> let session open or create new every X
entry (see new parameters)
- clean up code
- added 'send_partial', to let mmdarwin send body if not all fields were
retrieved, or not; default false = only send complete bodies
- added 'socket_max_use' to open new session every X packet, useful for
some versions of Darwin (prior to 1.1)
default is 0 = do not open new session/keep only one
- added 'evt_id' to the darwin header (Darwin v1+ compatibility)
Note: mmdarwin is a contributed module
Thanks to github user frikilax for the patch.
- 2019-11-01: mmkubernetes bugfix: improper use of realloc()
could cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this
- 2019-10-31: imjournal: set the journal data threshold to MaxMessageSize
When data is read from the journal using sd_journal_get_data it may be
truncated to a certain threshold (64K by default).
If the rsyslog MaxMessageSize is larger than the threshold, there is a
chance rsyslog will receive incomplete messages from the journal.
Empirically, this appears to happen reliably when XZ compression is
used by journald. Systems where journald uses LZ4 compression do not
appear to suffer this issue reliably--if at all.
This change sets the threshold to the MaxMessageSize when the
journal is opened.
Thanks to Robert Winslow Dalpe for the patch.
- 2019-10-30: improg bugfix: allow improg to handle multi-line inputs
miscellaneous bug fixes in improg:
* properly truncate string after an input event is submitted
* set msgoffset to 0.
* tests added to check above fixes
Thanks to Nelson Yen for the fix.
- 2019-10-30: mmdblookup bugfix: missing space in city name
This fixes the issue that spaces in city names are dropped. However, the
fix is more or less a work-around. As it turns out, the libmaxminddb API
is not correctly used. In the somewhat longer term, we should fix this.
see also https://github.com/maxmind/libmaxminddb/issues/218
closes https://github.com/rsyslog/rsyslog/issues/1650
- 2019-10-30: core/queue: provide ability to run diskqueue on multiple threads
Up until this release, disk queues could only use a single thread,
what limited their performance with outputs like ElasticSearch.
Now disk queues can utilize multiple threads just like any other
queue type. Most importantly, the disk queue part of a DA queue
now inherits the max number of threads from its memory queue
counterpart.
NOTE: the new multi-threaded DA disk queue is actually a change of
behavior. We have not guarded it by a new config switch as we
assume the new behavior is most often exactly within user
expectations. In any case, we cannot see any harm from running
the disk queue on multiple threads.
see also https://github.com/rsyslog/rsyslog/issues/3543
closes https://github.com/rsyslog/rsyslog/issues/3833
- 2019-10-25: omfile bugfix: file handle leak
The stream class does not close re-opened file descriptors.
This lead to leaking file handles and ultimately to the inability
to open any files/sockets/etc as rsyslog ran out of handles.
The bug was depending on timing. This involved different OS
thread scheduler timing as well as workload. The bug was more
common under the following conditions:
- async writing of files
- dynafiles
- not committing file data at end of transaction
However it could be triggered under other conditions as well.
The refactoring done in 8.1908 increased the likelihood of
experiencing this bug. But it was not a real regression, the new
code was valid, but changed the timing so that the race was more
likely.
Thanks to Michael Biebl for reporting this bug and helping to
analyze it.
closes https://github.com/rsyslog/rsyslog/issues/3885
- 2019-10-22: imfile bugfix: improper use of calloc()
could cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this
- 2019-10-22: TLS driver bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this
- 2019-10-22: imuxsock bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this
- 2019-10-17: build system bugfix: incorrect default in ./configure help text
closes https://github.com/rsyslog/rsyslog/issues/3904
Thanks to Michael Biebl for pointing this out.
- 2019-10-17: mmkubernetes bugfix: improper use of calloc()
can cause problems under extreme memory shortage - very unlikely
credits to LGTM.COM for detecting this
- 2019-10-16: core queue bugfix: propagate batch size to DA queue
This was a long-standing bug where the DA queue always had a fixed small batch
size because the setting was not propagated from the memory queue. This also
removes a needless and counter-productive "debug aid" which seemed to be in
the code for quite some while. It did not cause harm because of the batch
size issue.
- 2019-10-16: testbench: fix unreliable gzipwrite test
The test was timing-sensitive as we did not properly check all data
was output to the output file - we just relied on sleep periods.
This has been changed. Also, we made some changes to the testing
framework to fully support sequence checking of multiple ZIP files.
- 2019-10-16: core queue bugfix: handle multi-queue-file delete correctly
Rsyslog may leave some dangling disk queue files under the following
conditions:
- batch sizes and/or messages are large
- queue files are comparatively small
- a batch spans more than two queue files (from n to n+m with m>1)
In this case, queue files n+1 to (n+m-1) are not deleted. This can
lead to problems when the queue is re-opened again. In extreme cases
this can also lead to stalled processing when the max disk space is
used up by such left-over queue files.
Using defaults this scenario is very unlikely, but it can happen,
especially when large messages are being processed.
- 2019-10-16: imjournal: fix regression from yesterday's patch
commit 78976a9bc059 introduced a regression that caused writing
the journal state file to fail. This happens when the state file
is given as relative file name and the working directory is also
a relative path. This situation is very uncommon. So most deployments
will never experience it. We discovered the issue during CI runs
where the trigger condition is given. Note that it also takes
multiple times of loading the journal to actually see the bug.
see also https://github.com/rsyslog/rsyslog/pull/3878
- 2019-10-15: imjournal plugin code restructuring, added remote option
Decomposed ReadJournal() a bit, also now coupling journald
variables in one struct, added few warning messages and debug
prints to help with bug hunts in future, also got rid of two
needless journald calls. WorkAroundJournalBug now deprecated.
Added option to pull journald records from outside local machine.
Thanks to Jiri Vymazal for the patch.
- 2019-10-11: core bugfix: potential abort on very long action name
The action name is stored in modified form for the debug header and
some messages. If it is extremely long, a buffer can be overrun,
resulting in misaddressing and potential segfault for rsyslog. This
can also happen if the action is NOT named, but a custom path to
the output module is given and that path is very long. This triggers
the same issue because by default the module load path is included
in the action name.
This patch corrects the problem and truncates overly long names
when being used for name generation.
The problem was detected during testbench work. We did never receive
a bug report from practice.
- 2019-10-10: testbench: add test for mmpstrucdata with RFC5424 escape sequences
----------------------------------------------------------------------------------------
Scheduled Release 8.1910.0 (aka 2019.10) 2019-10-01
- 2019-10-01: core bugfix: incorrect error message on duplicate module load
A Null-pointer was passed to printf instead of the module name.
On some platforms this may lead to a segfault. On most platforms
printf check's for NULL pointers and uses the string "(null)"
instead. In any case, the module name is missing from the error message.
- 2019-10-01: imczmq nitfix: potential NULL ptr in printf on out-of-memory condition
very unlikely to happen but if it does without any real issue on most platforms.
- 2019-10-01: work around some compiler warning messages induced by pthreads API
- 2019-10-01: core ratelimiting: more verbose message when rate-limiting happens
When messages are rate-limited, the error message now also contains the
rate limiter setting. This enables the user to more quickly understand what
the problem is (especially if default values apply).
Thanks to Jiri Vymazal for the patch.
- 2019-10-01: openssl TLS driver: do not emit unnecessary error message
On older openssl versions, an API was missing to set user-defined parameters. If we
had such an older version, rsyslog emitted an error message even if the user did
not configure such parameters. This has been corrected, so that a message is only
emitted if there really is a problem. Based on user feedback the severity has also
been downgraded to "warning".
- 2019-10-01: pmcisconames (contributed module) bugfix: potential misaddressing
- 2019-09-30: pmaixforwardedfrom (contributed module) bugfix: potential misaddressing
- 2019-09-30: pmdb2diag (contributed module) bugfix: Out of bounds issue
Add a new sanity check after determining the level len.
Thanks to Philippe Duveau for the patch.
see also: https://nvd.nist.gov/vuln/detail/CVE-2019-17040
- 2019-09-02: ability to set stricter TLS operation modes
- checking of extendedKeyUsage certificate field
- stricter checking of certificate name/addresses
Thanks to Jiri Vymazal for the patch.
- 2019-08-21: testbench: add basic test for immark
- 2019-08-20: core: do not unnecessarily set hostname on each HUP
- 2019-08-20: build system: support cross-platform build for mysql/mariadb
rsyslog fails to cross build from source, because it uses mysql_config
and mysql_config is unfixably broken for cross compilation. It would be
better to use pkg-config. The attached patch makes rsyslog try
pkg-config first and fall back to mysql_config.
Thanks to Helmut Grohne for providing a base patch.
- 2019-08-20: core/tcpsrv: potential race on startup/shutdown
if the tcpsrv component is started and quickly terminated, it may hang
for a short period of time. Also a very small amount of memory is leaked
immediately before shutdown. While this leak is irrelevant in practice
(the OS clean up the process anyways), it leads to CI failures. The hang,
however, can lead to longer than expected shutdown times for rsyslog.
The problem can be experienced via imtcp, imgssapi and imdiag (users
of affected core component).
----------------------------------------------------------------------------------------
Scheduled Release 8.1908.0 (aka 2019.08) 2019-08-20
- 2019-08-19: testbench: add test for $allowedSender functionality
- 2019-08-19: testbench: harden some tests against very slow CI machines
- 2019-08-16: testbench: make most tests use a port file and assign listen port 0
This makes the test much more robust against heavily loaded test systems.
- 2019-08-16: core/action: guard action.externalstate.file content against whitespace
remove trailing whitespace before checking the status string. This is
most important as a line usually ends with \n, which is considered
trailing whitespace. Accepting this increases usability.
- 2019-08-16: imtcp bugfix: multiple listenerPortFile parameter did not work
... because they were treated as module-global. If we had multiple imtcp
listeners with multiple port files, only the last filename was always used.
closes https://github.com/rsyslog/rsyslog/issues/3817
- 2019-08-16: testbench: improve testbench plumbing for gzip and fail cases
We have added new capabilities to the testbench plumbing to automatically
deal with gzip-compressed files. This also permits to use the wait_seq_check
function to work for gzip tests as well. The known-timing-sensitive
gzipwr_large test now makes use of the new capabilities. This enables us