Merge pull request #3411 from lbudai/version/3.29

Version/3.29

NEWS.md

@@ -1,104 +1,144 @@
-3.28.1
+3.29.1
 ======
 
 ## Highlights
 
- * `http`: add support for proxy option
+ * `panos-parser()`: parse Palo Alto PAN-OS logs
 
    Example:
    ```
+   @include "scl.conf"
+
    log {
-      source { system(); };
-      destination { http( url("SYSLOG_SERVER_IP:PORT") proxy("PROXY_IP:PORT") method("POST") ); };
+     source { network(transport("udp")); };
+
+     parser { panos-parser(); };
+
+     destination {
+      elasticsearch-http(
+        index("syslog-ng-${YEAR}-${MONTH}-${DAY}")
+        type("")
+        url("http://localhost:9200/_bulk")
+        template("$(format-json
+          --scope rfc5424
+          --scope dot-nv-pairs --rekey .* --shift 1 --exclude *future_* --exclude *dg_hier_level_*
+          --scope nv-pairs --exclude DATE --key ISODATE @timestamp=${ISODATE})")
+      );
+     };
    };
    ```
-   ([#3253](https://github.com/syslog-ng/syslog-ng/pull/3253))
+   ([#3234](https://github.com/syslog-ng/syslog-ng/pull/3234))
 
 ## Features
 
- * `map`: template function
-
-   This template function applies a function to all elements of a list. For example: `$(map $(+ 1 $_) 0,1,2)` => 1,2,3.
-   ([#3301](https://github.com/syslog-ng/syslog-ng/pull/3301))
- * `use-syslogng-pid()`: new option to all sources
+ * snmptrap: improve error message when missing dependency
+   ([#3363](https://github.com/syslog-ng/syslog-ng/pull/3363))
+ * disk queue: reduce memory usage during load
+   ([#3352](https://github.com/syslog-ng/syslog-ng/pull/3352))
+ * config: support `@version: current`
+   ([#3368](https://github.com/syslog-ng/syslog-ng/pull/3368))
+ * Allow dupnames flag to be used in PCRE expressions, allowing duplicate names for named subpatterns
+   as explained here: https://www.pcre.org/original/doc/html/pcrepattern.html#SEC16 .
 
-   If set to `yes`, `syslog-ng` overwrites the message's `${PID}` macro to its own PID.
-   ([#3323](https://github.com/syslog-ng/syslog-ng/pull/3323))
+   Example:
+   ```
+   filter f_filter1 {
+     match("(?<FOOBAR>bar)|(?<FOOBAR>foo)" value(MSG) flags(store-matches, dupnames));
+   };
+   ```
+   ([#3381](https://github.com/syslog-ng/syslog-ng/pull/3381))
 
 ## Bugfixes
 
- * `affile`: eliminate infinite loop in case of a spurious file path
-
-   If the template evaluation of a log message will result to a spurious
-   path in the file destination, syslog-ng refuses to create that file.
-   However the problematic log message was left in the msg queue, so
-   syslog-ng was trying to create that file again in time-reopen periods.
-   From now on syslog-ng will handle "permanent" file errors, and drop
-   the relevant msg.
-   ([#3230](https://github.com/syslog-ng/syslog-ng/pull/3230))
- * Fix minor memory leaks in error scenarios
-   ([#3265](https://github.com/syslog-ng/syslog-ng/pull/3265))
- * `crypto`: fix hang on boot due to lack of entropy
-   ([#3271](https://github.com/syslog-ng/syslog-ng/pull/3271))
- * Fix IPv4 UDP destinations on FreeBSD
-
-   UDP-based destinations crashed when receiving the first message on FreeBSD due
-   to a bug in destination IP extraction logic.
-   ([#3278](https://github.com/syslog-ng/syslog-ng/pull/3278))
- * `network sources`: fix TLS connection closure
-
-   RFC 5425 specifies that once the transport receiver gets `close_notify` from the
-   transport sender, it MUST reply with a `close_notify`.
-
-   The `close_notify` alert is now sent back correctly in case of TLS network sources.
-   ([#2811](https://github.com/syslog-ng/syslog-ng/pull/2811))
- * `disk-buffer`: fixes possible crash, or fetching wrong value for logmsg nvpair
-   ([#3281](https://github.com/syslog-ng/syslog-ng/pull/3281))
- * `packaging/debian`: fix mod-rdkafka Debian packaging
-   ([#3282](https://github.com/syslog-ng/syslog-ng/pull/3282))
- * `kafka destination`: destination halts if consumer is down, and kafka's queue is filled
-   ([#3305](https://github.com/syslog-ng/syslog-ng/pull/3305))
- * `file-source`: Throw error, when `follow-freq()` is set with a negative float number.
-   ([#3306](https://github.com/syslog-ng/syslog-ng/pull/3306))
- * `stats-freq`: with high stats-freq syslog-ng emits stats immediately causing high memory and CPU usage
-   ([#3320](https://github.com/syslog-ng/syslog-ng/pull/3320))
- * `secure-logging`: bug fixes ([#3284](https://github.com/syslog-ng/syslog-ng/pull/3284))
-    - template arguments are now consistently checked
-    - fixed errors when mac file not provided
-    - fixed abort when derived key not provided
-    - fixed crash with slogkey missing parameters
-    - fixed secure-logging on 32-bit architectures
-    - fixed CMake build
+ * filter/regex: if there was a named match (?<named>..)? that is optional to match, the previose or the next named matches might not be saved as named match.
+   ([#3393](https://github.com/syslog-ng/syslog-ng/pull/3393))
+ * `tls`: Fixed a bug, where `ecdh-curve-list()` were not applied at client side.
+   ([#3356](https://github.com/syslog-ng/syslog-ng/pull/3356))
+ * scratch-buffers: fix `global.scratch_buffers_bytes.queued` counter bug
+   This bug only affected the stats_counter value, not the actual memory usage (i.e. memory usage was fine before)
+   ([#3355](https://github.com/syslog-ng/syslog-ng/pull/3355))
+ * wsl: fix infinite loop during startup
+   ([#3340](https://github.com/syslog-ng/syslog-ng/pull/3340))
+ * `openbsd`: showing grammar debug info for openbsd too, when `-y` command line option is used
+   ([#3339](https://github.com/syslog-ng/syslog-ng/pull/3339))
+ * `stats-query`: speedup `syslog-ng-ctl query get "*"` command.
+
+   An algorithmic error view made `syslog-ng-ctl query get "*"` very slow with large number of counters.
+   ([#3376](https://github.com/syslog-ng/syslog-ng/pull/3376))
+ * syslogformat: fixing crashing with small invalid formatted logs see example in #3328
+   ([#3364](https://github.com/syslog-ng/syslog-ng/pull/3364))
+ * `cfg`: fix config reload crash via introducing `on_config_inited` in LogPipe
+   ([#3176](https://github.com/syslog-ng/syslog-ng/pull/3176))
+ * config: fix error reporting
+
+   - Error reporting was fixed for lines longer than 1024 characters.
+   - The location of the error was incorrectly reported in some cases.
+   ([#3383](https://github.com/syslog-ng/syslog-ng/pull/3383))
+ * `disk queue`: fix possible crash during load, and possible false positive corruption detection
+   ([#3342](https://github.com/syslog-ng/syslog-ng/pull/3342))
+ * db-parser, pdbtool, graphite-output: fix glib assertion error
+
+   The assertion happened in these cases
+   * dbparser database load
+   * argument parsing in graphite-output
+   * pdbtool merge commad
+
+   Syslog-ng emitted a glib assertion warning in the cases above, even in successful executions.
+
+   If `G_DEBUG=fatal-warnings` environment variable was used, the warning turned into a crash.
+   ([#3344](https://github.com/syslog-ng/syslog-ng/pull/3344))
+ * stats: fix stats-ctl query crash when trying to reset all the counters
+   `syslog-ng-ctl query get '*' --reset`
+   ([#3361](https://github.com/syslog-ng/syslog-ng/pull/3361))
+
+## Packaging
+
+ * RHEL 7 packaging: fix logrotate file conflict with rsyslog
+   ([#3324](https://github.com/syslog-ng/syslog-ng/pull/3324))
+ * Debian packaging: python3-nose was removed from package dependencies.
+   Pytest will run Python related unittests (for modules/python/pylib/syslogng/debuggercli/tests/)
+   instead of nose.
+   ([#3343](https://github.com/syslog-ng/syslog-ng/pull/3343))
+
+## Notes to developers
+
+ * light: test for assertion errors in glib for each testcases
+   ([#3344](https://github.com/syslog-ng/syslog-ng/pull/3344))
+ * Fix signal handling when an external library/plugin sets SIG_IGN
+
+   Previously, setting SIG_IGN in a plugin/library (for example, in a Python module) resulted in a crash.
+   ([#3338](https://github.com/syslog-ng/syslog-ng/pull/3338))
+ * `func-test`: removed logstore_reader check, which was never reached
+   ([#3236](https://github.com/syslog-ng/syslog-ng/pull/3236))
+ * `plugin_skeleton_creator`: fixing a compiler switch
+
+   Wrong compiler switch used in `plugin_skeleton_creator`. This caused a compiler warning. The grammar debug info did not appear for that module, when `-y` command line option was used.
+   ([#3339](https://github.com/syslog-ng/syslog-ng/pull/3339))
+ * Light test framework: get_stats and get_query functions to DestinationDriver class
+
+   Two new functions added to DestinationDriver class which can be used for getting the stats
+   and query output of syslog-ng-ctl.
+   ([#3211](https://github.com/syslog-ng/syslog-ng/pull/3211))
 
 ## Other changes
 
- * `dbld`: Fedora 32 support ([#3315](https://github.com/syslog-ng/syslog-ng/pull/3315))
- * `dbld`: Removed Ubuntu Eoan ([#3313](https://github.com/syslog-ng/syslog-ng/pull/3313))
- * `secure-logging`: improvements ([#3284](https://github.com/syslog-ng/syslog-ng/pull/3284))
-    - removed 1500 message length limitation
-    - `slogimport` has been renamed to `slogencrypt`
-    - `$(slog)` will not start anymore when key is not found
-    - internal messaging (warning, debug) improvements
-    - improved memory handling and error information display
-    - CMake build improvements
-    - switched to GLib command line argument parsing
-    - the output of `slogkey -s` is now parsable
-    - manpage improvements
+ * `internal()`:  limit the size of internal()'s temporary queue
 
-## Notes to developers
+   The `internal()` source uses a temporary queue to buffer messages.
+   From now on, the queue has a maximum capacity, the `log-fifo-size()` option
+   can be used to change the default limit (10000).
+
+   This change prevents consuming all the available memory in special rare cases.
+   ([#3229](https://github.com/syslog-ng/syslog-ng/pull/3229))
+ * network plugins: better timer defaults for TCP keepalive
 
- * `dbld`: devshell is now upgraded to Ubuntu Focal
-   ([#3277](https://github.com/syslog-ng/syslog-ng/pull/3277))
- * `dbld/devshell`: Multiple changes:
-    * Added snmptrapd package.
-    * Added support for both `python2` and `python3`.
-   ([#3222](https://github.com/syslog-ng/syslog-ng/pull/3222))
- * `threaded-source`: fully support default-priority() and default-facility()
-   ([#3304](https://github.com/syslog-ng/syslog-ng/pull/3304))
- * `CMake`: fix libcap detection
-   ([#3294](https://github.com/syslog-ng/syslog-ng/pull/3294))
- * Fix atomic_gssize_set() warning with new glib versions
-   ([#3286](https://github.com/syslog-ng/syslog-ng/pull/3286))
+   From now on, syslog-ng uses the following defaults for TCP keepalive:
+   - `tcp-keepalive-time()`: 60
+   - `tcp-keepalive-intvl()`: 10
+   - `tcp-keepalive-probes()`: 6
+
+   Note: `so-keepalive()` is enabled by default.
+   ([#3357](https://github.com/syslog-ng/syslog-ng/pull/3357))
 
 ## Credits
 
@@ -110,7 +150,4 @@ feedback are all important contributions, so please if you are a user
 of syslog-ng, contribute.
 
 We would like to thank the following people for their contribution:
-
-Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs,
-Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady,
-Péter Kókai, Vatsal Sisodiya, Vivin Peris.
+Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Christian Tramnitz, chunmeng, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, MileK, Norbert Takacs, Peter Czanik, Péter Kókai, Terez Nemes.

README.md

@@ -15,7 +15,7 @@ applications or forwarded by systemd) and writes everything to a single
 file:
 
 ```
-@version: 3.28
+@version: 3.29
 @include "scl.conf"
 
 log {
@@ -27,7 +27,7 @@ log {
 This one additionally processes logs from the network (TCP/514 by default):
 
 ```
-@version: 3.28
+@version: 3.29
 @include "scl.conf"
 
 log {
@@ -41,7 +41,7 @@ log {
 This config is designed for structured/application logging, using local submission via JSON, and outputting in key=value format:
 
 ```
-@version: 3.28
+@version: 3.29
 @include "scl.conf"
 
 log {

VERSION

@@ -1 +1 @@
-3.28.1
+3.29.1

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide the same behavior as OpenBSD's syslog.conf(5).
 # 2010-07-18 steven@openbsd.org
 
-@version: 3.28
+@version: 3.29
 
 options {
 	use_dns(no);

doc/man/dqtool.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -95,7 +95,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.28 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.29 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -258,7 +258,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.28 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.29 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -445,7 +445,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.28 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.29 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -8,7 +8,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">3.28</refmiscinfo>
+            <refmiscinfo class="version">3.29</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -108,7 +108,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.28 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  3.29 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -168,7 +168,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 3.28
+@version: 3.29
 @include "scl.conf"
 
 source s_local {

doc/man/slogencrypt.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>slogencrypt</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>

doc/man/slogkey.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>slogkey</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>

doc/man/slogverify.1.xml

@@ -8,7 +8,7 @@
     <refmeta>
       <refentrytitle>slogverify</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">3.28</refmiscinfo>
+      <refmiscinfo class="version">3.29</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>