syslog-ng-3.16.1

@lbudai lbudai released this Jun 18, 2018 · 163 commits to master since this release

3.16.1

Features

  • Telegram destination and $(urlencode) template function (#2085)
  • Error reporting on misspelled block args (#1952)
  • New ignore_tns_config Oracle SQL destination config option (#2054)
  • Per-source "src.host" and "src.sender" counters (#2013)

Bugfixes

  • Fix possible loss of log messages in the systemd-journal() source (#1570, #1587, #1612)
  • Fix file source location information in internal logs (#2028)
  • Fix SDATA deserialization (disk-buffer crash) (#1919, #2067)
  • Fix unaccepted embedded 'file' keyword (file source and destination) (#2076)
  • Fix memory leaks in appmodel and varargs (#2086)
  • Fix a bug in the old LogMessage deserialization (#2103)
  • Fix reading the output of the confgen program (#1780, #2108)
  • Add safer mem_zero() to secret-storage (#2049)
  • Fix undefined behavior in syslog-ng-ctl query (#2043)
  • Fix lloc tracking for multi line blockrefs (#2035)
  • Added missing 'else {};' to default-network-drivers() to forward unparsable messages (#2027)
  • Fix mixed linking (#2020, #2022)
  • Fix compilation of evtlog on FreeBSD (#2014, #2015)
  • Fix thread_id allocation for more than 32 CPUs (crash) (#2008)
  • Add safe logging of errno (#1990, #1999)
  • Fix warnings related to floating point operations (#1959, #1996)
  • Partial revert of plugin discovery to bring back valgrind (#1953, #1995)
  • Fix connection close in network sources (#1991)
  • Fix file deletion in the wildcard-file() source (#1974)
  • Disable the DNS cache if use-dns(no) is used (#1923)
  • Fix compiler error for gcc 4.4 (#2044)
  • Fix emitted warnings due to -no-pie detection for gcc 4.4 (#2037)
  • Fix date format in functional tests (#2036)
  • Dbld fixes (#2034)
  • Rename PAGESIZE variables to pagesize in secret-storage (compilation fix) (#2111)
  • Fix the lifetime of TLSContext to prevent crash on reload (#2080, #2109)
  • Fix reaping program() source and destination when a Java-based destination is used (#2099)

Other changes

  • Add debug message to program source/destination about successful start (#2046)
  • Report memory exhaustion errors during config parsing (#2033)
  • Improved debug logs (#2032)
  • Dbld coverage (#2031)
  • LogTransportMock enhancement (#2017)
  • Modify the license of loggen from GPL to LGPL (#2006)
  • Loggen refactor (#1987)
  • Update RPM generation (#1980, #2113)
  • Support ENABLE_EXTRA_WARNINGS with CMake (#2072)
  • Rewrite unit tests based on Criterion (#2026, #2058, #2039, #2018, #2003)
  • Lexer test coverage improvements (#2062)
  • preparation for 3.16 OSE rhel/packaging (#2113)

Notes to the developers

  • Do not ship rabbitmq-c with syslog-ng (remove as submodule) (#2052)
  • Multitransport (#2057)
  • Timeout support for LogReader and LogWriter (#2056)
  • Update ivykis to 0.42.3 (submodule) (#2012)
  • Drop explicit version numbers from requirements.txt (#2050)
  • CMake modernization pt. 1 (#2007)
  • Assert when log_pipe_clone() is not implemented (#2019)
  • Small Java code refactor (#2066)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Andrej Valek, Antal Nemes, Balazs Scheidler, Gabor Nagy,
Gergely Nagy, German Service Network, Jakub Wilk, Laszlo Budai, Laszlo Szemere,
Laszlo Varady, Mehul Prajapati, Norbert Takacs, Peter Czanik, Peter Kokai,
Tomasz Kazimierczak, Viktor Juhasz

syslog-ng-3.15.1

@lbudai lbudai released this May 2, 2018 · 470 commits to master since this release

3.15.1

Features

  • Support added for if/elif/else blocks to the configuration file syntax.
    (#1856)
  • Dramatically improved debug messages during filter/parser evaluation. (#1898)
  • Similarly improved the error messages shown on syntax errors, they now show a
    full backtrace of inclusions, among other things. (#1932)
  • The hook-commands module was added, allowing one to run custom commands on
    source or destination setup and teardown. (#1951)
  • Implemented a way to skip processing included config file snippets in case a
    dependency is missing: The @requires json pragma. (#827, #1956)
  • Basic client-side failover support was implemented. (#1905)
  • Errors from python destinations are now reported together with any exception
    text (if any). (#1931)
  • add-contextual-data gained a new ignore-case() option. (#1911)

Bugfixes

  • Fix a crash that happened on disk queue restart. (#1886)
  • Fixed another crash when a corrupted disk queue file was being moved away.
    (#1924)
  • Fixed a crash that could happen during nvtable deserialization. (#1967)
  • Fixed a crash that occurred when NVTables were stored on low memory
    addresses. (#1970)
  • Fixed an issue with TLS session resumption, the session id context value is
    now properly set. (#1936, #2000)
  • We now link directly to the evtlog shipped with syslog-ng, and are not
    using the system library, not even when present. (#1915)
  • TLS destinations now work again without key-file or cert-file specified.
    (#1916, #1917)
  • SDATA block names are now sanitized, in order to not break the spec when we
    get our SDATA from sources that are more lax (such as JSON). (#1948)
  • Some internal messages contained key-value pairs where the key had spaces in
    it, this has been addressed, they do not contain spaces anymore.
  • The STOMP destination will now correctly use template options when formatting
    its body part. (#1957)
  • Fix compilation with OpenSSL 1.1.0 (#1921, #1997)
  • Fix compilation on FreeBSD. (#1901)
  • Fix compilation on SLES 11. (#1897)
  • Fix compilation on Hurd. (#1912, #1914)
  • Fix compiltaion on Solaris 10. (#1982, #1983)
  • Fix compilation on MacOS.
  • Fixed a value conflict in the afstreams module's grammar file.
  • Various compiler warning-related fixes all over the codebase.

Other changes

  • POSIX RegExp support was dropped from the filters, PCRE remains available. (#1899)
  • Miscellaneous build-system related fixes and improvements (both autotools and
    CMake).
  • Update lib/json-c to json-c-0.13-20171207. (#1900)

Notes to the developers

  • The init() function is now optional for Python destinations. (#1756)
  • The Docker environment (dbld/) has seen significant changes, among them an
    upgrade to Ubuntu Xenial. (#1876)
  • dbld/rules gained two new targets: login and build, that do what their
    names suggest. (#1927)
  • The LogPipe object gained a pre_init() and a post_deinit() method, used
    by the hook-commands module.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Balazs Scheidler, Budai Laszlo, Gabor Nagy, Gábor
Nagy, Gergely Nagy, Juhasz Viktor, Kókai Péter, Laszlo Budai, László Szemere,
László Várady, Mehul Prajapati, Norbert Takacs, Robert Fekete, SZALAY Attila,
Tamas Nagy, Terez Nemes, Utsav Krishnan, Videet Singhai, Vivek Raj

syslog-ng-3.14

@lbudai lbudai released this Feb 27, 2018 · 770 commits to master since this release

3.14.1

Features

  • Password protected ssl keys (#1888)
  • Add OpenBSD module to system() source (#1875)
  • Add Ubuntu Trusty support to Docker build (#1849)

Bugfixes

  • Fix increased memory usage during saving disk-buffer (#1867)
  • Fix maximum record length limitations of disk-buffer (#1874)
  • Fix a memory leak in cfg-lexer (#1843)
  • Fix some issues found by pylint in python module (#1881, #1830)
  • Fix a crash due to a race condition in kv-parser() (#1871)
  • Fix a crash due to a race condition in file() destination (#1858)
  • Fix deprecated API usage in python module tests (#1829)
  • Fix a race condition in internal() source (#1815)
  • Fix a locale issue in merge-grammar python tool (#1868)
  • Fix compile problems with autotools when '--disable-all-modules' used (#1853)
  • Fix a file descriptor leak in persist-state (#1847)
  • Fix a file descriptor leak in pseudofile() (#1846)
  • Fix memory/fd leaks in loggen tool (#1844, #1845)
  • Fix compile problems on Fedora, RHEL6, CentOS6 and SUSE based platforms (#1837)
  • Fix a crash when large variety of keys added to messages (#1836)
  • Fix compile problems when PATH_MAX not defined (#1828)
  • Fix integer overflow problems in grammar (#1823)
  • Fix a memory leak in filter() (#1812)
  • Fix memory leak of persist-name() option (#1816)
  • Fix message corruption caused by a bug in the subst() rewrite rule (#1801)
  • Fix silently dropped messages in elasticsearch2() when sending in bulk mode (#1800)
  • Fix broken disk-buffer() support in elasticsearch2() (#1807)
  • Fix Hy support in python module (#1754)
  • Fix an event scheduler related crash during reloading syslog-ng (#1711)
  • Fix a crash with SIGBUS when persist file cannot grow (#1785)

Other changes

  • Improve error reporting in "block" definitions in config (#1809)
  • Add warning message when disk-buffer() directory is changed in configuration (#1861)
  • Syslog-ng debun improvements (#1840)
  • Refactor in rewrite() module init (#1818)
  • Missing child program (exit status 127) handling is changed in program() destination:
    stopping destination instead of polling for the child program (#1817)
  • Refactor in filter() module (#1814)
  • Improve thread synchronization in mainloop and refactor (#1813)
  • Adapted json-c v0.13 API changes to json-parser (#1810)
  • Add filters as selectors in contextual data (#1838)

Notes to the developers

  • Full cmake support achieved (#1777, #1819, #1811, #1808, #1805, #1802, #1841, #1806)
  • Add support for modules to have module specific global options (#1885)
  • Improved MacOS support (#1862, #1864, #1865)
  • Add new option to exclude directories in style-checker tool (#1834)
  • Ivykis dependency updated to 0.42.2 release (#1711)
  • Journald grammar, source and header files are part of dist tarball (#1852)
  • Add valgrind support for unit tests (#1839)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Balazs Scheidler, Björn Esser, Fabien Wernli, Gabor Nagy, Gergely Nagy,
Janos Szigetvari, Juhász Viktor, Laszlo Budai, Laszlo Szemere, László Várady, Orion Poplawski,
Attila Szalay, Shen-Ta Hsieh, Tamas Nagy, Peter Kokai, Norbert Takacs, Zoltan Pallagi.

syslog-ng-3.13.2

@lbudai lbudai released this Dec 6, 2017 · 1089 commits to master since this release

3.13.2

Fixes

  • Missing manpages from release tarball (#1793)
  • Package syslog-ng-mod-json is removed from (#1794)
  • Drop syslog-ng-abi virtual packages (#1797)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Gergely Nagy, Laszlo Budai, Laszlo Varady, Peter Czanik.

syslog-ng-3.13.1

@MrAnno MrAnno released this Dec 4, 2017 · 1102 commits to master since this release

3.13.1

Features

  • Add app-parser() framework (automatic parsing of log messages) (#1689)
  • Support microseconds in Riemann destination (#1710)
  • Add osquery destination as an SCL plugin (#1728)
  • Add network load balancer destination (#1706)
  • Add possibility to only signal re-open of file handles (SIGUSR1) (#1530)
  • It is possible from now to limit the number of registered dynamic counters (#1743)
  • Add $(binary) template function (#1679)
  • Add experimental transport for transferring messages in whole between syslog-ng instances (EWMM) (#1689)
  • Docker based build and debian package generation (#1783)
  • Add auto-parse(yes/no) to app-paser(), system() and default-network-drivers() (#1788)
  • Add Graylog2 destination and $(format-gelf) template function (#1680)

Bugfixes

  • Exit when a read fails on an included config file instead of
    starting up with an empty configuration. (#1721)
  • Fix double free (#1720)
  • Add missing discarded counter to groupingby (#1748)
  • Fix a reference leak in Python destination (#1716)
  • Fix timezone issue in snmptrapd parser (#1746)
  • Fix potential crash in stdin driver (#1741)
  • Fix a crash when initializing new config fails for socket with keep_alive off (#1723)
  • Fix filter evaluation in case of contexts with multiple elements (#1718)
  • Various grouping-by fixes (#1718)
  • Fix potential use after free around dns-cache during shutdown (#1666)
  • Fix access to indirect values within Java destination (#1732)
  • Fix a crash in affile (#1725)
  • Fix a memory leak (#1724)
  • Fix a crash when getent is used empty group (#1691)
  • Fix jvm-options() (#1704)
  • Fix a crash in Python language binding (#1694)
  • Fix a crash in afmongodb (#1765)
  • Fix a memory leak in afmongodb (#1766)
  • Fix name-to-GID calculation in the $(getent) template function (#1764)
  • Fix a crash when redis is configured without the command() option (#1767)
  • Fix a race condition in kv-parser() (#1789)

Other changes

  • Cleanup diskq related warning messages (#1752)
  • Provide tls block for tls options in amqp(), http(), riemann() destination drivers (#1715)
  • It it possible from now to register blocks and generators as plugins (#1657)
  • Drop compatiblity with configurations below 3.0 (#1709)
  • Do not change permissions of a file by default (#1782)
  • Allow source files to specify permissions locally (#1782)
  • Minor performance improvement (#1729)
  • The current config version can be queried with "--version" (#1740)
  • Increase the performance of kv-parser() (#1789)

Notes to the developers

  • Change configure default option for jsonc and mongoc from auto to internal (#1735)
  • Disable ASLR when running unit tests (#1753)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy,
Jakub Jankowski, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Laszlo Szemere,
Marton Illes, Mate Farkas, Peter Kokai, Pontus Andersson, Sam Stephenson,
Sebastian Roland, Viktor Juhasz, Zoltan Pallagi.

syslog-ng-3.12.1

@lbudai lbudai released this Sep 25, 2017 · 1330 commits to master since this release

3.12.1

Features

  • HDFS: support macro in filename (#1638)
  • HDFS: add append support (#1675)
  • Java: allow to use sequence numbers in templates (#1628)
  • TLS improvements (#1603, #1636)
    • Add PKCS 12 support with the new pkcs12-file() TLS option
    • startup time ssl-options() and peer-verify() check
    • startup time key_file, cert_file, ca_dir, crl_dir and cipher_suite check
    • ECDH cipher support (OpenSSL 1.0.1, 1.0.2, 1.1.0) with the ecdh-curve-list() option (only available >= 1.0.2)
      • for < 1.0.2, a hard-coded curve is used
      • for >= 1.0.2, automatic curve selection is used (the ecdh-curve-list() option can restrict this list)
    • DH cipher support with the dhparam-file() option
      • if the option is not specified, fallback RFC 3526 parameters are used
    • minor fixes
  • stdin() source driver (#1605)
  • Implement read_old_records option for systemd-journal source (#1642)
  • Add tags-parser: a new module to parse $TAGS values (#1658)
  • Add a Windows eventlog parser scl module (#1572)
  • Add XML parser module (#1659, #1684)

Bugfixes

  • Fix cannot parse ipv6 into hostname (#1617)
  • Speedup add-contextual-data by making ordering optional (#1645)
  • Fix monitor-method() option not working for wildcard-file() source (#1651)
  • Sanitize SDATA keys in syslog-protocol messages to avoid generating non-valid messages (#1650, #1654)
  • Fix memory leaks reported using Valgrind (#1649)
  • Fix memory leak related to cloning pipes and reload (#1647)
  • Fix getent protocol number returns incorrect value (#1665)
  • Fix elasticsearch2 destination flush mechanism (#1668)
  • Fix file destination related memory leak (#1685)
  • Fix a possible memory leak around affile destination (#1685)

Other changes

  • Improve syslog-ng debun functionality (#1633, #1641, #1663)
  • Java: allow to set JVM options form global syslog-ng options (#1639)
  • Do steps towards Python 3 support:
    • Fix string compatibility for Python 3 (#1632)
    • Improve Python version auto detection (#1660)
  • HTTP destination: display verbose logs on debug level (#1526)
  • Improvements for Solaris packing (#1664)

Notes to the Developers

  • Update internal RabbitMQ (#1662)
  • Update internal ivykis to v0.42 (#1566)
  • Fix Travis and test related issues (#1566, #1644, #1674)
  • Update docker images (#1637)
  • Fix some clang compile errors (#1662)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy,
Gergely Orosz, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Mate Farkas,
Marton Suranyi, Peter Kokai, Szilard Pfeiffer, Tamas Nagy, Zoltan Pallagi.

syslog-ng-3.11.1

@lbudai lbudai released this Aug 1, 2017 · 1516 commits to master since this release

3.11.1

Features

  • Add geoip2 parser and template function.
    It is based on the libmaxminddb(MaxMindDB).
    It will replace the old geoip parser and template function,
    so they are deprecated from 3.11 (but still available).

  • Add SSL support to AMQP.

  • Add template option to apache-accesslog-parser.

  • Add configurable event time to Riemann destination.

  • Add drop-unmatched() option to dbparser.

  • Add Ubuntu Xenial to the bundled docker images.

  • Support multi-instance support for Solaris 10 and 11.

  • Support multi-instance for systemd.

  • Add configurable timeout to HTTP destination.

  • Add prefix() option to cisco-parser.

Bugfixes

  • Fix a memory usage counter underflow for threaded destination drivers
    and writers.

  • Fix a potential crash in AMQP.

  • Fix a potential crash during reload.

  • Fix a reload/shutdown issue.
    Under heavy load, worker might never exit from the fetch loop from the
    queue.

  • Fix a potential crash in afsocket destination during reload.

  • Fix a counter registration bug.
    In some cases not all the required counters are registered.

  • Fix a build issue on FreeBSD.

  • Fix a memory leak in diskq plugin.

  • Fix systemd-journal error codes validation.

  • Fix a potential crash in diskq when it is used with file
    destination and the file is reaped.

  • Fix a memory leak in HTTP destination

  • Fix ENABLE_DEBUG in dbparser.

  • Fix a unit tests that caused build issue on 32 bit platforms.

Other changes

  • The eventlog library is part of syslog-ng from now.

  • Improve error messages when the config cannot be initialized.

  • Improve source suspended/resumed debug messages.

  • Rename syslog-debun to syslog-ng-debun.

  • Update manpages to v3.11

  • Remove tgz2build directory.

Notes to the Developers

  • Rewrite merge-grammar script from Perl to Python.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Fabien Wernli,
Gabor Nagy, Giuseppe D'Anna, Janos Szigetvari, Laszlo Budai, Laszlo Varady,
Lorand Muzamel, Mate Farkas, Noemi Vanyi, Peter Czanik, Tamas Nagy,
Tibor Bodnar, Tomasz Kazimierczak, Zoltan Pallagi

syslog-ng-3.10.1

@lbudai lbudai released this Jun 19, 2017 · 1702 commits to master since this release

3.10.1

Features

  • Support https in http (curl) module

  • Docker support : from now Dockerfile for CentOS7, Ubuntu Zesty and for
    Debian Jessie is part of our upstream

  • Add --database parameter for geoip template function

  • Metric improvements

    • add discarded messages for parsers
    • add matched/not matched counter for filters
    • add memory_usage counter to logqueue
    • add written counter
      • Written is a calculated counter which return the written messages
        by destinations. Written message is which was processed but not
        queued and not dropped. (written = processed - queued - dropped)
    • stats-counters: rename stored counter to queued
    • add global_allocated_logmsg_size counter for tracking memory logmsg
      related allocations
  • Add snmp-parser (v1, v2)

    • parses snmptrapd log
    • The parsed information is available as key-value pairs, which can be
      used/serialized (macros, format-json, etc.) in the log path.
      If you want to send the message in a structured way, you can disable the
      default message generation with the generate-message(no) option.
  • Add snmp-soure

    • available as an SCL block that containing a filesource and an SNMP parser
      modules: add snmptrapd parser
  • Add osquery source

    • available as an SCL block
    • It reads the osquery log file and parses with the JSON parser,
      creating name-vaule pairs with an .osquery. prefix by default.
  • Add cisco-parser

    • available as an SCL block
  • Add wildcard filesource

  • Add startdate template function

  • Add $(basename) and $(dirname) template functions

  • Add Kerberos support for HDFS destination

  • Add AUTH support for redis destination

  • Add map-value-pairs() parser

    • it can be used to map existing name-value pairs to a different set during
      processing, in bulk. Normal value-pairs expressions can be used, just
      like with value-pairs based destinations.
  • Extend Python language binding by Python parser

  • Add support for extract-stray-words() option in kv-parser()

    • stray words: those words that happen to be between key-value pairs and
      are otherwise not recognized either as keys nor as values.
  • Add $(context-values) template function

  • Add $(context-lookup) function

  • Add list related template functions

    • $(list-head list ...) returns the first element (unquoted)
    • $(list-nth NDX list ...) returns the specific element (unquoted)
    • $(list-concat list1 list2 ...) returns a list containing the concatenated
      list
    • $(list-append list elem1 elem2 ...) returns a list, appending elem1,
      elem2 ...
    • $(list-tail list ...) returns a list containing everything except
      for the first element
    • $(list-slice FROM:TO list ...) returns a list containing the slice
      [FROM:TO), Python style slice
      boundaries are supported (e.g. negative)
    • $(list-count ...) returns the number of elements in list
  • Add add query commands to syslog-ng-ctl

    • query list List names of counters which match the filter
    • query get Get names and values of counters which match the
      filter
    • query get --sum Get the sum of values of counters which match the
      filter
  • Support multiple servers in elasticsearch2-http destination

  • Implements elastic-v2 https in http mode

  • Add getent module (ported from incubator)

    • This module adds $(getent) that allows one to look up various NSS based
      databases, such as passwd, services or protocols.
  • Add support for IP_FREEBIND

Bugfixes

  • Fix a libnet detection check error that caused problem configuring
    enable-spoof-source.

  • Avoid warnings about _DEFAULT_SOURCE on recent glibc versions
    With the glibc on zesty, using _GNU_SOURCE and not defining _DEFAULT_SOURCE
    results in a warning, avoid that by defining _DEFAULT_SOURCE as well.

  • Fix invalid database warning for geoip parser

  • Fix prefix() default in systemd-journal for new config versions

  • Fix a potential message loss in Riemann destination

  • Fix a potential crash in the Riemann destination when the client is not
    connected to the Riemann server.

  • Fix a possible add-contextual-data() related data loss in case of multiple
    reference to the same add-contextual-data parser in several logpaths.

  • Fix dbparser deadlock

  • Fix Python destination

    • open() was not called in every time_reopen()
    • python destination is not defined in stats output
  • Fix processed stats counter for afsocket

  • Fix stats source for pipes

    • Previously pipe source is shown as file
  • Fix csv-parser multithreaded support
    In some cases (when csv-parser attached to network source), the parser
    randomly filled the column macros with garbage.

  • Fix a message loss in case of filesource when syslog-ng was restarted and
    the log_msg_size > file size.

  • Fix a potential crash in cryptofuncs

  • Fix a potential crash in syslog-ng-ctl when no command line parameters was
    set.

  • Fix token duplication in the output of '--preprocess-into'

  • Fix UTF-8 support in syslog-ng-ctl

  • Fix a potential crash during X.509 certificate validation.

  • Fix a segfault in Python module startup

  • Fix a possible endless reading loop issue in case of multi-line filesource.

  • Fix soname for the http module from "curl" to "http"

  • Avoid openssl 1.1.0 deprecated APIs
    When openssl is built with --api=1.1 disable-deprecated, use of deprecated
    APIs results in build failure.

Other changes

  • Increase processed counter by queued counter after reload or restart when
    diskqueue is used otherwise the newly added written counter would underflow.

  • Set the default time-zone to UTC for elasticsearch2
    Elasticsearch and Kibana use UTC internally.

  • Add retries support for python destination

  • Prefer server side cipher suite order

  • Always include librabbitmq in the dist tarball

  • Always include ivykis in the dist tarball

  • Marking parse error locations with >@<.

  • Default log_msg_size is increased to 64Kbyte from 8Kb

  • Tons of syslog-debun improvements

  • Exit with 0 return code when --help is specified for syslog-ng-ctl

  • syslog-ng: make '--preprocess-into' foreground only

  • Add debug messages on log_msg_set_value()

  • Add more detail to filter evaluation related debug messages

Notes to the Developers

  • Extract template perf test function to testlib

  • Print a debug message when logmsg passed to the Python side

  • Allow http module (curl) to be build with cmake

  • astylerc: allow continuation lines to start until column 60

  • Move kv-scanner under syslog-ng/lib

  • scratch-buffers2: implement an alternative to current scratch buffers
    This new API is aimed a bit easier to use in situations where a throw away
    buffer is needed that will automatically be freed at the next message.
    It also gets does away with GTrashStack that is deprecated in recent glib
    versions.

  • Several refactors in stats module.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Balazs Scheidler, eroen, Fabien Wernli, Gabor Nagy,
Gergely Nagy, Janos Szigetvari, Jason Hensley, Laszlo Varady, Laszlo Budai, Mate Farkas,
Noemi Vanyi, Peter Czanik, Peter Gervai, Todd C. Miller, Philip Prindeville,
Zoltan Pallagi

syslog-ng-3.9.1

@lbudai lbudai released this Dec 21, 2016 · 2319 commits to master since this release

3.9.1

Features

  • Improve parsing performance in case of keep-timestamp(no), earlier the
    timestamp was parsed and then dropped, now we don't parse it, which is a
    2x performance improvement in reception speed.

  • TLS based transports will publish the peer's certificate in a set of
    name-value pairs, as follows:

    • .tls.x509_cn - X.509 common name
    • .tls.x509_o - X.509 organization string
    • .tls.x509_ou - X.509 organizational unit
  • Improve performance of the tcp() source, due to a bug, syslog-ng
    attempted to apply position tracking to messages coming over a TCP
    transport, which is used for file position tracking and causing
    performance degradation. This bug is fixed, causing performance to be
    increased. (#1195)

  • Make it possible to configure the listen-backlog() for any stream based
    transports (unix-stream and tcp). Earlier this was hard-wired at 256
    connections, now can be tuned using an option. For example:

    tcp(port(6514) listen-backlog(2048));

  • Add a groupunset() rewrite rule that pairs up with groupset() but instead
    of setting values it unsets them. (#1235)

  • Add support for Elastic Shield (#1228) and SearchGuard (#1223)

  • kv-parser() is now able to cope with unquoted values with an embedded
    space in them, it also trims whitespace from keys/values and is in
    general more reliable in extracting key-value pairs from arbitrary log
    messages.

  • Improve performance for java based destinations. (#1243)

  • Add prefix() option to add-contextual-data()

Bugfixes

  • Fix a potential crash in the file destination, in case it is a template
    based filename and time-reap() is elapsed. (#1183)

  • Fix a potential ACK problem within syslog-ng that can cause input windows
    to overflow queue sizes over time, effectively causing message drops that
    shouldn't occur. (#1230)

  • Fix a heap corruption bug in the DNS cache, in case the maximum number of
    DNS cache entries is reached. (#1218)

  • Fix timestamp for suppression messages. (#1233)

  • Fix add-contextual-data() to support CRLF line endings in its CSV input
    files.

  • Fixed key() option parsing in riemann() destinations.

  • Find libsystemd-journal related functions in both libsystemd-journal.so
    and libsystemd.so, as recent systemd versions bundled all systemd
    related libs into the same library.

  • Fixed the build-time detection of system-wide installed librabbitmq,
    libmongoc and libcap.

  • Fix the file source to repeatedly check for unexisting files, as a bug
    caused syslog-ng to stop after two attempts previously. (#841)

  • The performance testing tool "loggen" crashed if it was used to generate
    messages on multiple threads over TLS. This was now fixed. (#1182)

  • Fix an issue in the syslog-parser() parser, so that timestamps parsed
    earlier in the log path are properly overwritten. Earlier a time-zone
    setting may have remained in the timestamp in case the first timestamp
    did contain a timezone and then the one parsed by syslog-parser() didn't.
    (#1206)

  • Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and
    tcp-keepalive-probes() were not working, now they are again. (#1214)

  • The --disable-shm-counters option is now passed to mongo-c-driver to work
    around a minor security issue (#1219).

    https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/1405406

  • Fix compilation issues on FreeBSD. (#1252)

  • Add support to month names in all caps in syslog timestamps. At least one
    device seems to generate these. (#1263)

  • The options() option to java destination can now accept numbers and not
    just strings.

  • Fix a memory leak in the java destination driver, that may affect java
    based destinations like ElasticSearch, Kafka & HDFS.

Other changes

  • HDFS was updated to 2.7.3
  • Elasticsearch was updated to 2.4.0
  • Support was added for OpenSSL 1.1.x (#1281)

Notes to the Developers

  • We started to standardize our tests on the criterion unit testing
    framework, please submit all new tests using this framework. Patches to
    convert existing ones are also welcome.
    https://github.com/Snaipe/Criterion
  • We also added a configuration file for astyle and accompanying make
    targets to check/reformat the source code to meet syslog-ng's style.
  • debian/ directory has been removed from the "master" branch and is now
    maintained in a separate "release" branch.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai,
Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi,
Attila Szalay, Tibor Bodnár, Zoltán Pallagi

syslog-ng-3.8.1

@lbudai lbudai released this Aug 19, 2016 · 2564 commits to master since this release

3.8.1

Library updates

  • Kafka-client updated to version to 0.9.0.0
  • Minimal required version of hiredis is set to 0.11.0 to avoid
    possible deadlocks
  • Minimal version of libdbi is set to 0.9.0

New dependencies

  • From now autoconf-archive package is a build-dependency.

Improvements and features

  • Added the long-waited disk-buffer.
  • date-parser ported from incubator to upstream
  • New template functions: min, max, sum, average
  • Added Apache-accesslog-parser
  • Added loggly destination
  • Added logmatic destination
  • Added template function for supporting CEF.
  • cURL-based HTTP destination driver added (implemented in C
    programming language)
  • SELinux policy installer script now has support for Red Hat Enterprise Linux/CentOS/
    Oracle Linux 5, 6 and 7.
  • Implemented add-contextual-data:
    With add-context-data syslog-ng can use an external database file to append
    custom name-value pairs on incoming logs (to enrich messages). The 'database'
    is actually a file that containing <selector, name, value> records.
    Currently only CSV format is supported.
    It is like geoip parser where the selector is $HOST, but in this case,
    the user can define the selector, and also the database contents.

Drivers

  • Program destination/source drivers
    • Added inherit-environment configuration option to program source and
      destination. When it is set to true then the process will inherit the
      entire environment of the parent process.
    • Added keep-alive option to program destination (afprog).
      This option will control whether the destination program should be
      terminated at reload, or should be left running.
  • Java drivers
    • HTTP destination
      • Added the ability to use templates in both url and message.
    • ElasticSearch Destination driver :
      • Support 2.2.x series of ElasticSearch (transport and node mode) .
      • Support Shield plugin for both ElasticSearch 1.x and ElasticSearch
        2.x .
      • Implemented new mode (HTTP) that can work with ElasticSearch 1.x,
        ElasticSearch 2.x, and even with Elastic 5. HTTP mode is based on
        a Java HTTP Rest client (Jest : https://github.com/searchbox-io/Jest).
        Note: make install will copy Jest library to the syslog-ng install
        directory.
  • MongoDB destination driver
    • Replaced submodule limongo-client with mongo-c-driver.
    • Additional support for previous syntax used by libmongo-client
      before we started using mongo-c-driver and its URI syntax exclusively.
      Note that these are plainly translated to a connection URI without
      much sanity checking or preserving their former semantic meaning.
      So various aspects of the MongoDB connection like health checks, retries,
      error reporting and synchronicity will still follow the slightly altered
      semantics of mongo-c-driver.
  • Riemann destination driver
    • Use cert-file() and key-file() options to match afsocket keywords as
      the same way as afsocket drivers use these options. The old one still
      work though.

Rewrite rules

  • Introduced template options in rewrite rules.
  • Added unset operation to make it possible to unset a specific name-value pair
    for a logmessage.

Parsers

  • kvformat: make it possible to specify name-value separator
  • linux-audit-scanner: recognize a0-a9* as fields to be decoded
    Argument lists are encoded in a0, a1, ... fields that can potentially be
    hex-encoded.
  • csv-parser has been refactored, extended with new dialect and prefix options.
    Dialect is to convey CSV format information, instead of using flags
    Prefix option gets prefixed to all column names, just like with
    other parsers.

PatternDB

  • added groupingby() parser that can perform simple correlation on
    log messages. In a way it is similar to the SQL GROUP BY operation, where
    an aggregate of a set of input records can be calculated.
    The major difference between SQL GROUP BY and groupingby() is that the
    first always operates on a enumerable list of records, whereas
    groupingby() works on a stream of data.
    A few use-cases where this can be useful:
    • Linux audit logs
    • postfix logs
  • added create-context action
    Added a new possible action in the element, to create
    a new correlation context out of the current message and its associated
    context. This can be used to "split" a state.
  • Added NLSTRING parser that captures a string until the
    following end-of-line. It can be used in patterns as: @nlstring:value@
    It doesn't expect any additional parameters. This makes it pretty easy to
    parse multi-line Windows logs.

Miscellaneous features

  • syslog-debun (debug bundle script for syslog-ng) has been improved

Bugfixes

  • geoip-parser: When default database if not specified, syslog-ng crashed.
  • Added support for multiple drivers with the same name in syslog-ng config.
  • Fixed aack counting logic for junctions that have branches that modify
    the LogMessage.
  • Fixed a potential crash for code that uses log_msg_clear()
    in production (e.g. syslog-parser()).
  • Fixed potential crash in reload logic
  • system(): use string comparison instead of numeric in PID rewrite
    The meaning of the != operator has been fixed to refer to numeric comparison
    in @Version: 3.8, so make sure we are using string comparison.
  • Support encoding on glib compiled with libiconv
  • pdbtool: Fix the ordering of the debug-info list in PatternDB
  • afprog: Don't kill our own process group
    If, for some reason, the program source/destination failed to set up its
    own process group, we need to make sure we do not run killpg() on that
    process group, as it would kill ourselves.
  • Handle option names with hyphen (-) characters in java scls
  • dnscache performance improved
    Instead of getting rid off the per-thread DNSCache when a worker thread
    exits, store them in a linked list and acquire them as a new thread starts.
    The set of cached hostnames are valuable as worker threads come and
    go (they exit after 10seconds of inactivity), but without this
    reusing of cache instances, our DNS cache is filled again and again.
  • Fixed IPv6 parser in patterndb.
  • Fixed journald program name flapping
  • Fixed create-dirs() inheritance in file destinations
  • Fixed pass-unix-credentials() global inheritance in afunix
    The global pass-unix-credentials option was not inherited in afunix-source
    if the options{}; block was positioned lower in the
    configuration file than the given module declaration.
  • Fixed create-dirs() global inheritance in afunix
    When the global create-dirs option was set to yes, the local one was ignored.
  • Fixed byteorder handling on bigendian systems in netmask6 filter
  • Fixed flow-control issue when overflow queue is full
    (suspending source by setting the window size to 0).
  • Log HTTP response error codes in HTTPDestination (Java).
  • Fixed potential leaks related $(sanitize) argument parsing in basicfuncs.
  • Fixed a memory leak in python debugger
  • Fixed a use-after-free bug in templates.
  • Fixed a memory leak around reload in netmask6 filter.
  • Fixed a memory leak in LogProtoBufferedServer in case the
    encoding() option is used.
  • configure: don't override $enable_python while executing pkg-config
  • Fixed BSD timestamp parsing in syslog-format.
  • Fixed a SIGPIPE bug in program destination.
  • Error handling has been improved in AMQP destination.
  • value-pairs performance improvements, memleak fixes
  • Various issues around UTF-8 support fixed.
  • Fixed integer overflow in numerical operations template function
  • Fixed an integer underflow in afsocket.
  • Fixed numerical comperisons issues around filters.
    There's a problem in straight fixing this issue though: anyone who used
    the numeric operators erroneously will have their behaviour changed, therefore
    this patch also adds a configuration update warning in case
    someone is using the wrong syntax.
  • Fixed kernel log message time drift on Linux.
  • Take CRLF sequences equivalent to an LF in patterndb.
    Windows logs contain embedded CRLFs which is difficult to match against
    from db-parser(), as we use a UNIX text file to store the patterns. Also,
    the fact that the input contains CRLF whereas our patterns only contain
    an LF makes it a very unintuitive non-match, which is difficult to debug.
  • When syslog-ng failed to insert data into Redis, it has crashed.
  • When device file is set as a file destination then syslog-ng will not try
    to change the permission of the device file.
  • Various fixes around config file parsing:
    • in some circumstances syslog-ng crashed when the config
      file contained non-readable characters
    • fixed a memory leak
    • fixed memory leak around backtick substitution

Notes to the Developers

  • copyright cleanup in source tree
  • install tools and scl under a syslog-ng specific subdirectory
    These should never be installed in /usr/share directly, but rather under a
    subdirectory and as described in
    https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
    we should do that right within the source and not rely on packaging tools
    to do it for us.This will trigger a required change in packaging scripts to
    avoid changing the --datadir, as the default of
    /usr/share should work out-of-the-box.
  • Support for native-lanugage (compiled languages, like Rust) bindings.
    These bindings just forward the calls to the native side.
    This whole module compiles into a static library
    (libsyslog-ng-native-connector.a) which is linked to all external native
    modules. A native module defines the required functions
    (like native_parser_proxy_new()) so those symbols will be resolved.
    Some symbols have the visibility(hidden) attribute applied to them. Those
    symbols are defined by the other half of the native bindings, we only need
    their signature here. They are hidden because their definition is contained
    in other source files but we would like to keep them as "library private".
    If they are exported, the dynamic linker will resolve them when a module is
    loaded, therefore every module would be mapped to the first loaded one.
    It is best to hide everything in this native connector.
  • Support system librabbitmq-c (AMQP destination)
    Currently only the internal librabbitmq-c is supported,
    if we want to use the preexisting library, the configuration
    will fail.
    This change is required if we want to get rid of the
    internal libraries.
  • Added check-valgrind target to Makefile
  • Remove dependency on Gradle for building Java language bindings
    (not modules, just the language binding)
  • Experimental CMake support added
  • Experimental OSX support added
  • Improved Travis build matrix
  • Added plugin skeleton creator.
  • Debian packaging improved: Debian packaging from unofficial
    OBS repository has been ported to upstream.
    From now, someone could easily build debian packages from upstream source.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Adam Istvan Mozes, Andras Mitzki, Arnaud Vamorec, Balazs Scheidler,
David Schweikert, Fabien Wernli, Flavio Medeiros, Hanno Böck,
Henrik Grindal Bakken, Gergo Nagy, Gyorgy Pasztor, Laszlo Budai,
Laszlo Varady, Marc Falzon, Noemi Vanyi, Peter Czanik, Robert Fekete,
Tamas Nagy,Tibor Benke, Viktor Juhasz, Viktor Tusa, Vincent Bernat,
Zdenek Styblik, Zoltan Fried, Zoltan Pallagi, Yilin Li