v0.44.0-beta2

What's Changed

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

The focus of this update is on maintenance to prepare future stable release

• Fixed support for IP fragments with AFPACKET sniffer.
• Major code refactoring of the base code.
• Completely redesigned the configuration checker.

What's Changed

• feat: support IP defrag on afpacket sniffer by @dmachard in #691
• feat: redesign config checker to support freeform keys by @dmachard in #678
• feat: add Scalyr defaults by @pieterlexis-tomtom in #677
• feat: major code refacto in collectors package by @dmachard in #667
• fix: quick fix to avoid panic with fragmented packet by @dmachard in #684
• chore(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8 by @dependabot in #680

Bug reporters

• @top-secrett #683
• @ pieterlexis-tomtom #676

Full Changelog: v0.43.0...v0.44.0-beta2

v0.43.0

What's Changed

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some great features

• Compression support on DNSTap collector and logger
• Custom Flat-JSON relabeling
• New ClickHouse logger thank to @zunnurainbadar

What's Changed

• feat: transform: normalize lowercase RR's by @dmachard in #665
• feat: build with golang 1.22 and 1.20 support removed by @dmachard in #660
• feat: added Clickhouse Logger by @zunnurainbadar in #349
• feat: add compression support on DNSTap client and server by @dmachard in #538
• feat: new relabeling transform by @dmachard in #645
• fix: relabel-config error in loki client by @dmachard in #664
• chore(docs): add more docs for the atags transformer by @dmachard in #656 #659
• chore(deps): bump golang.org/x/sys from 0.17.0 to 0.18.0 by @dependabot in #651
• chore(deps): bump golang.org/x/net from 0.21.0 to 0.22.0 by @dependabot in #652
• chore(deps): bump Loki to v2.9.6
• chore: code factory on transformers by @dmachard in #666

New Contributors

• @zunnurainbadar made their first contribution in #349

Bug reporters

• @rajmohanram #643

Full Changelog: v0.42.0...v0.43.0

v0.42.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some important bug fixes (memory leak) and minor features

• Fix memory leak with ElasticSearch logger, thank to @misaki-kawakami to report that.
• Optimizations to reduce CPU usage, more particularly for flat-json
• Add support for query-zone field on DNStap collector

What's Changed

• feat: decode dns class on queries and replies by @dmachard in #635
• feat: more docs for kafka and fluentd loggers, change channel buffer to 4096 by @dmachard in #637 #639
• feat: normalize transformer: add icann managed flag in dnsmessage by @dmachard in #640
• feat: refactor to optimize dnsmessage model by @dmachard in #632
• feat: dnstap collector: decode query zone field by @dmachard in #621
• feat: dnstap collector: ignore empty raw dns payload by @dmachard in #622
• fix: elasticsearch: release buffer properly to avoid memory leaks by @dmachard in #630 #644
• fix: build to support 32b platform by @dmachard in #620
• chore(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7 by @dependabot in #625
• chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #633
• chore(deps): bump golang.org/x/sys from 0.16.0 to 0.17.0 by @dependabot in #606
• chore(deps): bump golang.org/x/net from 0.20.0 to 0.21.0 by @dependabot in #607
• chore(deps): bump github.com/prometheus/client_model from 0.5.0 to 0.6.0 by @dependabot in #616
• chore(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 by @dependabot in #647
• chore(deps): bump github.com/prometheus/client_golang from 1.18.0 to 1.19.0 by @dependabot in #648
• chore(ci): add timeout in github actions tests by @dmachard in #623
• chore(ci): add bench-go github actions by @dmachard in #638
• chore(ci): bump softprops/action-gh-release from 1 to 2 by @dependabot in #646

Bug reporters

• @mikeS7
• @misaki-kawakami

Full Changelog: v0.41.0...v0.42.0

v0.41.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some new features and bug fixes.

• RPZ policy fields support on DNStap/PowerDNS protobuf collectors
• A new peer-name text-format directive and JSON field, thanks to @peterand-pa for idea
• Support the DoH3 HTTP field added with the release of DNSdist 1.9

What's Changed

• feat: add doh3 http version field support and test for DoQ by @dmachard in #610
• feat: add peer name text-format directive by @dmachard in #608
• feat: add configuration examples for pipeline mode by @dmachard in #599
• feat: add DNStap policy support by @dmachard in #593 #530
• feat: add RPZ policy details with powerdns protobuf by @dmachard in #590 #572
• feat: expose dns id in json model by @dmachard in #586
• feat: detect and exit on error with invalid text directives by @dmachard in #576
• fix: fluentd: remove msgpack lib and duplicates entries #611 by @dmachard in #612
• fix: add missing rd and cd flags in json model by @dmachard in #589
• fix: dnstap collector: stuck on stop can be possible in specific situation by @dmachard in #597
• fix: syslog logger - fix protocol wrong type for unix socket by @dmachard in #598
• chore(docs): add documentation to use docker-compose by @dmachard in #563
• chore(docs): update makefile and docs by @dmachard in #604
• chore(deps): bump golangci/golangci-lint-action from 3 to 4 by @dependabot in #603
• chore(deps): bump alpine from 3.19.0 to 3.19.1 by @dependabot in #577
• chore(deps): bump github.com/google/uuid from 1.5.0 to 1.6.0 by @dependabot in #579
• chore(deps): bump github.com/klauspost/compress from 1.17.4 to 1.17.6 by @dependabot in #596
• chore(deps): bump github.com/dmachard/go-topmap from 0.5.0 to 1.0.0 by @dependabot in #580
• chore(deps): bump github.com/dmachard/go-powerdns-protobuf from 1.0.0 to 1.1.0 by @dependabot in #578

Full Changelog: v0.40.2...v0.41.0

Bug reporters

• @johnhtodd
• @peterand-pa
• @Jon1760

v0.40.2

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

What's Changed

• fix(major): patch to avoid deadlock situations in multiplexer mode - regression by @dmachard in #569
• fix(minor): detect invalid logger type in configuration file by @dmachard in #573
• fix(minor): add tests to check and avoid deadlocks with pipeline mode by @dmachard in #571
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.46 to 0.4.47 by @dependabot in #517
• chore(deps): bump github.com/google/uuid from 1.4.0 to 1.5.0 by @dependabot in #518
• chore(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0 by @dependabot in #519
• chore(deps): bump google.golang.org/protobuf from 1.31.0 to 1.32.0 by @dependabot in #524
• chore(deps): clean security code scanning and some dependencies by @dmachard in #554
• chore(deps): bump Loki to v2.9.3 by @dmachard in #555 #556
• chore(deps): bump golang.org/x/sys from 0.15.0 to 0.16.0 by @dependabot in #559
• chore(deps): bump github.com/dmachard/go-framestream from 0.7.0 to 0.10.0 by @dependabot in #560
• chore(deps): bump github.com/miekg/dns from 1.1.57 to 1.1.58 by @dependabot in #561
• chore(deps): bump github.com/prometheus/common from 0.45.0 to 0.46.0 by @dependabot in #558

Full Changelog: v0.40.0...v0.40.2

v0.40.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release is major and introduces some important breaking changes and bug fixes.

• A new codebase to support the future pipelining mode, adding more flexibility to handle your incoming logs.
• New Prometheus metrics has been added to fix a potential unlimited memory usage, which is not acceptable as resources are not limitless.
• The possibility to extend ougoing DNStap streams with metadata such as geoip, filtering, ... It applied to have a DNScollector in remote side to decode these extended DNStap messages
• Invalid configurations are now properly detected.

What's Changed

• feat (#516): dnstap logger/collector - support extended field by @dmachard in #548
• feat: dnstap logger: support of DNSTap extra field by @dmachard in #547
• feat: experimental support for pipelining by @dmachard in #503
• feat (#473): configurable CIDR mask and hash for user-privacy transformer by @dmachard in #528
• feat (#522): more checks to detect invalid configurations by @dmachard in #523
• feat: rename dnstap logger to dnstapclient in YAML config (breaking changes) by @dmachard in #526
• feat: avoid unlimited mem usage with prometheus logger (breaking changes in metrics) by @dmachard in #529
• feat: update grafana dashboard according to PR#529 by @dmachard in #542
• fix: kafka, redis, fluentd and tcpclient loggers: no flush occurred on specific conditions with connection attempts by @dmachard in #552
• fix: syslog logger: fix crash when remote server refused connections by @dmachard in #551
• fix: bad yaml key for dnstap-relay collector by @dmachard in #546
• fix (#539): missing latency when quiet-text is enabled by @dmachard in #540
• fix: typo error - text format directive reducer-occurences renamed to reducer-occurrences
• chore(deps): bump golang.org/x/sys from 0.14.1-.... to 0.15.0 by @dependabot in #486
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.45 to 0.4.46 by @dependabot in #494
• chore(deps): bump golang.org/x/net from 0.18.0 to 0.19.0 by @dependabot in #495
• chore(deps): bump github.com/klauspost/compress from 1.17.3 to 1.17.4 by @dependabot in #496
• chore(deps): bump alpine from 3.18.5 to 3.19.0 by @dependabot in #505
• chore(deps): bump golang from 1.21.4-alpine3.17 to 1.21.5-alpine3.17 by @dependabot in #506
• chore(ci): update testing-dnstap.yml with unbound 1.19 by @dmachard in #543
• chore(docs): docs updated by @dmachard in #541

Full Changelog: v0.39.0...v0.40.0

Bug reporters

• @top-secrett
• @johnhtodd

v0.39.0

What's Changed

• feat: optimize docker image for #470 by @dmachard in #492
• feat: new length-unit attribute for text-format #478 by @dmachard in #491
• feat: add sampling rate to downsampled items (first shot) by @dmachard in #480
• feat #481 : add compression support for kafka logger by @dmachard in #482
• feat: #453 introduced metrics etldplusone_total and etldplusone_top by @kabenin in #465
• feat: syslog: support batch mode to send logs by @dmachard in #489
• feat #499 : detect go version in a dynamic way for Makefile by @dmachard in #502
• fix #509 : logger prometheus - fix panic error with invalid qname by @dmachard in #514
• fix #461 : regression - closing dnstap stream on bind 9 very slow by @dmachard in #472
• fix #483 : syslog logger - remove NULL char in qname by @dmachard in #484
• refactor: add more linters to golangci-lint by @pieterlexis-tomtom in #456
• refactor: code factory on multiplexer mode by @dmachard in #497
• refactor: code factory on config part to prepare future versions by @dmachard in #498
• doc: update README.md by @dmachard in #479
• chore(deps): bump github.com/miekg/dns from 1.1.56 to 1.1.57 by @dependabot in #476
• chore(deps): bump github.com/cilium/ebpf from 0.12.2 to 0.12.3 by @dependabot in #463
• chore(deps): bump golang.org/x/net from 0.17.0 to 0.18.0 by @dependabot in #464
• chore(deps): bump golang from 1.21.3-alpine3.17 to 1.21.4-alpine3.17 by @dependabot in #462
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.44 to 0.4.45 by @dependabot in #485
• chore(deps): bump github.com/klauspost/compress from 1.17.2 to 1.17.3 by @dependabot in #487
• chore(deps): bump alpine from 3.18.4 to 3.18.5 by @dependabot in #493
• chore(tests): 25 more unit tests for regression added
• chore(ci): bump actions/setup-go and checkout from 2 to 5 by @dependabot in #469 #501 #468
• chore(ci): bump actions/setup-python from 4 to 5 by @dependabot in #500

Full Changelog: v0.38.0...v0.39.0

Bug reporters

• @HackBeiChen
• @johnhtodd
• @james-stevens

v0.38.0

What's Changed

• feat: add badges through github-actions to track test counts and lines of code by @dmachard in #466
• feat: dnstap collector: new settings to disable the embedded dns parser by @dmachard in #458
• feat: support full reload of the configuration on SIGHUP signal by @dmachard in #440
• feat: some code factory in TLS client config and new package for processors by @dmachard in #440 #445
• feat: syslog logger: add new settings to support custom hostname and program name by @dmachard in #442
• feat: syslog logger: rename format settings with formatter (breacking change) by @dmachard in #442
• feat: syslog logger: add new framer option by @dmachard in #442
• feat: add mTLS support on all loggers by @dmachard in #442
• feat: set flat-json as default format on tcp and redispub logger by @dmachard in #445
• fix: powerdns collector: replace go/dnsmessage by miekg/dns by @dmachard in #460
• fix: #450 update tcpclient logger to use flush interval by @mattb-nn in #452
• fix: #434 improve logging in publicsuffix transform by @dmachard in #439
• fix: avoid blocking situation during stop in the ReadFromConnection function on redispub and backport to tcpclient and fluentd loggers by @dmachard in #445
• docs: update loggers settings by @dmachard in #445
• chore(deps): bump github.com/google/uuid from 1.3.1 to 1.4.0 by @dependabot in #444
• chore(deps): replace unmaintained RackSec/srslog by dmachard/go-clientsyslog by @dmachard in #442
• chore(deps): bump github.com/cilium/ebpf from 0.12.0 to 0.12.2 by @dependabot in #435
• chore(deps): bump github.com/klauspost/compress from 1.17.1 to 1.17.2 by @dependabot in #438
• chore(deps): bump github.com/fsnotify/fsnotify from 1.6.0 to 1.7.0 by @dependabot in #437
• chore(deps): bump github.com/prometheus/common from 0.44.0 to 0.45.0 by @dependabot in #436

New Contributors

• @step-security-bot made their first contribution in #447
• @mattb-nn made their first contribution in #452

Full Changelog: v0.37.0...v0.38.0-beta4

v0.37.0

What's Changed

• feat: prometheus logger: change on build_info metric by @dmachard in #426
• feat: disable debug informations to get smaller binary by @dmachard in #427
• feat: enable multiarch builds with Dockerfile by @dmachard in #428
• feat: support building for FreeBSD by @pieterlexis-tomtom in #420
• feat: syslog logger: add tag and framing support by @dmachard in #422
• fix: redispub logger: reading redis responses and potential fix for #353 by @gallypette in #411
• fix: ci/gh-actions: fix failing loggers tests by @dmachard in #418
• docs: update logger_restapi.md by @brankomijuskovic in #431
• chore(deps): bump github.com/prometheus/client_golang from 1.16.0 to 1.17.0 by @dependabot in #409
• chore(deps): bump alpine from 3.18.3 to 3.18.4 by @dependabot in #408
• chore(deps): bump golang from 1.21.1-alpine3.17 to 1.21.2-alpine3.17 by @dependabot in #416
• chore(deps): bump golang.org/x/net from 0.15.0 to 0.17.0 by @dependabot in #415 #419
• chore(deps): bump github.com/prometheus/client_model to 0.5.0 by @dependabot in #413
• chore(deps): bump github.com/cilium/ebpf from 0.11.0 to 0.12.0 by @dependabot in #421
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.43 to 0.4.44 by @dependabot in #425

New Contributors

• @brankomijuskovic made their first contribution in #431

Full Changelog: v0.36.0...v0.37.0

v0.36.0

What's Changed

• feat: elasticsearch logger: change op_type to allow sending to Data Streams by @ruffy91 in #400
• feat: support golang 1.21 by @dmachard in #399
• fix: decouple keep-query-ip from drop-query-ip to fix filtering by @arvchristos in #405
• fix: panic error in docker because of go 1.21 by @dmachard in #402
• chore(ci): update github-actions by @dmachard in #404
• chore(deps): bump github.com/miekg/dns from 1.1.55 to 1.1.56 by @dependabot in #398
• chore(deps): bump github.com/klauspost/compress from 1.16.7 to 1.17.0 by @dependabot in #397
• chore(deps): bump golang from 1.21.0-alpine3.17 to 1.21.1-alpine3.17 by @dependabot in #396
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.42 to 0.4.43 by @dependabot in #406

New Contributors

• @ruffy91 made their first contribution in #400

Full Changelog: v0.35.0...v0.36.0