Skip to content

Latest commit

 

History

History
913 lines (835 loc) · 71.2 KB

CHANGELOG.next.asciidoc

File metadata and controls

913 lines (835 loc) · 71.2 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

  • Update to Golang 1.12.1. 11330

  • Disable Alibaba Cloud and Tencent Cloud metadata providers by default. 12812

  • Allow Metricbeat’s beat module to read monitoring information over a named pipe or unix domain socket. 14558

  • Remove version information from default ILM policy for improved upgrade experience on custom policies. 14745

  • Running setup cmd respects setup.ilm.overwrite setting for improved support of custom policies. 14741

  • Libbeat: Do not overwrite agent.*, ecs.version, and host.name. 14407

  • Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. 15091

  • Users can now specify monitoring.cloud. to override monitoring.elasticsearch. settings. 14399 15254

  • Refactor metadata generator to support adding metadata across resources 14875

  • Update to ECS 1.4.0. 14844

  • The document id fields has been renamed from @metadata.id to @metadata._id 15859

  • Variable substitution from environment variables is not longer supported. #15937

  • Change aws_elb autodiscover provider field name from elb_listener.* to aws.elb.*. 16219 #16402

  • Remove AddDockerMetadata and AddKubernetesMetadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

  • Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. 17938

  • Make error message about locked data path actionable. 18667

  • Ensure dynamic template names are unique for the same field. 18849

  • Autodiscover doesn’t generate any configuration when a variable is missing. Previously it generated an incomplete configuration. 20898

  • Added certificate TLS verification mode to ignore server name mismatch. 12283 20293

  • Remove redundant cloudfoundry.*.timestamp fields. This value is set in @timestamp. 21175

  • Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs #21179

  • API address is a required setting in add_cloudfoundry_metadata. 21759

Auditbeat

  • Change network.direction values to ECS recommended values (inbound, outbound). 12445 20695

  • Docker container needs to be explicitly run as user root for auditing. 21202

  • File integrity dataset no longer includes the leading dot in file.extension values (e.g. it will report "png" instead of ".png") to comply with ECS. 21644

Filebeat

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • CEF extensions are now mapped to the data types defined in the CEF guide. 14342

  • Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. 16025 17910

  • Improve ECS categorization field mappings for nginx module. http.request.referrer only populated when nginx sets a value 16174 17844

  • Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . 16180 17982

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta) will no longer send the host field that contains information about the host Filebeat is running on. This is because the host field specifies the host on which the event happened. 13920 18223

  • With the default configuration the following modules will no longer send the host field. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920

    • Cisco 18753

    • CrowdStrike 19132

    • Fortinet 19133

    • iptables 18756

    • Checkpoint 18754

    • Netflow 19087

    • Zeek 19113 (forwarded tag is not included by default)

    • Suricata 19107 (forwarded tag is not included by default)

    • CoreDNS 19134 (forwarded tag is not included by default)

    • Envoy Proxy 19134 (forwarded tag is not included by default)

  • Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta) will no longer send the host field that contains information about the host Filebeat is running on. This is because the host field specifies the host on which the event happened. 13920 18223

  • With the default configuration the cef and panw modules will no longer send the host field. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920 18223

  • Adds check on <no value> config option value for the azure input resource_manager_endpoint. 18890

  • Okta module now requires objects instead of JSON strings for the http_headers, http_request_body, pagination, rate_limit, and ssl variables. 18953

  • Adds oauth support for httpjson input. 18415 18892

  • Adds split_events_by option to httpjson input. 19246

  • Adds date_cursor option to httpjson input. 19483

  • Adds Gsuite module with SAML support. 19329

  • Adds Gsuite User Accounts support. 19329

  • Adds Gsuite Login audit support. 19702

  • Adds Gsuite Admin support. 19769

  • Adds Gsuite Drive support. 19704

  • Adds Gsuite Groups support. 19725

  • Move file metrics to dataset endpoint 19977

  • Disable the option of running --machine-learning on its own. 20241

  • Fix PANW field spelling "veredict" to "verdict" on event.action 18808

  • Tracking session end reason in panw module. 18705

  • Removed experimental modules citrix, kaspersky, rapid7 and tenable. 20706

  • Add support for GMT timezone offsets in decode_cef. 20993

  • API address and shard ID are required settings in the Cloud Foundry input. 21759

  • Remove suricata.eve.timestamp alias field. 10535 22095

Heartbeat

Journalbeat

  • Remove broken dashboard. 15288

  • Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

  • Add new dashboard for VSphere host cluster and virtual machine 14135

  • kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

  • Update cloudwatch metricset mapping for both metrics and dimensions. 15245

  • Make use of secure port when accessing Kubelet API 16063

  • Move service config under metrics and simplify metric types. 18691

  • Fix ECS compliance of user.id field in system/users metricset 19019

  • Rename googlecloud stackdriver metricset to metrics. 19718

  • Remove "invalid zero" metrics on Windows and Darwin, don’t report linux-only memory and diskio metrics when running under agent. 21457

  • Change cloud.provider from googlecloud to gcp. 21775

  • API address and shard ID are required settings in the Cloud Foundry module. 21759

Packetbeat

  • TLS: Fields have been changed to adapt to ECS. 15497

  • TLS: The behavior of send_certificates and include_raw_certificates options has changed. 15497

  • Added redact_headers configuration option, to allow HTTP request headers to be redacted whilst keeping the header field included in the beat. 15353

  • Add dns.question.subdomain and dns.question.top_level_domain fields. 14578

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

  • Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

  • Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

  • Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

  • Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

  • Add Powershell module. Support for event ID’s: 400, 403, 600, 800, 4103, 4014, 4105, 4106. 16262 18526

  • Fix Powershell processing of downgraded engine events. 18966

  • Fix unprefixed fields in fields.yml for Powershell module 18984

Functionbeat

Bugfixes

Affecting all Beats

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Fix spooling to disk blocking infinitely if the lock file can not be acquired. 15338

  • Update replicaset group to apps/v1 15802

  • Fix metricbeat test output with an ipv6 ES host in the output.hosts. 15368

  • Fix convert processor conversion of string to integer with leading zeros. 15513 15557

  • Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data 17223

  • Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

  • Fix panic in the Logstash output when trying to send events to closed connection. 15568

  • Fix missing output in dockerlogbeat 15719

  • Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

  • Do not load dashboards where not available. 15802

  • Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. #15516

  • Remove superfluous use of number_of_routing_shards setting from the default template. 16038

  • Fix index names for indexing not always guaranteed to be lower case. 16081

  • Upgrade go-ucfg to latest v0.8.1. #15937

  • Fix loading processors from annotation hints. 16348

  • Fix an issue that could cause redundant configuration reloads. 16440

  • Fix k8s pods labels broken schema. 16480

  • Fix k8s pods annotations broken schema. 16554

  • Upgrade go-ucfg to latest v0.8.3. #16450

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

  • Improve some logging messages for add_kubernetes_metadata processor #16866

  • Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

  • Fail to start if httpprof is used and it cannot be initialized. 17028

  • Fix concurrency issues in convert processor when used in the global context. 17032

  • Fix bug with monitoring.cluster_uuid setting not always being exposed via GET /state Beats API. 16732 17420

  • Fix building on FreeBSD by removing build flags from add_cloudfoundry_metadata processor. 17486

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix setup.dashboards.index setting not working. 17749

  • Fix goroutine leak and Elasticsearch output file descriptor leak when output reloading is in use. 10491 17381

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Fix panic when assigning a key to a nil value in an event. 18143

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • [Autodiscover] Check if runner is already running before starting again. 18564

  • Fix keystore add hanging under Windows. 18649 18654

  • Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

  • Fix potential race condition in fingerprint processor. 18738

  • Add better handling for Kubernetes Update and Delete watcher events. 18882

  • Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

  • Fixed a service restart failure under Windows. 18914 18916

  • The monitoring.elasticsearch.api_key value is correctly base64-encoded before being sent to the monitoring Elasticsearch cluster. 18939 18945

  • Fix kafka topic setting not allowing upper case characters. 18854 18640

  • Fix redis key setting not allowing upper case characters. 18854 18640

  • Fix config reload metrics (libbeat.config.module.start/stops/running). 19168

  • Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed 18979

  • Server-side TLS config now validates certificate and key are both specified 19584

  • Fix terminating pod autodiscover issue. 20084

  • Fix seccomp policy for calls to chmod and chown. 20054

  • Remove unnecessary restarts of metricsets while using Node autodiscover 19974

  • Output errors when Kibana index pattern setup fails. 20121

  • Fix issue in autodiscover that kept inputs stopped after config updates. 20305

  • Add service resource in k8s cluster role. 20546

  • [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. 20571

  • Rename cloud.provider az value to azure inside the add_cloud_metadata processor. 20689

  • Add missing country_name geo field in add_host_metadata and add_observer_metadata processors. 20796 20811

  • [Autodiscover] Handle input-not-finished errors in config reload. 20915

  • Explicitly detect missing variables in autodiscover configuration, log them at the debug level. 20568 20898

  • Fix libbeat.output.write.bytes and libbeat.output.read.bytes metrics of the Elasticsearch output. 20752 21197

  • The o365input and o365 module now recover from an authentication problem or other fatal errors, instead of terminating. 21258

  • Orderly close processors when processing pipelines are not needed anymore to release their resources. 16349

  • Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. 21851

Auditbeat

  • system/socket: Fixed compatibility issue with kernel 5.x. 15771

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

  • system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

  • system/socket: Fix dataset using 100% CPU and becoming unresponsive in some scenarios. 19033 19764

  • system/socket: Fixed tracking of long-running connections. 19033

  • system/package: Fix librpm loading on Fedora 31/32. NNNN

  • auditd: Fix spelling of anomaly in event.category.

  • auditd: Fix typo in event.action of removed-user-role-from. 19300

  • auditd: Fix typo in event.action of used-suspicious-link. 19300

  • system/socket: Fix kprobe grouping to allow running more than one instance. 20325

  • system/socket: Fixed a crash due to concurrent map read and write. 21192 21690

Filebeat

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Fix a problem in Filebeat input httpjson where interval is not used as time.Duration. 14728

  • Fix SSL config in input.yml for Filebeat httpjson input in the MISP module. 14767

  • Check content-type when creating new reader in s3 input. 15252 15225

  • Fix session reset detection and a crash in Netflow input. 14904

  • Handle errors in handleS3Objects function and add more debug messages for s3 input. 15545

  • netflow: Allow for options templates without scope fields. 15449

  • netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). 15449

  • netflow: Fix compatibility with some Cisco devices by changing the field class_id from short to long. 15449

  • Fixed dashboard for Cisco ASA Firewall. 15420 15553

  • Fix mapping of fortinet.firewall.mem as integer. 19335

  • Ensure all zeek timestamps include millisecond precision. 14599 16766

  • Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix typos in zeek notice fileset config file. 15764 15765

  • Fix mapping error when zeek weird logs do not contain IP addresses. 15906

  • Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

  • Fix mapping error for cloudtrail additionalEventData field 16088

  • Fix a connection error in httpjson input. 16123

  • Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Rewrite azure filebeat dashboards, due to changes in kibana. 16466

  • Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

  • Fix merging of fileset inputs to replace paths and append processors. #16450

  • Add queue_url definition in manifest file for aws module. #16640

  • Fix issue where autodiscover hints default configuration was not being copied. 16987

  • Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

  • Fixed various Cisco FTD parsing issues. 16863 16889

  • Fix default index pattern in IBM MQ filebeat dashboard. 17146

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

  • Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

  • Fix elasticsearch.audit data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406

  • CEF: Fixed decoding errors caused by trailing spaces in messages. 17253

  • Fixed activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Unescape file name from SQS message. 18370

  • Improve cisco asa and ftd pipelines' failure handler to avoid mapping temporary fields. 18391 18392

  • Fix source.address not being set for nginx ingress_controller 18511

  • Fix PANW module wrong mappings for bytes and packets counters. 18522 18525

  • Fix googlecloud.audit pipeline to only take in fields that are explicitly defined by the dataset. 18465 18472

  • Fix a rate limit related issue in httpjson input for Okta module. 18530 18534

  • Fixed ingestion of some Cisco ASA and FTD messages when a hostname was used instead of an IP for NAT fields. 14034 18376

  • Fix o365.audit failing to ingest events when ip address is surrounded by square brackets. 18587 18591

  • Fix Kubernetes Watcher goroutine leaks when input config is invalid and input.reload is enabled. 18629 18630

  • Fix o365 module ignoring var.api settings. 18948

  • Okta module now sets the Elasticsearch _id field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. 18953

  • Fix netflow module to support 7 bytepad for IPFIX template. 18098

  • Fix improper nesting of session_issuer object in aws cloudtrail fileset. 18894 18915

  • Fix Cisco ASA ASA 3020** and 106023 messages 17964

  • Fix date and timestamp formats for fortigate module 19316

  • Add missing default_field: false to aws filesets fields.yml. 19568

  • Fix tls mapping in suricata module 19492 19494

  • Fix memory leak in tcp and unix input sources. 19459

  • Fix Cisco ASA dissect pattern for 313008 & 313009 messages. 19149

  • Update container name for the azure filesets. 19899

  • Fix bug with empty filter values in system/service 19812

  • Fix S3 input to trim delimiter /n from each log line. 19972

  • Fix auditd module syscall table for ppc64 and ppc64le. 20052

  • Fix Filebeat OOMs on very long lines 19500, 19552

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962

  • Ignore missing in Zeek module when dropping unecessary fields. 19984

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

  • Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

  • Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

  • Fix event.outcome logic for azure/siginlogs fileset 20254

  • Fix fortinet setting event.timezone to the system one when no tz field present 20273

  • Fix okta geoip lookup in pipeline for destination.ip 20454

  • Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

  • Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

  • Improve validation checks for Azure configuration 20369 20389

  • Fix event.kind for system/syslog pipeline 20365 20390

  • Fix event.type for zeek/ssl and duplicate event.category for zeek/connection 20696

  • Fix long registry migration times. 20717 20705

  • Fix event types and categories in auditd module to comply with ECS 20652

  • Update documentation in the azure module filebeat. 20815

  • Remove wrongly mapped tls.client.server_name from fortinet/firewall fileset. 20983

  • Fix an error updating file size being logged when EOF is reached. 21048

  • Fix error when processing AWS Cloudtrail Digest logs. 21086 20943

  • Provide backwards compatibility for the set processor when Elasticsearch is less than 7.9.0. 20908

  • Handle multiple upstreams in ingress-controller. 21215

  • Provide backwards compatibility for the append processor when Elasticsearch is less than 7.10.0. 21159

  • Fix checkpoint module when logs contain time field. 20567

  • Add field limit check for AWS Cloudtrail flattened fields. 21388 21382

Heartbeat

  • Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. 13687

  • Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

  • Fixed scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. 16397

  • Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. 17549

  • Add support for new service_name option to all monitors. 19932.

  • Stop rescheduling tasks of stopped monitors. 20570

Heartbeat

Journalbeat

Metricbeat

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fix regular expression to detect instance name in perfmon metricset. 14273 14666

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

  • Fix docker.container.size fields values 14979 15224

  • Make kibana module more resilient to Kibana unavailability. 15258 15270

  • Fix panic exception with some unicode strings in perfmon metricset. 15264

  • Make logstash module more resilient to Logstash unavailability. 15276 15306

  • Add username/password in Metricbeat autodiscover hints 15349

  • Fix CPU count in docker/cpu in cases where no online_cpus are reported 15070

  • Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

  • Use RFC3339 format for timestamps collected using the SQL module. 15847

  • Change lookup_fields from metricset.host to service.address 15883

  • Avoid parsing errors returned from prometheus endpoints. 15712

  • Add dedot for cloudwatch metric name. 15916 15917

  • Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

  • Fix skipping protocol scheme by light modules. pull

  • Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

  • Change sqs metricset to use average as statistic method. 16438

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

  • Fix detection and logging of some error cases with light modules. 14706

  • Add dashboard for redisenterprise module. 16752

  • Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. 14233

  • Dynamically choose a method for the system/service metricset to support older linux distros. 16902

  • Use max in k8s apiserver dashboard aggregations. 17018

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

  • Use max in k8s overview dashboard aggregations. 17015

  • Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

  • Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Further revise check for bad data in docker/memory. 17400

  • Combine cloudwatch aggregated metrics into single event. 17345

  • Fix how we filter services by name in system/service 17400

  • Fix cloudwatch metricset missing tags collection. 17419 17424

  • check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418

  • Fix aws.s3.bucket.name terms_field in s3 overview dashboard. 17542

  • Fix Unix socket path in memcached. 17512

  • Fix vsphere VM dashboard host aggregation visualizations. 17555

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

  • Add a switch to the driver definition on SQL module to use pretty names 17378

  • Remove specific win32 api errors from events in perfmon. 18292 18361

  • Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

  • Fix application_pool metricset after pdh changes. 18477

  • Fix tags_filter for cloudwatch metricset in aws. 18524

  • Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

  • Fix getting gcp compute instance metadata with partial zone/region in config. 18757

  • Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

  • Fix compute and pubsub dashboard for googlecloud module. 18962 18980

  • Fix crash on vsphere module when Host information is not available. 18996 19078

  • Fix incorrect usage of hints builder when exposed port is a substring of the hint 19052

  • Stop counterCache only when already started 19103

  • Remove dedot for tag values in aws module. 19112 19221

  • Fix empty field name errors in the application pool metricset. 19537

  • Set tags correctly if the dimension value is ARN 19111 19433

  • Fix bug incorrect parsing of float numbers as integers in Couchbase module 18949 19055

  • Fix mapping of service start type in the service metricset, windows module. 19551

  • Fix config example in the perfmon configuration files. 19539

  • Add missing info about the rest of the azure metricsets in the documentation. 19601

  • Fix k8s scheduler compatibility issue. 19699

  • Fix SQL module mapping NULL values as string 18955 #18898[18898

  • Modify doc for app_insights metricset to contain example of config. 20185

  • Add required option for metrics in app_insights. 20406

  • Groups same timestamp metric values to one event in the app_insights metricset. 20403

  • Updates vm_compute metricset with more info on guest metrics. 20448

  • Add fallback for PdhExpandWildCardPathW failing in perfmon metricset. 20139 20630

  • Fix resource tags in aws cloudwatch metricset 20326 20385

  • Add support for azure light metricset app_stats. 20639

  • Fix ec2 disk and network metrics to use Sum statistic method. 20680

  • Fill cloud.account.name with accountID if account alias doesn’t exist. 20736

  • Fix ec2 disk and network metrics to use Sum statistic method. 20680

  • The Kibana collector applies backoff when errored at getting usage stats 20772

  • Update fields.yml in the azure module, missing metrics field. 20918

  • The elasticsearch/index metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. 20938

  • Disable Kafka metricsets based on Jolokia by default. They require a different configuration. 20989

  • Fix panic index out of range error when getting AWS account name. 21101 21095

  • Handle missing counters in the application_pool metricset. 21071

  • Fix timestamp handling in remote_write. 21166

  • Fix remote_write flaky test. 21173

  • Visualization title fixes in aws, azure and googlecloud compute dashboards. 21098

  • Fix retrieving resources by ID for the azure module. 21711 21707

  • Use timestamp from CloudWatch API when creating events. 21498

  • Report the correct windows events for system/filesystem 21758

  • Fix azure storage event format. 21845

  • Fix panic in kubernetes autodiscover related to keystores 21843 21880

  • [Kubernetes] Remove redundant dockersock volume mount 22009

  • Revert change to report process.memory.rss as process.memory.wss on Windows. 22055

Packetbeat

  • Enable setting promiscuous mode automatically. 11366

  • Fix process monitoring when ipv6 is disabled under Linux. 19941 19945

  • Add "network" to event.category 20364 20392

Winlogbeat

  • Fix invalid IP addresses in DNS query results from Sysmon data. 18432 18436

  • Fix event.outcome in the security module for non-English languages. 20079 #20564[2056

  • Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. 20521 20540

  • Fields from Winlogbeat modules were not being included in index templates and patterns. 18983

Functionbeat

  • Fix timeout option of GCP functions. 16282 16287

  • Do not need Google credentials if not required for the operation. 17329 21072

  • Fix dependency issues of GCP functions. 20830 21070

  • Fix catchall bucket config errors by adding more validation. 16282 16287

  • Fix Google Cloud Function configuration issue. 20864 22156

Added

Affecting all Beats

  • Add a friendly log message when a request to docker has exceeded the deadline. 15336

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. 9260

  • Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. 11394

  • add_host_metadata is no GA. 13148

  • Add providers setting to add_cloud_metadata processor. 13812

  • GA the script processor. 14325

  • Add fingerprint processor. 11173 14205

  • Add support for API keys in Elasticsearch outputs. 14324

  • Ensure that init containers are no longer tailed after they stop 14394

  • Add consumer_lag in Kafka consumergroup metricset 14822

  • Make use of consumer_lag in Kafka dashboard 14863

  • Refactor kubernetes autodiscover to enable different resource based discovery 14738

  • Add add_id processor. 14524

  • Enable TLS 1.3 in all beats. 12973

  • Spooling to disk creates a lockfile on each platform. 15338

  • Fingerprint processor adds a new xxhash hashing algorithm 15418

  • Enable DEP (Data Execution Protection) for Windows packages. 15149

  • Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938

  • Add document_id setting to decode_json_fields processor. 15859

  • Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

  • Add aws_ec2 provider for autodiscover. 12518 14823

  • Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

  • Add support for multiple password in redis output. 16058 16206

  • Add support for Histogram type in fields.yml 16570

  • Windows .exe files now have embedded file version info. 15232t

  • Remove experimental flag from setup.template.append_fields 16576

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

  • Add support for kubernetes provider to recognize namespace level defaults 16321

  • Add capability of enrich container.id with process id in add_process_metadata processor 15947

  • Update RPM packages contained in Beat Docker images. 17035

  • Add Kerberos support to Kafka input and output. 16781

  • Update supported versions of redis output. 17198

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add optional regex based cid extractor to add_kubernetes_metadata processor. 17360

  • Add replace processor for replacing string values of fields. 17342

  • Add urldecode processor to for decoding URL-encoded fields. 17505

  • Add support for AWS IAM role_arn in credentials config. 17658 12464

  • Add Kerberos support to Elasticsearch output. 17927

  • Add k8s keystore backend. 18096

  • Set agent.name to the hostname by default. 16377 18000

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add support for basic ECS logging. 17974

  • Add config example of how to skip the add_host_metadata processor when forwarding logs. 13920 18153

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add backoff configuration options for the Kafka output. 16777 17808

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add Kerberos support to Elasticsearch output. 17927

  • Add support for fixed length extraction in dissect processor. 17191

  • Update RPM packages contained in Beat Docker images. 17035

  • Add TLS support to Kerberos authentication in Elasticsearch. 18607

  • Change ownership of files in docker images so they can be used in secured environments. 12905

  • Upgrade k8s.io/client-go and k8s keystore tests. 18817

  • Add support for multiple sets of hints on autodiscover 18883

  • Add a configurable delay between retries when an app metadata cannot be retrieved by add_cloudfoundry_metadata. 19181

  • Add data type conversion in dissect processor for converting string values to other basic data types. 18683

  • Add the ignore_failure configuration option to the dissect processor. 19464

  • Add the overwrite_keys configuration option to the dissect processor. 19464

  • Add support to trim captured values in the dissect processor. 19464

  • Added the max_cached_sessions option to the script processor. 19562

  • Add minimum cache TTL for successful DNS responses. 18986

  • Add support for DNS over TLS for the dns processor. 19321

  • Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

  • Add leader election for Kubernetes autodiscover. 20281

  • Add capability of enriching process metadata with contianer id also for non-privileged containers in add_process_metadata processor. 19767

  • Add replace_fields config option in add_host_metadata for replacing host fields. 20490 20464

  • Add container ECS fields in kubernetes metadata. 20984

  • Add ingress controller dashboards. 21052

  • Added experimental citrix module. 20820

  • Added experimental cyberark module. 20820

  • Added experimental proofpoint module. 20820

  • Added experimental snort module. 20820

  • Added experimental symantec module. 20820

  • Added experimental dataset barracuda/spamfirewall. 20820

  • Added experimental dataset cisco/meraki. 20820

  • Added experimental dataset f5/bigipafm. 20820

  • Added experimental dataset fortinet/fortimail. 20820

  • Added experimental dataset fortinet/fortimanager. 20820

  • Added experimental dataset juniper/netscreen. 20820

  • Added experimental dataset sophos/utm. 20820

  • Add Cloud Foundry tags in related events. 21177

  • Cloud Foundry metadata is cached to disk. 20775

  • Add option to select the type of index template to load: legacy, component, index. 21212

  • Add istiod metricset. 21519

  • Release add_cloudfoundry_metadata as GA. 21525

  • Add support for OpenStack SSL metadata APIs in add_cloud_metadata. 21590

  • Add cloud.account.id for GCP into add_cloud_metadata processor. 21776

  • Add proxy metricset for istio module. 21751

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

  • Reference kubernetes manifests mount data directory from the host, so data persist between executions in the same node. 17429

  • Log to stderr when running using reference kubernetes manifests. 174443

  • Fix syscall kprobe arguments for 32-bit systems in socket module. 17500

  • Fix memory leak on when we miss socket close kprobe events. 17500

  • Add system module process dataset ECS categorization fields. 18032

  • Add system module user dataset ECS categorization fields. 18035

  • Add system module login dataset ECS categorization fields. 18034

  • Add system module package dataset ECS categorization fields. 18033

  • Add ECS categories for system module host dataset. 18031

  • Add system module socket dataset ECS categorization fields. 18036

  • Add file integrity module ECS categorization fields. 18012

  • Add file.mime_type, file.extension, and file.drive_letter for file integrity module. 18012

  • Add ECS categorization info for auditd module 18596

  • Add enrichment of auditd seccomp events with name of the architecture, syscall, and signal. 14055 19300

Filebeat

  • Add dashboard for AWS ELB fileset. 15804

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add index option to all inputs to directly set a per-input index value. 14010

  • Add new fileset googlecloud/audit for ingesting Google Cloud Audit logs. 15200

  • Add dashboards to the CEF module (ported from the Logstash ArcSight module). 14342

  • Add expand_event_list_from_field support in s3 input for reading json format AWS logs. 15357 15370

  • Add azure-eventhub input which will use the azure eventhub go sdk. 14092 14882

  • Expose more metrics of harvesters (e.g. read_offset, start_time). 13395

  • Include log.source.address for unparseable syslog messages. 13268 15453

  • Release aws elb fileset as GA. 15426 15380

  • Integrate the azure-eventhub with filebeat azure module (replace the kafka input). 15480

  • Release aws s3access fileset to GA. 15431 15430

  • Add cloudtrail fileset to AWS module. 14657 15227

  • New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. 14553

  • google-pubsub input: ACK pub/sub message when acknowledged by publisher. 13346 14715

  • Remove Beta label from google-pubsub input. 13346 14715

  • Set event.outcome field based on googlecloud audit log output. 15731

  • Add dashboard for AWS vpcflow fileset. 16007

  • Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

  • Add ingress nginx controller fileset 16197

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Add ECS tls and categorization fields to apache module. 16032 16121

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add MQTT input. 15602 16204

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Add ECS categorization fields to activemq module. 16151 16201

  • Improve ECS field mappings in aws module. 16154 16307

  • Improve ECS categorization field mappings in googlecloud module. 16030 16500

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve ECS categorization field mappings in kibana module. 16168 16652

  • Add cloudfoundry input to send events from Cloud Foundry. 16586

  • Improve ECS field mappings in haproxy module. 16162 16529

  • Allow users to override pipeline ID in fileset input config. 9531 16561

  • Improve ECS categorization field mappings in logstash module. 16169 16668

  • Improve ECS categorization field mappings in iis module. 16165 16618

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Improve ECS categorization field mapping in kafka module. 16167 16645

  • Improve ECS categorization field mapping in icinga module. 16164 16533

  • Improve ECS categorization field mappings in ibmmq module. 16163 16532

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Improve ECS categorization field mappings in suricata module. 16181 16843

  • Release ActiveMQ module as GA. 17047 17049

  • Improve ECS categorization field mappings in iptables module. 16166 16637

  • Add pattern for Cisco ASA / FTD Message 734001 16212 16612

  • Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

  • Add custom string mapping to CEF module to support Check Point devices. 16041 16907

  • Added new module o365 for ingesting Office 365 management activity API events. 16196 16386

  • Add Filebeat Okta module. 16362

  • Add source field in k8s events 17209

  • Improve AWS cloudtrail field mappings 16086 16110 17155

  • Added new module crowdstrike for ingesting Crowdstrike Falcon streaming API endpoint event data. 16988

  • Move azure-eventhub input to GA. 15671 17313

  • Improve ECS categorization field mappings in mongodb module. 16170 17371

  • Improve ECS categorization field mappings for mssql module. 16171 17376

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Added access_key_id, secret_access_key and session_token into aws module config. 17456

  • Improve ECS categorization field mappings for mysql module. 16172 17491

  • Release Google Cloud module as GA. 17511

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Added new Checkpoint Syslog filebeat module. 17682

  • Add config option to select a different azure cloud env in the azure-eventhub input and azure module. 17649 17659

  • Enhance elasticsearch/server fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17714

  • Added Unix stream socket support as an input source and a syslog input source. 17492

  • Improve ECS categorization field mappings in misp module. 16026 17344

  • Enhance elasticsearch/deprecation fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17728

  • Make decode_cef processor GA. 17944

  • Added new Fortigate Syslog filebeat module. 17890

  • Improve ECS categorization field mappings in redis module. 16179 17918

  • Improve ECS categorization field mappings in rabbitmq module. 16178 17916

  • Improve ECS categorization field mappings in postgresql module. 16177 17914

  • Improve ECS categorization field mappings for nginx module. 16174 17844

  • Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

  • Improve ECS categorization field mappings for zeek module. 16029 17738

  • Improve ECS categorization field mappings for netflow module. 16135 18108

  • Added an input option publisher_pipeline.disable_host to disable host.name from being added to events by default. 18159

  • Improve ECS categorization field mappings in system module. 16031 18065

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Improve ECS categorization field mappings in osquery module. 16176 17881

  • Add support for v10, v11 and v12 logs on Postgres 13810 17732

  • Add dashboard for Google Cloud Audit and AWS CloudTrail. 17379

  • Added http_endpoint inputhttps://github.com//pull/18298[18298]

  • Add http_endpoint input. 18298

  • Add support for array parsing in azure-eventhub input. 18585

  • Added observer.vendor, observer.product, and observer.type to PANW module events. 18223

  • The logstash module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. 9964 18095

  • Improve ECS categorization field mappings in coredns module. 16159 18424

  • Improve ECS categorization field mappings in envoyproxy module. 16161 18395

  • Improve ECS categorization field mappings in cisco module. 16028 18537

  • The s3 input can now automatically detect gzipped objects. 18283 18764

  • Add geoip AS lookup & improve ECS categorization in aws cloudtrail fileset. 18644 18958

  • Improved performance of PANW sample dashboards. 19031 19032

  • Add support for v1 consumer API in Cloud Foundry input, use it by default. 19125

  • Add new mode to multiline reader to aggregate constant number of lines 18352

  • Explicitly set ECS version in all Filebeat modules. 19198

  • Add awscloudwatch input. 19025

  • Add automatic retries and exponential backoff to httpjson input. 18956

  • Changed the panw module to pass through (rather than drop) message types other than threat and traffic. 16815 19375

  • Add support for timezone offsets and Z to decode_cef timestamp parser. 19346

  • Improve ECS categorization field mappings in traefik module. 16183 19379

  • Improve ECS categorization field mappings in azure module. 16155 19376

  • Add automatic retries and exponential backoff to httpjson input. 18956

  • Add text & flattened versions of fields with unknown subfields in aws cloudtrail fileset. 18866 19121

  • Added Microsoft Defender ATP Module. 17997 19197

  • Add initial support for configurable file identity tracking. 18748

  • Add experimental dataset tomcat/log for Apache TomCat logs 19713

  • Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs 19713

  • Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs 19713

  • Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs 19713

  • Add experimental dataset bluecoat/director for Bluecoat Director logs 19713

  • Add experimental dataset cisco/nexus for Cisco Nexus logs 19713

  • Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs 19713

  • Add experimental dataset cylance/protect for Cylance Protect logs 19713

  • Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs 19713

  • Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs 19713

  • Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs 19713

  • Add experimental dataset juniper/junos for Juniper Junos OS logs 19713

  • Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs 19713

  • Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs 19713

  • Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs 19713

  • Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs 19713

  • Add experimental dataset radware/defensepro for Radware DefensePro logs 19713

  • Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs 19713

  • Add experimental dataset squid/log for Squid Proxy Server logs 19713

  • Add experimental dataset zscaler/zia for Zscaler Internet Access logs 19713

  • Add support for reading auditd logs that are prefixed with node=. 19659

  • Add event.ingested for CrowdStrike module 20138

  • Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

  • Add event.ingested to all Filebeat modules. 20386

  • Add event.ingested for Suricata module 20220

  • Add support for custom header and headersecret for filebeat http_endpoint input 20435

  • Convert httpjson to v2 input 20226

  • Add event.ingested to all Filebeat modules. 20386

  • Return error when log harvester tries to open a named pipe. 18682 20450

  • Avoid goroutine leaks in Filebeat readers. 19193 20455

  • Improve Zeek x509 module with x509 ECS mappings 20867

  • Improve Zeek SSL module with x509 ECS mappings 20927

  • Added new properties field support for event.outcome in azure module 20998

  • Improve Zeek Kerberos module with x509 ECS mappings 20958

  • Improve Fortinet firewall module with x509 ECS mappings 20983

  • Improve Santa module with x509 ECS mappings 20976

  • Improve Suricata Eve module with x509 ECS mappings 20973

  • Added new module for Zoom webhooks 20414

  • Add type and sub_type to panw panos fileset 20912

  • Always attempt community_id processor on zeek module 21155

  • Add related.hosts ecs field to all modules 21160

  • Keep cursor state between httpjson input restarts 20751

  • Convert aws s3 to v2 input 20005

  • Add support for additional fields from V2 ALB logs. 21540

  • Release Cloud Foundry input as GA. 21525

  • New Cisco Umbrella dataset 21504

  • New juniper.srx dataset for Juniper SRX logs. 20017

  • Adding support for Microsoft 365 Defender (Microsoft Threat Protection) 21446

  • Adding support for FIPS in s3 input 21446

  • Add max_number_of_messages config into s3 input. 21993

Heartbeat

  • Allow a list of status codes for HTTP checks. 15587

  • Add additional ECS compatible fields for TLS information. 17687

  • Record HTTP response headers. 18327

  • Add index and pipeline settings to monitor configurations. 20610

Heartbeat

Journalbeat

  • Added an id config option to inputs to allow running multiple inputs on the same journal. 18467

  • Add basic ECS categorization and log.syslog fields. 19176

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add lambda metricset in aws module. 15260

  • Expand data for the system/memory metricset 15492

  • Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

  • Add cost warnings for the azure module. 15356

  • Add DynamoDB AWS Metricbeat light module 15097

  • Release elb module as GA. 15485

  • Add a system/network_summary metricset 15196

  • Add IBM MQ light-weight Metricbeat module 15301

  • Enable script processor. 14711

  • Add mixer metricset for Istio Metricbeat module 15696

  • Add mesh metricset for Istio Metricbeat modulehttps://github.com//pull/15535[15535]

  • Add pilot metricset for Istio Metricbeat module 15761

  • Add galley metricset for Istio Metricbeat module 15857

  • Add STAN dashboard 15654

  • Add key/value mode for SQL module. 15770 {pull]15845[15845]

  • Add support for Unix socket in Memcached metricbeat module. 13685 15822

  • Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

  • Add kubernetes storage class support via kube-state-metrics. 16145

  • Add up metric to prometheus metrics collected from host 15948

  • Add citadel metricset for Istio Metricbeat module 15990

  • Add support for processors in light modules. 14740 15923

  • Add collecting AuroraDB metrics in rds metricset. 14142 16004

  • Reuse connections in SQL module. 16001

  • Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

  • Add region parameter in googlecloud module. 15780 16203

  • Add database_account azure metricset. 15758

  • Add support for Dropwizard metrics 4.1. 16332

  • Add support for NATS 2.1. 16317

  • Add azure container metricset in order to monitor containers. 15751 16421

  • Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

  • Add filtering option for prometheus collector. 16420

  • Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

  • Add Load Balancing metricset to GCP 15559

  • Release statsd module as GA. 16447 14280

  • Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

  • Add OpenMetrics Metricbeat module 16596

  • Add redisenterprise module. 16482 15269

  • Add cloudfoundry module to send events from Cloud Foundry. 16671

  • Add system/users metricset as beta 16569

  • Align fields to ECS and add more tests for the azure module. 16024 16754

  • Add additional cgroup fields to docker/diskiohttps://github.com//pull/16638[16638]

  • Add overview dashboard for googlecloud compute metricset. 16534 16819

  • Add Prometheus remote write endpoint 16609

  • Release STAN module as GA. 16980

  • Add query metricset for prometheus module. 17104

  • Release ActiveMQ module as GA. 17047 17049

  • Add support for CouchDB v2 16352 16455

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Add dashboards for the azure container metricsets. 17194

  • Replace vpc metricset into vpn, transitgateway and natgateway metricsets. 16892

  • Use Elasticsearch histogram type to store Prometheus histograms 17061

  • Allow to rate Prometheus counters when scraping them 17061

  • Release Oracle module as GA. 14279 16833

  • Add Storage metricsets to GCP module 15598

  • Release vsphere module as GA. 15798 17119

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add test for documented fields check for metricsets without a http input. 17315 17334

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Refactor windows/perfmon metricset configuration options and event output. 17596

  • Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. 17429

  • Add state_statefulset metricset to Metricbeat recommended configuration for k8s. 17627

  • Add more detailed error messages, system tests and small refactoring to the service metricset in windows. 17725

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Add Metricbeat IIS module dashboards. 17966

  • Add dashboard for the azure database account metricset. 17901

  • Allow partial region and zone name in googlecloud module config. 17913

  • Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. [17141 17719

  • Move the perfmon metricset to GA. 16608 17879

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Add static mapping for metricsets under aws module. 17614 17650

  • Add dashboard for googlecloud storage metricset. 18172

  • Collect new bulk indexing metrics from Elasticsearch when xpack.enabled:true is set. https://github.com/elastic/beats/issues/ 17992

  • Remove requirement to connect as sysdba in Oracle module 15846 18182

  • Update MSSQL module to fix some SSPI authentication and add brackets to USE statements 17862]

  • Add client address to events from http server module 18336

  • Add memory metrics into compute googlecloud. 18802

  • Add new fields to HAProxy module. 18523

  • Add Tomcat overview dashboard 14026

  • Accept prefix as metric_types config parameter in googlecloud stackdriver metricset. 19345

  • Add dashboards for googlecloud load balancing metricset. 18369

  • Update Couchbase to version 6.5 18595 19055

  • Add support for v1 consumer API in Cloud Foundry module, use it by default. 19268

  • Add support for named ports in autodiscover. 19398

  • Add param aws_partition to support aws-cn, aws-us-gov regions. 18850 19423

  • Add support for wildcard * in dimension value of AWS CloudWatch metrics config. 18050 19660

  • The elasticsearch/index metricset now collects metrics for hidden indices as well. 18639 18703

  • Added performance and query metricsets to mysql module. 18955

  • The elasticsearch-xpack/index metricset now reports hidden indices as such. 18639 18706

  • Adds support for app insights metrics in the azure module. 18570 18940

  • Infer types in Prometheus remote_write. 19944

  • Added cache and connection_errors metrics to status metricset of MySQL module 16955 19844

  • Update MySQL dashboard with connection errors and cache metrics 19913 16955

  • Add cloud.instance.name into aws ec2 metricset. 20077

  • Add host inventory metrics into aws ec2 metricset. 20171

  • Add scope setting for elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. 18539 18547

  • Add state_daemonset metricset for Kubernetes Metricbeat module 20649

  • Add host inventory metrics to googlecloud compute metricset. 20391

  • Add host inventory metrics to azure compute_vm metricset. 20641

  • Add host inventory metrics to system module. 20415

  • Add billing data collection from Cost Explorer into aws billing metricset. 20527 20103

  • Migrate compute_vm metricset to a light one, map cloud.instance.id field. 20889

  • Request prometheus endpoints to be gzipped by default 20766

  • Add latency config parameter into aws module. 20875

  • Add billing metricset into googlecloud module. 20812 20738

  • Release all kubernetes state metricsets as GA 20901

  • Move compute_vm_scaleset to light metricset. 21038 20985

  • Sanitize event.host. 21022

  • Add overview and platform health dashboards to Cloud Foundry module. 21124

  • Release lambda metricset in aws module as GA. 21251 21255

  • Add dashboard for pubsub metricset in googlecloud module. 21326 17137

  • Move Prometheus query & remote_write to GA. 21507

  • Map cloud data filed cloud.account.id to azure subscription. 21483 21381

  • Expand unsupported option from namespace to metrics in the azure module. 21486

  • Move s3_daily_storage and s3_request metricsets to use cloudwatch input. 21703

Packetbeat

  • Add an example to packetbeat.yml of using the forwarded tag to disable host metadata fields when processing network data from network tap or mirror port. 19209

  • Add ECS fields for x509 certs, event categorization, and related IP info. 19167

  • Add 100-continue support 15830 19349

  • Add initial SIP protocol support 21221

Functionbeat

  • Add monitoring info about triggered functions. 14876

  • Add Google Cloud Platform support. 13598

  • Add basic ECS categorization and cloud fields. 19174

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add Audit and Log Management, Computer Object Management, and Distribution Group related events to the Security module. 15217

  • Add experimental event log reader implementation that should be faster in most cases. 6585 16849

  • Set process.command_line and process.parent.command_line from Sysmon Event ID 1. 17327

  • Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module 17517

  • Add registry and code signature information and ECS categorization fields for sysmon module 18058

Elastic Log Driver - Add support for docker logs command 19531 - Add support to change beat name, and support for Kibana Logs. 20522

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

  • Deprecate tags config parameter in cloudwatch metricset. 16733

  • Deprecate tags.resource_type_filter config parameter and replace with resource_type. 19688

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat