This changelog is intended for community Beat developers. It covers the major
breaking changes to the internal APIs in the official Beats and changes related
to developing a Beat like code generators or fields.yml
. Only the major
changes will be covered in this changelog that are expected to affect community
developers. Each breaking change added here should have an explanation on how
other Beats should be migrated.
Note: This changelog documents the current changes which are not yet present in an actual release.
The list below covers the major changes between 7.0.0-rc2 and main only.
-
Move Fields from package libbeat/common to libbeat/mapping. 11198
-
For "metricbeat style" generated custom beats, the mage target
GoTestIntegration
has changed toGoIntegTest
andGoTestUnit
has changed toGoUnitTest
. 13341 -
Build docker and kubernetes features only on supported platforms. 13509
-
Need to register new processors to be used in the JS processor in their
init
functions. 13509 -
The custom beat generator now uses mage instead of python,
mage GenerateCustomBeat
can be used to create a new beat, andmage vendorUpdate
to update the vendored libbeat in a custom beat. 13610 -
Altered all remaining uses of mapval to use the renamed and enhanced version: go-lookslike instead, which is a separate project. The mapval tree is now gone. 14165
-
Move light modules to OSS. 14369
-
Deprecate test flags,
generate
andupdate_expected
, in favor ofdata
. 15292 -
Python 3 is required now to run python tests and tools. 14798
-
The type
memqueue.Broker
is no longer exported; instead ofmemqueue.NewBroker
, callmemqueue.NewQueue
(which provides the same public interface). 16667 -
The disk spool types
spool.Spool
andspool.Settings
have been renamed to the internal typesspool.diskSpool
andspool.settings
. 16693 -
queue.Eventer
has been renamed toqueue.ACKListener
16691 -
Require logger as first parameter for
outputs.transport.transport#ProxyDialer
andoutputs.elasticsearch.client#BulkReadItemStatus
. 16761 -
The
libbeat/outputs/transport
package has been moved tolibbeat/common/transport
. 16734 -
The
libbeat/outputs/tls.go
file has been removed. All exported symbols in that file (libbeat/outputs.
) are now available aslibbeat/common/tlscommon.
. 16734 -
The newly generated Beats are using go modules to manage dependencies. 16288
-
Extract Elasticsearch client logic from
outputs/elasticsearch
package into newesclientleg
package. 16150 -
Rename
queue.BufferConfig.Events
toqueue.BufferConfig.MaxEvents
. 17622 -
Remove
queue.Feature
and replacequeue.RegisterType
withqueue.RegisterQueueType
. 17666 -
Introduce APM libbeat instrumentation.
Publish
method onClient
interface now takes a Context as first argument. 17938 -
The way configuration files are generated has changed to make it easier to customize parts of the config without requiring changes to libbeat config templates. Generation is now fully based on Go text/template and no longer uses file concatenation to generate the config. Your magefile.go will require a change to adapt the devtool API. See the pull request for more details. 18148
-
The Elasticsearch client settings expect the API key to be raw (not base64-encoded). 18939 18945
-
management.ConfigManager
has been renamed tomanagement.Manager
. 19114 -
UpdateStatus
has been added to themanagement.Manager
interface. 19114 -
Remove
common.MapStrPointer
parameter fromcfgfile.Runnerfactory
interface. 19135 -
Replace
ACKCount
,ACKEvents
, andACKLastEvent
callbacks withACKHandler
and interface inbeat.ClientConfig
. 19632 -
Remove global ACK handler support via
SetACKHandler
from publisher pipeline. 19632 -
Make implementing
Close
required forreader.Reader
interfaces. 20455 -
Remove
NumCPU
as clients should update the CPU count on the fly in case of config changes in a VM. 23154 -
Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. 25093
-
Update Darwin build image to a debian 10 base that increases the MacOS SDK and minimum supported version used in build to 10.14. 24193
-
Removed Beat generators. 28816
-
libbeat.logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573
-
Removed deprecated disk spool from Beats. Use disk queue instead. 28869
-
Wildcard fields no longer have a default ignore_above setting of 1024. 30096 30668
-
Remove
common.MapStr
and usemapstr.M
fromgithub.com/elastic/elastic-agent-libs
instead. 31420 -
Remove
queue.Consumer
. Queues can now be read via aGet
call directly on the queue object. 31502 -
The
queue.Batch
API now provides access to individual events instead of an array. 31699 -
Rename
queue.Batch.ACK()
toqueue.Batch.Done()
. 31903 -
queue.ACKListener
has been removed. Queue configurations now accept an explicit callback function for ACK handling. 35078 -
Split split httpmon out of x-pack/filebeat/input/internal/httplog. 36385
-
Beats publishing pipeline does not propagate the close signal to its clients any more. It’s responsibility of the user to close the pipeline client. 38197 38556
-
Debug log entries from the acker (
stateful ack …
orstateless ack …
) removed. 39672 -
Rename x-pack/filebeat websocket input to streaming. 40264 40421
-
Journald input now calls
journalctl
instead of usinggithub.com/coreos/go-systemd/v22@v22.5.0/sdjournal
, the CGO dependency has been removed from Filebeat 40061
-
Handle the starting of namespace and node watchers for metadata enrichment according to
add_resource_metadata
configuration.elastic#38762 -
Fix multiple metricbeat instances reporting same metrics when using autodiscover with provider kubernetes, and ensure leader elector is always running in autodiscover mode.elastic#38471
-
Fix how Prometheus histograms are calculated when percentiles are provide.elastic#36537
-
Stop using
mage:import
in community beats. This was ignoring the vendorized beats directory for some mage targets, using the code available in GOPATH, this causes inconsistencies and compilation problems if the version of the code in the GOPATH is different to the vendored one. Use ofmage:import
will continue to be unsupported in custom beats till beats is migrated to go modules, or mage supports vendored dependencies. 13998 14162 -
Metricbeat module builders call host parser only once when instantiating light modules. 20149
-
Fix export dashboard command when running against Elastic Cloud hosted Kibana. 22746
-
Remove
event.dataset
(ECS) annotion fromlibbeat.logp
. 27404 -
Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the
error
log level. 27804 -
Avoid panicking in
add_fields
processor when input event.Fields is a nil map. 28219 -
Drop event batch when get HTTP status 413 from Elasticsearch to avoid infinite loop 14350 29368
-
Allow to use metricbeat for named mssql instances. 24076 30859
-
Setting DEV=true when running
mage build
now correctly generates binaries without optimisations and with debug symbols 31955 -
The beat.cgroup.memory.mem.usage.bytes metric is now a gauge 31582 32652
-
Fix the integration testcase docker port mapping for sql and oracle modules 34221
-
Fix the ingest pipeline for mysql slowlog to parse schema name with dash 34372
-
Fix the multiple host support for mongodb module 34624
-
Fix ingest pipeline for panw module to parse url scheme correctly 35757
-
Renamed an httpjson input metric to follow naming conventions.
httpjson_interval_pages_total
was renamed tohttpjson_interval_pages
because the_total
suffix is reserved for counters. 35933 36169 -
Fixed some race conditions in tests 36185
-
Fix Stringer implementation of fingerprint processor 35174
-
Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. 36650
-
Cleaned up documentation errors & fixed a minor bug in Filebeat Azure blob storage input. 36714
-
Fix copy arguments for strict aligned architectures. 36976
-
Fix panic when more than 32767 pipeline clients are active. 38197 38556
-
Skip flakey metrics test on windows in filebeat httpjson input. 39676 39678
-
Fix flakey test on Windows 2022 in packetbeat/route. 39698 39822
-
Fix bug in minimum length for request trace logging. 39834
-
Close connections properly in Filbeat’s HTTPJSON input. 39790
-
Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. 39873 39653
-
Update watchers to be shared between metricsets in Kubernetes module. 37332
-
Add new metricset in Kubernetes module,
state_namespace
. 36406 -
Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938
-
Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline
-
Metricset generator generates beta modules by default now. 10657
-
The
beat.Event
accessor methods now support@metadata
keys. 10761 -
Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. 10921
-
Support for Logger in the Metricset base instance. 11106
-
Filebeat modules can now use ingest pipelines in YAML format. 11209
-
Prometheus helper for metricbeat contains now
Namespace
field forprometheus.MetricsMappings
11424 -
Update Jinja2 version to 2.10.1. 11817
-
Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. 11777,elastic#12065
-
Update urllib3 version to 1.24.2 11930
-
Add libbeat/common/cleanup package. 12134
-
New helper to check for leaked goroutines on tests. 12106
-
Only Load minimal template if no fields are provided. 12103
-
Add new option
IgnoreAllErrors
tolibbeat.common.schema
for skipping fields that failed while converting. 12089 -
Deprecate setup cmds for
template
andilm-policy
. Add new setup cmd forindex-management
. 12132 -
Use the go-lookslike library for testing in heartbeat. Eventually the mapval package will be replaced with it. 12540
-
New ReporterV2 interfaces that can receive a context on
Fetch(ctx, reporter)
, orRun(ctx, reporter)
. 11981 -
Generate configuration from
mage
for all Beats. 12618 -
Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. 8543
-
Introduce beat.OutputChooses publisher mode. 12996
-
Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. 12996
-
Add support to close beat.Client via beat.CloseRef (a subset of context.Context). 13031
-
Add checks for types and formats used in fields definitions in
fields.yml
files. 13188 -
Makefile included in generator copies files from beats repository using
git archive
instead of cp. 13193 -
Strip debug symbols from binaries to reduce binary sizes. 12768
-
Compare event by event in
testadata
framework to avoid sorting problems 13747 -
Added a
default_field
option to fields in fields.yml to offer a way to exclude fields from the default_field list. 14262 14341 -
supported-versions.yml
can be used in metricbeat python system tests to obtain the build args for docker compose builds. 14520 -
Fix dropped errors in the tests for the metricbeat Azure module. 13773
-
New mage target for Functionbeat: generate pkg folder to make manager easier. 15880
-
Add support for MODULE environment variable in
mage goIntegTest
in metricbeat to run integration tests for a single module. 17147 -
Add support for a
TEST_TAGS
environment variable to add tags for tests selection following go build tags semantics, this environment variable is used by mage test targets to add build tags. Python tests can also be tagged with a decorator (@beat.tag('sometag')
). 16937 17075 -
Add fields validation for histogram subfields. 17759
-
Add IP* fields to
fields.yml
generator script in Filebeat. 17998 18256 -
Events intended for the Elasticsearch output can now take an
op_type
metadata field of type events.OpType or string to indicate theop_type
to use for bulk indexing. 12606 -
Remove vendor folder from repository. 18655
-
Added SQL helper that can be used from any Metricbeat module 18955
-
Update Go version to 1.14.4. 19753
-
Update Go version to 1.14.7. 20508
-
Add packaging for docker image based on UBI minimal 8. 20576
-
Make the mage binary used by the build process in the docker container to be statically compiled. 20827
-
Add Pensando distributed firewall module. 21063
-
Update ecszap to v0.3.0 for using ECS 1.6.0 in logs 22267
-
Add support for customized monitoring API. 22605
-
Update Go version to 1.15.7. 22495
-
Update Go version to 1.15.8. 23955
-
Update Go version to 1.15.9. 24442
-
Update Go version to 1.15.10. 24606
-
Update Go version to 1.15.12. 25629
-
Add sorting to array fields for generated data files (*-generated.json) 25320
-
Update to go-concert 0.2.0 27162
-
Introduce
libbeat/beat.Beat.OutputConfigReloader
28048 -
Update Go version to 1.17.1. 27543
-
Whitelist
GCP_*
environment variables in dev tools 28364 -
Add support for
credentials_json
ingcp
module, all metricsets 29584 -
Add gcp firestore metricset. 29918
-
Added TESTING_FILEBEAT_FILEPATTERN option for filebeat module pytests 30103
-
Improve tests files with shorter statements. 35667
-
Add gcp dataproc metricset. 30008
-
Add Github action for linting
-
Add regex support for drop_fields processor.
-
Improve compatibility and reduce flakyness of Python tests 31588
-
Added
.python-version
file 32323 -
Use
T.TempDir
to create temporary test directory 33082 -
Add an option to disable event normalization when creating a
beat.Client
. 33657 -
Add the file path of the instance lock on the error when it’s is already locked 33788
-
Add DropFields processor to js API 33458
-
Add support for different folders when testing data 34467
-
Add logging of metric registration in inputmon. 35647
-
Add Okta API package for entity analytics. 35478
-
Add benchmarking to HTTPJSON input testing. 35138
-
Allow non-AWS endpoints for testing Filebeat awss3 input. 35496 35520
-
Add AUTH (username) and SSL/TLS support for Redis module 35240
-
Pin PyYAML version to 5.3.1 to avoid CI errors temporarily 36091
-
Skip dependabot updates for github.com/elastic/mito. 36158
-
Add device handling to Okta API package for entity analytics. 35980
-
Make Filebeat HTTPJSON input process responses sequentially. 36493
-
Add initial infrastructure for a caching enrichment processor. 36619
-
Add file-backed cache for cache enrichment processor. 36686 36696
-
Elide retryable HTTP client construction in Filebeat HTTPJSON and CEL inputs if not needed. 36916
-
Allow assignment of packetbeat protocols to interfaces. 36564 36852
-
Add Active Directory entity collector for Filebeat entity analytics. 37854
-
Make logs for empty and small files less noisy when using fingerprint file identity in filestream. 38421
-
Improve robustness and error reporting from packetbeat default route testing. 39757
-
Move x-pack/filebeat/input/salesforce jwt import to v5. 39823
-
Drop x-pack/filebeat/input dependency on github.com/lestrrat-go/jwx/v2. 39968
-
Added
ignore_empty_values
flag indecode_cef
Filebeat processor. 40268 -
Bump version of elastic/toutoumomoma to remove internal forks of stdlib debug packages. 40325
-
Refactor x-pack/filebeat/input/websocket for generalisation. 40308
-
Add a configuration option for TCP/UDP network type. 40407 40623
-
Added debug logging to parquet reader in x-pack/libbeat/reader. 40651