-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement setting to keep null values in events #13928
Implement setting to keep null values in events #13928
Conversation
Reviewers:
|
|
@urso If we make the setting per input then that implies it will only be available in Filebeat. However, other Beats could also generate events with |
You would need to add these setting to metricbeat/heartbeat module setup as well. We already do this for 'processors', 'fields', 'tags' settings. Each beat passes these setting along when calling |
===== `keep_null` | ||
|
||
If this option is set to true, fields with `null` values will be published in | ||
the output document. By default, `keep_null` is set to `false`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @dedemorton, I was wondering if you could review this doc snippet before I copy it to other Beats' docs. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
@urso This PR is ready for review again. It has become rather large in size, primarily due to changes to the reference yml and docs files. If you want I can pull those changes out into a separate follow-up PR to make it easier to review this one. Just let me know! |
@@ -3,3 +3,6 @@ | |||
enabled: true | |||
period: 10s | |||
hosts: ["localhost:3000"] | |||
|
|||
# Set to true to publish fields with null values in events. | |||
#keep_null: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer not to add this to the reference config of all modules, we don't do it with other common options like fields
, service.name
in metricbeat or pipeline
in filebeat.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in de6bb5d724212d844919b514480ab5cc2d732c32.
f0f3c60
to
de6bb5d
Compare
@@ -69,6 +69,7 @@ func NewInput( | |||
out, err := connector.ConnectWith(cfg, beat.ClientConfig{ | |||
Processing: beat.ProcessingConfig{ | |||
DynamicFields: inputContext.DynamicFields, | |||
KeepNull: conf.KeepNull, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
KeepNull will be set be automatically set by the pipelineConnector
in filebeat/channel/connector.go
. This makes me wonder if we need to set Processing
at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes me wonder if we need to set Processing at all.
I do think we need to set Processing here because of the DynamicFields config option. The pipelineConnector doesn't set it in its ConnectWith method.
makes sense.
@urso This is ready for another round of review, when you get a chance. Thanks! |
136ff57
to
2901545
Compare
2901545
to
a552fd2
Compare
Resolves #5522.
This PR proposes to add a new setting in Beats' configuration,
keep_null
, that will allow published events to containnull
values in them. The default value of this setting will befalse
, which preserves the current behavior of Beats (i.e. prior to this setting being introduced).Depending on the Beat, this setting can be specified at different places in the configuration.
Filebeat
Specify the setting under an input's configuration. For example:
With the above configuration, input text such as this:
Results in the following output event:
Notice, in particular, that the output event contains the
json.test-null
field whose value isnull
.The setting can also be specified under Filebeat modules'
input
configuration.Metricbeat
Specify the setting under a module's configuration. For example:
With the above configuration, the following HTTP request:
Results in the following output event:
Notice, in particular, that the output event contains the
http.server.bar
field whose value isnull
.Auditbeat
Specify the setting under a module's configuration. For example:
Packetbeat
Specify the setting under a protocol's configuration or the flows configuration. For example:
Winlogbeat
Specify the setting under an event log's configuration. For example:
Heartbeat
Specify the setting under a monitor's configuration. For example:
Functionbeat
Specify the setting under a function's configuration. For example: