Workaround for Beats issue with default_field growing too big #687
Conversation
There was a problem hiding this comment.
+1 on this addition. This is a "breaking" change to users which used this for the template but not for the fields itself. So I think I agree that it is only an addition.
@webmat I didn't check the fields that were marked with false, you might have an opinion here. Otherwise good to go.
There was a problem hiding this comment.
I'm fine with this toggle being driven from the ECS side for now. I'd like to have a few adjustments made before merging, however.
- Since this is Beats specific, I would like the attribute to reflect this. Perhaps simply
beats_default_fields, or if there's a possibility of requiring more Beats-specific toggles in the future, perhaps simply nest this underbeats:. I'm fine with both. - Please document the meaning of this new attribute in
schemas/README.md
Also kind of a higher level question. If I understand this correctly, this is meant to limit the amount of fields that are added to the beat templates' default_field setting? If that's the case, wouldn't you want to adjust this per Beats, instead of overall? Concrete example: default_fields:false on TLS seems detrimental for Packetbeat and Filebeat (with Suricata and Zeek modules), but seems to make sense for Metricbeat. Same goes for default_fields:false on process, which seems detrimental for Auditbeat in particular.
|
Note that I'm happy to adjust the generators a bit more around this as well. For example, generating 1 output file per Beat instead of a global one. |
This updates the Beats generator to undestand the `default_field` option and add it to the generated yaml, as well as update the fields added by 1.3.0 to include the `default_field` tag.
Co-Authored-By: Mathieu Martin <webmat@gmail.com>
This is so that Beats' default_fields don't go above 1000 field limit.
adriansr
force-pushed
the
beats_default_fields
branch
from
4c514ac
to
2bbebc0
Compare
|
@webmat I think I've addressed all your comments. Now it's using Your higher level suggestions make sense but those would require a bigger refactor in how Beats integrates fields from ECS. |
|
Yeah I understand why this is easier for now. I'm not clear on the semantics, however. This PR adds |
|
With the assumption that this is more or a temporary fix, I would like do this in a way that the files under schema/ never need to have changes. This way ECS contributors don't need to be burdened with understanding the effect of setting default_field or even what Beats are. This could be achieved by having a snapshot of the current fields we allow for Beats to go into default_field. Then in the generator for the Beat's fields.yml we "whitelist" those fields and for any other fields not in that list we give them |
|
Thanks @andrewkroh, picking this up now. Are you ok with me wording this (e.g. changelog, schema/readme, etc) specifically as a temporary workaround for Beats 7.6? |
|
Actually I'd put that wording in only for changelog. Because I agree with your assessment that this shouldn't burden ECS contributors, so I'll remove the readme mention. |
There was a problem hiding this comment.
After discussion with @andrewkroh, here's my thinking on this fix.
This is a temporary workaround. Leaving this as it is now would effectively mean that any field added after ECS 1.2 wouldn't be searchable from the search box until the end of the 7.x line. This is far from ideal. As such, Andrew reopened elastic/beats#14262 for further discussion. I'll chime in with ideas over there.
This situation is Beats-specific, and has no bearing on anyone else using ECS (Elastic Endpoint, partners, users building custom pipelines, etc.) So I've made this entirely self-contained in the generator for the Beats artifact. I want to avoid having the burden to always remember to add "beats.default_fields: false" explicitly for other field additions. This would likely have led to mistakes.
So this implementation is based on the YAML file scripts/generators/beats_default_fields_whitelist.yml. At this time it's populated only with ECS 1.2 fields, as was the original intention of this PR. You'll note that this also gives us the flexibility to strategically add a few more fields to the list, if needed.
I'll wait for a review by @andrewkroh or @ruflin before merging. But I'm good with this implementation now.
Note: the changes to "host" and "rule" definitions are simply trailing whitespaces getting nixed.
webmat
changed the title
…c#687) This is so that Beats' default_fields don't go above 1024 field limit. See also elastic/beats#14262
…c#687) This is so that Beats' default_fields don't go above 1024 field limit. See also elastic/beats#14262
…709) This is so that Beats' default_fields don't go above 1024 field limit. See also elastic/beats#14262
…710) This is so that Beats' default_fields don't go above 1024 field limit. See also elastic/beats#14262
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch Includes elastic/ecs#717 and elastic/ecs#687 not in released v1.4.0 - Remove field `process.exit_code` from Metricbeat, now in ECS.
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch Includes elastic/ecs#717 and elastic/ecs#687 not in released v1.4.0 - Remove field `process.exit_code` from Metricbeat, now in ECS. (cherry picked from commit cab56a3)
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch Includes elastic/ecs#717 and elastic/ecs#687 not in released v1.4.0 - Remove field `process.exit_code` from Metricbeat, now in ECS. (cherry picked from commit cab56a3)
…c#687) This is so that Beats' default_fields don't go above 1024 field limit. See also elastic/beats#14262
This updates the Beats generator to undestand the
default_fieldoption and add it to the generated yaml, as well as update the fields added by 1.3.0 to include thedefault_fieldtag.This change was discussed in elastic/beats#14844