-
Notifications
You must be signed in to change notification settings - Fork 419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[aws] Remove hardcoded event.dataset field and use ECS field instead #8811
Conversation
🚀 Benchmarks reportPackage
|
Data stream | Previous EPS | New EPS | Diff (%) | Result |
---|---|---|---|---|
waf |
5524.86 | 4032.26 | -1492.6 (-27.02%) | 💔 |
cloudfront_logs |
2881.84 | 2314.81 | -567.03 (-19.68%) | 💔 |
emr_logs |
29411.76 | 22727.27 | -6684.49 (-22.73%) | 💔 |
firewall_logs |
2544.53 | 1953.13 | -591.4 (-23.24%) | 💔 |
To see the full report comment with /test benchmark fullreport
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: If the only goal is to remove the hard-coded value, could the field data type still be retained as constant_keyword for performance reasons? The field value in the first event determines its value within a backing index. data_stream.dataset
works this way.
You can still use the ECS definition with a type override:
- external: ecs
name: event.dataset
type: constant_keyword
Should we not make similar changes in all Integration packages? |
@andrewkroh The issue that led to this bug requires Do you know in ECS is that why we are keeping |
+1 @agithomas That's also my next question. |
In #8810 it says
And it implies that If a data stream handles multiplexed data like perhaps cloudwatch logs does and the event.dataset is customized based on the log type then this would make sense to relax the constant_keyword type for that data stream. But for something like cloudtrail that only handles cloudtrail data I don't see any reason to relax the constant_keyword type for event.dataset. |
…es cloudwatch_logs
@andrewkroh Yes! Sorry I was only thinking about cloudwatch logs in the previous comment. Yes I will keep |
@kaiyan-sheng following list can be consider a candidate not to have type: constant_keyword? Also for my undertasting the |
@gizas Yes we don't need to add |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Package aws - 2.11.2 containing this change is available at https://epr.elastic.co/search?package=aws |
Proposed commit message
event.dataset
field and reference to the ECS field instead.The value for
event.dataset
is already applied from the elastic-agent side, please see this PR for more detail.With this change, when user tries to use a customized
dataset
value,event.dataset
should change as well.2. This PR also changes the default dataset name under Advanced option fromgeneric
toaws.cloudwatch_logs
.This is removed from the PR because I misunderstood why default is
generic
. In theory logs coming from cloudwatch should be rerouted to different data streams based on which service the log comes from. Whatever left in the default datastream are the ones not being routed.Checklist
changelog.yml
file.Related issues
event.dataset
is not correct ifdata_stream.dataset
is set to a customized value #8810