-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update to ECS 1.4.0 #14844
Update to ECS 1.4.0 #14844
Conversation
jenkins, test this |
e33a16a
to
2b6288a
Compare
@@ -30,7 +30,7 @@ import ( | |||
// MaxDefaultFieldLength is the limit on the number of default_field values | |||
// allowed by the test. This is less that the 1024 limit of Elasticsearch to | |||
// give a little room for custom fields and the expansion of `fields.*`. | |||
const MaxDefaultFieldLength = 1000 | |||
const MaxDefaultFieldLength = 1010 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about this change. It was complaining that there were 1006 fields after the ECS update.
Will exceed 1000 have an impact or as long as it's below 1024 we're safe?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We configure the total_fields.limit
per index to 10000. The default_fields limit is impacted by the indices.query.bool.max_clause_count
setting (default 1024, but not modifyable via templates).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andrewkroh I think touched this in the past?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One option to deal with this would be to add default_field: false
to the new fields being added in ECS. This problem will keep occurring each time new fields are added and this could be the solution until we move to a better indexing strategy. Bumping the limit up won't work for much longer.
f922aee
to
f1ae817
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall I'm good with this change. We should keep in mind that this keeps increasing the size of all templates even thought he fields are not used in winlogbeat for example.
I didn't check CI but I think we should ;-)
f1ae817
to
e16d24b
Compare
Updated to ECS 1.3.1 |
@adriansr What are your thoughts around the field limit (see comment from @andrewkroh ). |
I misunderstood the I can add them to the new fields this PR adds, but this implies a change in ECS (to add I don't mind doing either. Edit: Created a PR to elastic/ecs with the change: elastic/ecs#687 |
e16d24b
to
781eab6
Compare
This is still a work in progress as it currently depends on adding the A new ECS release (1.4.0?) will be needed after that. |
elastic/ecs#687 has been merged and backported to branch 1.3. |
781eab6
to
a093d11
Compare
e007571
to
a87acbc
Compare
libbeat/_meta/fields.ecs.yml
Outdated
@@ -1,5 +1,5 @@ | |||
# WARNING! Do not edit this file directly, it was generated by the ECS project, | |||
# based on ECS version 1.4.0. | |||
# based on ECS version 1.5.0-dev. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't forget to grab the version from elastic/ecs#718 ;-)
Jenkins, test this |
04a4fc8
to
3afe17d
Compare
Jenkins, test this |
This reverts commit aa563245e8ac232c334cf90cba1cf31ced181439.
3afe17d
to
335be5f
Compare
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch Includes elastic/ecs#717 and elastic/ecs#687 not in released v1.4.0 - Remove field `process.exit_code` from Metricbeat, now in ECS. (cherry picked from commit cab56a3)
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch Includes elastic/ecs#717 and elastic/ecs#687 not in released v1.4.0 - Remove field `process.exit_code` from Metricbeat, now in ECS. (cherry picked from commit cab56a3)
Updated vendored elastic/ecs to
v1.3.0v1.3.1v1.4.0 and fields.ecs.yml from same version.