Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch most fields to keyword as the type #137

Merged
merged 1 commit into from
Oct 19, 2018

Conversation

ruflin
Copy link
Member

@ruflin ruflin commented Oct 19, 2018

We started to use multi fields in ECS and if we used a multi field, the base field was text as is the default in Elasticsearch. The problem with this is we decide in the future to make a field multi field, it would be breaking change as what was before a keyword now becomes text. To prevent this all fields except message are by default a keyword. Making a multifield out of a field is then a non breaking change.

On the ECS side we still need to figure out what our recommendation is for naming multiple fields like .analyzed, .text or others.

This change has also an affect on Beats as in ttps://github.com/elastic/beats/pull/8313 the fields.yml from ECS was added to Beats. There was even a breaking change I think we missed when switch to ECS there as the http.response.body in packetbeat was text and in Metricbeat keyword.

The following fields were changed to keyword and multifield removed for now. We can add it later again when we figure out the convention:

  • device.vendor
  • file.path
  • file.target_path
  • http.response.body
  • network.name
  • organization.name
  • url.href
  • url.path
  • url.query
  • user_agent.original

We started to use multi fields in ECS and if we used a multi field, the base field was text as is the default in Elasticsearch. The problem with this is we decide in the future to make a field multi field, it would be breaking change as what was before a keyword now becomes text. To prevent this all fields except `message` are by default a keyword. Making a multifield out of a field is then a non breaking change.

On the ECS side we still need to figure out what our recommendation is for naming multiple fields like `.analyzed, .text` or others.

This change has also an affect on Beats as in ttps://github.com/elastic/beats/pull/8313 the fields.yml from ECS was added to Beats. There was even a breaking change I think we missed when switch to ECS there as the `http.response.body` in packetbeat was text and in Metricbeat keyword.

The following fields were changed to keyword and multifield removed for now. We can add it later again when we figure out the convention:

* device.vendor
* file.path
* file.target_path
* http.response.body
* network.name
* organization.name
* url.href
* url.path
* url.query
* user_agent.original
@ruflin ruflin added the review label Oct 19, 2018
@ruflin
Copy link
Member Author

ruflin commented Oct 19, 2018

@adriansr After this is merged we should update the ECS fields.yml in Beats again and mention the breaking change that happened in Packetbeat.

@ruflin ruflin changed the title Switch most fields to keyword as the base type Switch most fields to keyword as the type Oct 19, 2018
@ruflin ruflin requested a review from webmat October 19, 2018 12:08
Copy link
Contributor

@webmat webmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'll rewrite the readme section on multi-field in a separate PR. May be worthwhile to explicitly mention which two fields are left as text in the changelog. But I will also do this in that other PR.

@webmat webmat merged commit d0cc4f5 into elastic:master Oct 19, 2018
@webmat webmat mentioned this pull request Oct 19, 2018
26 tasks
webmat pushed a commit to webmat/ecs that referenced this pull request Oct 19, 2018
webmat pushed a commit to webmat/ecs that referenced this pull request Oct 23, 2018
webmat pushed a commit to webmat/ecs that referenced this pull request Oct 23, 2018
webmat pushed a commit to webmat/ecs that referenced this pull request Oct 24, 2018
webmat added a commit that referenced this pull request Oct 24, 2018
…ADME. (#140)

* Introduce the new convention for multi-fields text indexing to the README.
* Be a little more explicit in the changelog for #137
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants