[Security Solutions][Detection Engine] Adds a merge strategy key to kibana.yml and adds additional security keys to the Docker container that Elastic Security previously overlooked #103800
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…secuirty solution altogether
FrankHassanabad
changed the title
FrankHassanabad
added
v8.0.0
v7.14.0
auto-backport
FrankHassanabad
changed the title
jbudz
approved these changes
Jun 30, 2021
spong
reviewed
Jun 30, 2021
Comment on lines
+383
to
+392
| xpack.securitySolution.alertMergeStrategy | ||
| xpack.securitySolution.alertResultListDefaultDateRange | ||
| xpack.securitySolution.endpointResultListDefaultFirstPageIndex | ||
| xpack.securitySolution.endpointResultListDefaultPageSize | ||
| xpack.securitySolution.maxRuleImportExportSize | ||
| xpack.securitySolution.maxRuleImportPayloadBytes | ||
| xpack.securitySolution.maxTimelineImportExportSize | ||
| xpack.securitySolution.maxTimelineImportPayloadBytes | ||
| xpack.securitySolution.packagerTaskInterval | ||
| xpack.securitySolution.validateArtifactDownloads |
There was a problem hiding this comment.
Nice catching all these old entires and adding them as well! 🎁
spong
reviewed
Jun 30, 2021
| return mergeNoFields; | ||
| } | ||
| default: | ||
| return assertUnreachable(mergeStrategy); |
There was a problem hiding this comment.
++ on the exhaustive switch here. Nice errors coming from the kibana console as well when mis-configured 👍
spong
approved these changes
Jun 30, 2021
spong
added
Team: SecuritySolution
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 30, 2021
…ibana.yml and updates docker to have missing keys from security solutions (elastic#103800) ## Summary This is a follow up considered critical addition to: elastic#102280 This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions. The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are: * missingFields (The default) * allFields * noFields ## missingFields The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types. ## allFields A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy. Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array. ## noFields The behavior before elastic#102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well. The missing keys added for docker are: * xpack.securitySolution.alertMergeStrategy * xpack.securitySolution.alertResultListDefaultDateRange * xpack.securitySolution.endpointResultListDefaultFirstPageIndex * xpack.securitySolution.endpointResultListDefaultPageSize * xpack.securitySolution.maxRuleImportExportSize * xpack.securitySolution.maxRuleImportPayloadBytes * xpack.securitySolution.maxTimelineImportExportSize * xpack.securitySolution.maxTimelineImportPayloadBytes * xpack.securitySolution.packagerTaskInterval * xpack.securitySolution.validateArtifactDownloads I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us. ## Manual testing of the different strategies First add this mapping and document in the dev tools for basic tests ```json # Mapping with two constant_keywords and a runtime field DELETE frank-test-delme-17 PUT frank-test-delme-17 { "mappings": { "dynamic": "strict", "runtime": { "host.name": { "type": "keyword", "script": { "source": "emit('changed_hostname')" } } }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword" } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword", "value": "datastream_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "datastream_module_name_1" } } }, "event": { "properties": { "dataset": { "type": "constant_keyword", "value": "event_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "event_module_name_1" } } } } } } # Document without an existing host.name PUT frank-test-delme-17/_doc/1 { "@timestamp": "2021-06-30T15:46:31.800Z" } # Document with an existing host.name PUT frank-test-delme-17/_doc/2 { "@timestamp": "2021-06-30T15:46:31.800Z", "host": { "name": "host_name" } } # Query it to ensure the fields is returned with data that does not exist in _soruce GET frank-test-delme-17/_search { "fields": [ { "field": "*" } ] } ``` For all the different key combinations do the following: Run a single detection rule against the index: <img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png"> Ensure two signals are created: <img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png"> If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default): ```yml xpack.securitySolution.alertMergeStrategy: 'missingFields' ``` When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png"> However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now: First: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png"> Second: <img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'noFields' ``` Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`: <img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png"> Expect that `host.name` is missing in the first record and has the default value in the second: First: <img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png"> Second: <img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'allFields' ``` Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png"> Expect that both the first and second records contain the runtime field since we merge both of them: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 30, 2021
…ibana.yml and updates docker to have missing keys from security solutions (elastic#103800) ## Summary This is a follow up considered critical addition to: elastic#102280 This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions. The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are: * missingFields (The default) * allFields * noFields ## missingFields The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types. ## allFields A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy. Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array. ## noFields The behavior before elastic#102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well. The missing keys added for docker are: * xpack.securitySolution.alertMergeStrategy * xpack.securitySolution.alertResultListDefaultDateRange * xpack.securitySolution.endpointResultListDefaultFirstPageIndex * xpack.securitySolution.endpointResultListDefaultPageSize * xpack.securitySolution.maxRuleImportExportSize * xpack.securitySolution.maxRuleImportPayloadBytes * xpack.securitySolution.maxTimelineImportExportSize * xpack.securitySolution.maxTimelineImportPayloadBytes * xpack.securitySolution.packagerTaskInterval * xpack.securitySolution.validateArtifactDownloads I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us. ## Manual testing of the different strategies First add this mapping and document in the dev tools for basic tests ```json # Mapping with two constant_keywords and a runtime field DELETE frank-test-delme-17 PUT frank-test-delme-17 { "mappings": { "dynamic": "strict", "runtime": { "host.name": { "type": "keyword", "script": { "source": "emit('changed_hostname')" } } }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword" } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword", "value": "datastream_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "datastream_module_name_1" } } }, "event": { "properties": { "dataset": { "type": "constant_keyword", "value": "event_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "event_module_name_1" } } } } } } # Document without an existing host.name PUT frank-test-delme-17/_doc/1 { "@timestamp": "2021-06-30T15:46:31.800Z" } # Document with an existing host.name PUT frank-test-delme-17/_doc/2 { "@timestamp": "2021-06-30T15:46:31.800Z", "host": { "name": "host_name" } } # Query it to ensure the fields is returned with data that does not exist in _soruce GET frank-test-delme-17/_search { "fields": [ { "field": "*" } ] } ``` For all the different key combinations do the following: Run a single detection rule against the index: <img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png"> Ensure two signals are created: <img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png"> If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default): ```yml xpack.securitySolution.alertMergeStrategy: 'missingFields' ``` When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png"> However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now: First: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png"> Second: <img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'noFields' ``` Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`: <img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png"> Expect that `host.name` is missing in the first record and has the default value in the second: First: <img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png"> Second: <img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'allFields' ``` Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png"> Expect that both the first and second records contain the runtime field since we merge both of them: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
kibanamachine
added a commit
that referenced
this pull request
Jun 30, 2021
…ibana.yml and updates docker to have missing keys from security solutions (#103800) (#104020) ## Summary This is a follow up considered critical addition to: #102280 This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions. The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are: * missingFields (The default) * allFields * noFields ## missingFields The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types. ## allFields A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy. Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array. ## noFields The behavior before #102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well. The missing keys added for docker are: * xpack.securitySolution.alertMergeStrategy * xpack.securitySolution.alertResultListDefaultDateRange * xpack.securitySolution.endpointResultListDefaultFirstPageIndex * xpack.securitySolution.endpointResultListDefaultPageSize * xpack.securitySolution.maxRuleImportExportSize * xpack.securitySolution.maxRuleImportPayloadBytes * xpack.securitySolution.maxTimelineImportExportSize * xpack.securitySolution.maxTimelineImportPayloadBytes * xpack.securitySolution.packagerTaskInterval * xpack.securitySolution.validateArtifactDownloads I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us. ## Manual testing of the different strategies First add this mapping and document in the dev tools for basic tests ```json # Mapping with two constant_keywords and a runtime field DELETE frank-test-delme-17 PUT frank-test-delme-17 { "mappings": { "dynamic": "strict", "runtime": { "host.name": { "type": "keyword", "script": { "source": "emit('changed_hostname')" } } }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword" } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword", "value": "datastream_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "datastream_module_name_1" } } }, "event": { "properties": { "dataset": { "type": "constant_keyword", "value": "event_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "event_module_name_1" } } } } } } # Document without an existing host.name PUT frank-test-delme-17/_doc/1 { "@timestamp": "2021-06-30T15:46:31.800Z" } # Document with an existing host.name PUT frank-test-delme-17/_doc/2 { "@timestamp": "2021-06-30T15:46:31.800Z", "host": { "name": "host_name" } } # Query it to ensure the fields is returned with data that does not exist in _soruce GET frank-test-delme-17/_search { "fields": [ { "field": "*" } ] } ``` For all the different key combinations do the following: Run a single detection rule against the index: <img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png"> Ensure two signals are created: <img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png"> If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default): ```yml xpack.securitySolution.alertMergeStrategy: 'missingFields' ``` When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png"> However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now: First: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png"> Second: <img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'noFields' ``` Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`: <img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png"> Expect that `host.name` is missing in the first record and has the default value in the second: First: <img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png"> Second: <img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'allFields' ``` Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png"> Expect that both the first and second records contain the runtime field since we merge both of them: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker) Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Jun 30, 2021
…ibana.yml and updates docker to have missing keys from security solutions (#103800) (#104019) ## Summary This is a follow up considered critical addition to: #102280 This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions. The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are: * missingFields (The default) * allFields * noFields ## missingFields The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types. ## allFields A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy. Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array. ## noFields The behavior before #102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well. The missing keys added for docker are: * xpack.securitySolution.alertMergeStrategy * xpack.securitySolution.alertResultListDefaultDateRange * xpack.securitySolution.endpointResultListDefaultFirstPageIndex * xpack.securitySolution.endpointResultListDefaultPageSize * xpack.securitySolution.maxRuleImportExportSize * xpack.securitySolution.maxRuleImportPayloadBytes * xpack.securitySolution.maxTimelineImportExportSize * xpack.securitySolution.maxTimelineImportPayloadBytes * xpack.securitySolution.packagerTaskInterval * xpack.securitySolution.validateArtifactDownloads I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us. ## Manual testing of the different strategies First add this mapping and document in the dev tools for basic tests ```json # Mapping with two constant_keywords and a runtime field DELETE frank-test-delme-17 PUT frank-test-delme-17 { "mappings": { "dynamic": "strict", "runtime": { "host.name": { "type": "keyword", "script": { "source": "emit('changed_hostname')" } } }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword" } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword", "value": "datastream_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "datastream_module_name_1" } } }, "event": { "properties": { "dataset": { "type": "constant_keyword", "value": "event_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "event_module_name_1" } } } } } } # Document without an existing host.name PUT frank-test-delme-17/_doc/1 { "@timestamp": "2021-06-30T15:46:31.800Z" } # Document with an existing host.name PUT frank-test-delme-17/_doc/2 { "@timestamp": "2021-06-30T15:46:31.800Z", "host": { "name": "host_name" } } # Query it to ensure the fields is returned with data that does not exist in _soruce GET frank-test-delme-17/_search { "fields": [ { "field": "*" } ] } ``` For all the different key combinations do the following: Run a single detection rule against the index: <img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png"> Ensure two signals are created: <img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png"> If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default): ```yml xpack.securitySolution.alertMergeStrategy: 'missingFields' ``` When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png"> However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now: First: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png"> Second: <img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'noFields' ``` Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`: <img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png"> Expect that `host.name` is missing in the first record and has the default value in the second: First: <img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png"> Second: <img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'allFields' ``` Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png"> Expect that both the first and second records contain the runtime field since we merge both of them: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker) Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
madirey
pushed a commit
to madirey/kibana
that referenced
this pull request
Jul 6, 2021
…ibana.yml and updates docker to have missing keys from security solutions (elastic#103800) ## Summary This is a follow up considered critical addition to: elastic#102280 This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions. The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are: * missingFields (The default) * allFields * noFields ## missingFields The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types. ## allFields A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy. Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array. ## noFields The behavior before elastic#102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well. The missing keys added for docker are: * xpack.securitySolution.alertMergeStrategy * xpack.securitySolution.alertResultListDefaultDateRange * xpack.securitySolution.endpointResultListDefaultFirstPageIndex * xpack.securitySolution.endpointResultListDefaultPageSize * xpack.securitySolution.maxRuleImportExportSize * xpack.securitySolution.maxRuleImportPayloadBytes * xpack.securitySolution.maxTimelineImportExportSize * xpack.securitySolution.maxTimelineImportPayloadBytes * xpack.securitySolution.packagerTaskInterval * xpack.securitySolution.validateArtifactDownloads I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us. ## Manual testing of the different strategies First add this mapping and document in the dev tools for basic tests ```json # Mapping with two constant_keywords and a runtime field DELETE frank-test-delme-17 PUT frank-test-delme-17 { "mappings": { "dynamic": "strict", "runtime": { "host.name": { "type": "keyword", "script": { "source": "emit('changed_hostname')" } } }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword" } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword", "value": "datastream_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "datastream_module_name_1" } } }, "event": { "properties": { "dataset": { "type": "constant_keyword", "value": "event_dataset_name_1" }, "module": { "type": "constant_keyword", "value": "event_module_name_1" } } } } } } # Document without an existing host.name PUT frank-test-delme-17/_doc/1 { "@timestamp": "2021-06-30T15:46:31.800Z" } # Document with an existing host.name PUT frank-test-delme-17/_doc/2 { "@timestamp": "2021-06-30T15:46:31.800Z", "host": { "name": "host_name" } } # Query it to ensure the fields is returned with data that does not exist in _soruce GET frank-test-delme-17/_search { "fields": [ { "field": "*" } ] } ``` For all the different key combinations do the following: Run a single detection rule against the index: <img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png"> Ensure two signals are created: <img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png"> If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default): ```yml xpack.securitySolution.alertMergeStrategy: 'missingFields' ``` When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png"> However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now: First: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png"> Second: <img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'noFields' ``` Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`: <img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png"> Expect that `host.name` is missing in the first record and has the default value in the second: First: <img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png"> Second: <img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png"> When you set in your `kibana.yml` or `kibana.dev.yml` this key: ```yml xpack.securitySolution.alertMergeStrategy: 'allFields' ``` Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`: <img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png"> Expect that both the first and second records contain the runtime field since we merge both of them: <img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](https://github.com/elastic/kibana/blob/c29adfef29e921cc447d2a5ed06ac2047ceab552/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
FrankHassanabad
added
release_note:fix
and removed
release_note:feature
FrankHassanabad
changed the title
💔 Build Failed
Failed CI Steps
Test FailuresKibana Pipeline / general / task-queue-process-4 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/metadata·ts.Endpoint plugin test metadata api POST /api/endpoint/metadata when index is not empty metadata api should return one entry for each host with default pagingStandard OutStack TraceKibana Pipeline / general / task-queue-process-4 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/metadata·ts.Endpoint plugin test metadata api POST /api/endpoint/metadata when index is not empty metadata api should return one entry for each host with default pagingStandard OutStack TraceKibana Pipeline / general / Chrome UI Functional Tests.test/functional/apps/visualize/_vega_chart·ts.visualize app visualize ciGroup12 vega chart in visualize app vega chart initial render should have view and control containersStandard OutStack Traceand 2 more failures, only showing the first 3. Metrics [docs]
History
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is a follow up considered critical addition to:
#102280
This adds a key of
xpack.securitySolution.alertMergeStrategytokibana.ymlwhich allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions.The values you can use and add to to
xpack.securitySolution.alertMergeStrategyare:missingFields
The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the fields API into the resulting signal/alert. This will copy over fields such as
constant_keyword,copy_to,runtime fields,field aliaseswhich previously were not copied over as long as they are primitive data types such askeyword,text,numericand are not found in your original_sourcedocument. This will not copy copygeo points,nested objects, and in some cases if your_sourcecontains arrays or top level objects or conflicts/ambiguities it will not merge them. This will not merge existing values between_sourceandfieldsforruntime fieldsas well. It only merges missing primitive data types.allFields
A very aggressive merge strategy which should be considered experimental. It will do everything
missingFieldsdoes but in addition to that it will merge existing values between_sourceandfieldswhich means if you change values or override values withruntime fieldsthis strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not mergegeodata types due to ambiguities. If you have multi-fields this will choose your default field and merge that into_source. This can change a lot your data between your original_sourceandfieldswhen the data is copied into an alert/signal which is why it is considered an aggressive merge strategy.Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in
_sourceor if it sees multiple elements within an array.noFields
The behavior before #102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with
missingFields. We are not anticipating this, but if you are settingnoFieldsplease reach out to our forums and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well.The missing keys added for docker are:
I intentionally skipped adding the other
kibana.ymlkeys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us.Manual testing of the different strategies
First add this mapping and document in the dev tools for basic tests
For all the different key combinations do the following:
Run a single detection rule against the index:
Ensure two signals are created:
If your
kibana.ymlorkibana.dev.ymlyou set this key (or omit it as it is the default):When you click on each signal you should see that
event.moduleandevent.datasetwere copied over as well asdata_stream.datasetanddata_stream.modulesince they'reconstant_keyword:However since this only merges missing fields, you should see that in the first record the
host.nameis the runtime field defined sincehost.namedoes not exist in_sourceand that in the second record it still shows up ashost_namesince we do not override merges right now:First:
Second:
When you set in your
kibana.ymlorkibana.dev.ymlthis key:Expect that your
event.module,event.dataset,data_stream.module,data_stream.datasetare all non-existent since we do not copy anything over fromfieldsat all and only use things within_source:Expect that
host.nameis missing in the first record and has the default value in the second:First:
Second:
When you set in your
kibana.ymlorkibana.dev.ymlthis key:Expect that
event.moduleandevent.datasetwere copied over as well asdata_stream.datasetanddata_stream.modulesince they'reconstant_keyword:Expect that both the first and second records contain the runtime field since we merge both of them:
Checklist
Delete any items that are not applicable to this PR.