Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Data Quality dashboard (elastic#150063)
# [Security Solution] Data Quality dashboard ## Check ECS compatibility with just one click With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)  ## Create cases from results Create a single case containing all the results, or create cases for specific indices  ## Interactive tabs put results in context Expand any index to reveal interactive tabs - Summary - Incompatible fields - Custom fields - ECS complaint fields - All fields  ## Share comprehensive markdown reports Share markdown reports containing the same content as the dashboard  ### On page load When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed  Only `hot`, `warm`, or `unmanaged` indices are displayed by default Indices are not checked automatically when the dashboard loads Click either : - `Check all` to check all the indices on the page - The expand button to automatically check (just) one index, and instantly view results ### Check all When the `Check all` button is clicked - The `Check all` button changes to a `Cancel` button - The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check - The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary - The results tables begin updating with results - Pattern stats update to summarize each table - Rolled up results for the entire page update after every index is checked  <https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov> When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check. Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked. While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens. When all checks complete, the page looks like this:  ### Take action Click the `Take action` popover to share the entire page of results via one of the following actions: - Add to new case - Copy to clipboard    ### Expanding results The `Incompatible fields` tab is always displayed by default when a result is expanded The `Incompatible fields` tab shows a success message when a successful result is expanded  The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different  The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different  The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette <img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png"> The calllout has a call to action to create a case or copy a markdown report for just the expanded result - Add to new case - Copy to clipboard  ### Tabs The Summary tab displays a call to action when incompatible fields are found Click on any part of the Summary tab chart or legend to navigate to the corresponding tab  Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard  The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists  The All fields tab displays the union of all other tabs  ### Data view selection The `Data view` dropdown defaults to the `Security Default Data View`  The alerts index is always checked and included in the results, even when another Data View is selected  ### ILM phase options  Only `hot`, `warm`, or `unmanaged` indices may be selected for checking. The `cold` and `frozen` options are disabled. When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed  ### Errors Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs  Users may click the `View errors` button to view them, even while a check is in progress  The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu ### Markdown reports The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right  Stats rollups and tables are included in markdown reports  Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings ### Navigation The Data Quality dashboard is grouped with the existing Security Solution dashboards  It may also be launched via the side navigation  ## Privileges The privileges in the table below are required to check any pattern of indices, or any specific index: | Privilege | Required to | Required for API | |-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------| | `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` | | `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` | | `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` | | `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` | Users may have some of the privileges required to check an index, but not all of them. The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:  # An actual markdown report (all content below) The rest of the content below is pasted from an actual report, created via the `Take action` menu: # Data quality | Incompatible fields | Indices checked | Indices | Docs | |---------------------|-----------------|---------|------| | 17 | 15 | 17 | 1,404,514 | ## .alerts-security.alerts-default `hot(1)` | Incompatible fields | Indices checked | Indices | Docs | |---------------------|-----------------|---------|------| | 1 | 1 | 1 | 1,837 | | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` | ### .internal.alerts-security.alerts-default-000001 The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html). | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` | ### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408` #### 1 incompatible field Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0. ❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported #### Incompatible field values - .internal.alerts-security.alerts-default-000001 | Field | ECS values (expected) | Document values (actual) | |-------|-----------------------|--------------------------| | event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) | ## auditbeat-* `hot(11)` `unmanaged(1)` | Incompatible fields | Indices checked | Indices | Docs | |---------------------|-----------------|---------|------| | 13 | 10 | 12 | 29,182 | | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` | | -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` | | ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` | | ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` | | -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` | | ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` | | ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` | | ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` | | ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` | | ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` | | ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` | | ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` | ### .ds-auditbeat-8.6.0-2023.01.17-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759` ### .ds-auditbeat-8.2.3-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720` ### .ds-auditbeat-8.4.1-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719` ### auditbeat-7.16.0-2023.01.17-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634` ### .ds-auditbeat-8.1.1-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714` ### .ds-auditbeat-8.2.2-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720` ### .ds-auditbeat-8.0.0-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714` ### auditbeat-7.10.2-2023.01.24-000001 The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html). | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` | ### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081` #### 12 incompatible fields Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0. ❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported #### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | error.message | `match_only_text` | `text` | | error.stack_trace | `wildcard` | `keyword` | | http.request.body.content | `wildcard` | `keyword` | | http.response.body.content | `wildcard` | `keyword` | | message | `match_only_text` | `text` | | process.command_line | `wildcard` | `keyword` | | process.parent.command_line | `wildcard` | `keyword` | | registry.data.strings | `wildcard` | `keyword` | | url.full | `wildcard` | `keyword` | | url.original | `wildcard` | `keyword` | | url.path | `wildcard` | `keyword` | #### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001 | Field | ECS values (expected) | Document values (actual) | |-------|-----------------------|--------------------------| | event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) | ### .ds-auditbeat-8.5.0-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719` ### auditbeat-custom-empty-index-1 The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html). | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` | ### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0` #### 1 incompatible field Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0. ❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported #### Incompatible field mappings - auditbeat-custom-empty-index-1 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | @timestamp | `date` | `-` | ## logs-* `hot(2)` | Incompatible fields | Indices checked | Indices | Docs | |---------------------|-----------------|---------|------| | 3 | 2 | 2 | 602 | | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` | | ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` | ### .ds-logs-endpoint.alerts-default-2023.01.17-000001 The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html). | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` | ### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534` #### 2 incompatible fields Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0. ❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported #### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | process.env_vars | `keyword` | `object` | #### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001 | Field | ECS values (expected) | Document values (actual) | |-------|-----------------------|--------------------------| | event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) | ### .ds-logs-endpoint.events.process-default-2023.01.17-000001 The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html). | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` | ### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435` #### 1 incompatible field Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0. ❌ Detection engine rules referencing these fields may not match them correctly ❌ Pages may not display some events or fields due to unexpected field mappings or values ❌ Mappings or field values that don't comply with ECS are not supported #### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | process.env_vars | `keyword` | `object` | ## packetbeat-* `hot(2)` | Incompatible fields | Indices checked | Indices | Docs | |---------------------|-----------------|---------|------| | 0 | 2 | 2 | 1,372,893 | | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` | | ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` | ### .ds-packetbeat-8.6.0-2023.01.17-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813` ### .ds-packetbeat-8.4.1-2023.01.24-000001 | Result | Index | Docs | Incompatible fields | ILM Phase | |--------|-------|------|---------------------|-----------| | ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` | ### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813` ## Errors Some indices were not checked for Data Quality Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access The following privileges are required to check an index: - `monitor` or `manage` - `view_index_metadata` - `read` or `read_cross_cluster` | Pattern | Index | Error | |---------|-------|-------| | .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` | | auditbeat-* | -- | `Error loading stats: Error: Forbidden` | | logs-* | -- | `Error loading stats: Error: Forbidden` | | packetbeat-* | -- | `Error loading stats: Error: Forbidden` | See also: elastic/security-team#4559
- Loading branch information
1 parent
55b0948
commit 63ec95d