[Security Solution] Data Quality dashboard

[andrew-goldstein]

[Security Solution] Data Quality dashboard

Check ECS compatibility with just one click

With just one click, the Data Quality dashboard checks all the indices used by the Security Solution, (or anything else), for compatibility with the Elastic Common Schema (ECS)

Create cases from results

Create a single case containing all the results, or create cases for specific indices

Interactive tabs put results in context

Expand any index to reveal interactive tabs

• Summary
• Incompatible fields
• Custom fields
• ECS complaint fields
• All fields

Share comprehensive markdown reports

Share markdown reports containing the same content as the dashboard

On page load

When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected Data view are displayed

Only hot, warm, or unmanaged indices are displayed by default

Indices are not checked automatically when the dashboard loads

Click either :

• Check all to check all the indices on the page
• The expand button to automatically check (just) one index, and instantly view results

Check all

When the Check all button is clicked

• The Check all button changes to a Cancel button
• The Last checked: n <time unit> ago text is replaced with a progress bar indicating how many Indices are left to check
• The Checking <index name> text will update as each index is checked. Text will wrap if necessary
• The results tables begin updating with results
• Pattern stats update to summarize each table
• Rolled up results for the entire page update after every index is checked

check_all_trimmed.mov

When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.

Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.

While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.

When all checks complete, the page looks like this:

Take action

Click the Take action popover to share the entire page of results via one of the following actions:

• Add to new case
• Copy to clipboard

Expanding results

The Incompatible fields tab is always displayed by default when a result is expanded

The Incompatible fields tab shows a success message when a successful result is expanded

The Incompatible fields tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different

The Incompatible fields tab also compares field values expected by ECS vs the actual values in an index, when they are different

The Incompatible fields tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette

The calllout has a call to action to create a case or copy a markdown report for just the expanded result

• Add to new case
• Copy to clipboard

Tabs

The Summary tab displays a call to action when incompatible fields are found

Click on any part of the Summary tab chart or legend to navigate to the corresponding tab

Clicking on the Copy to clipboard call to action in the Custom fields tab copies a markdown version of the table to the clipboard

The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists

The All fields tab displays the union of all other tabs

Data view selection

The Data view dropdown defaults to the Security Default Data View

The alerts index is always checked and included in the results, even when another Data View is selected

ILM phase options

Only hot, warm, or unmanaged indices may be selected for checking.

The cold and frozen options are disabled.

When all options in the ILM phase box are cleared, an informative empty prompt is displayed

Errors

Errors may occur for some (or all) indices. The View errors button appears when the first error occurs

Users may click the View errors button to view them, even while a check is in progress

The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard

When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the Take action menu

Markdown reports

The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it

In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right

Stats rollups and tables are included in markdown reports

Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings

Navigation

The Data Quality dashboard is grouped with the existing Security Solution dashboards

It may also be launched via the side navigation

Privileges

The privileges in the table below are required to check any pattern of indices, or any specific index:

Privilege	Required to	Required for API
monitor or manage (manage builds on monitor)	List indices that match a pattern, and get document counts for an index example: GET logs-*/_stats	_stats
view_index_metadata or manage_ilm	List index ILM configs (e.g. hot) that match a pattern example: GET logs-*/_ilm/explain	_ilm/explain
view_index_metadata or manage	Get index mappings for a specific index example: GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping	_mapping
read or read_cross_cluster	Run aggregations to test for unallowed values example: GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search	_search

Users may have some of the privileges required to check an index, but not all of them.

The built-in viewer role does not have the monitor (or manage) role. The following screenshot illustrates what a user will see if they login as a user with the viewer role:

An actual markdown report (all content below)

The rest of the content below is pasted from an actual report, created via the Take action menu:

Data quality

Incompatible fields	Indices checked	Indices	Docs
17	15	17	1,404,514

.alerts-security.alerts-default

hot(1)

Incompatible fields	Indices checked	Indices	Docs
1	1	1	1,837

Result	Index	Docs	Incompatible fields	ILM Phase
❌	.internal.alerts-security.alerts-default-000001	1,837 (100.0%)	1	hot

.internal.alerts-security.alerts-default-000001

The .internal.alerts-security.alerts-default-000001 index has mappings or field values that are different than the Elastic Common Schema (ECS), version 8.6.0 definitions.

Result	Index	Docs	Incompatible fields	ILM Phase
❌	.internal.alerts-security.alerts-default-000001	1,837 (100.0%)	1	hot

Incompatible fields 1 Custom fields 188 ECS compliant fields 1219 All fields 1408

1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field values - .internal.alerts-security.alerts-default-000001

Field	ECS values (expected)	Document values (actual)
event.category	authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, vulnerability, web	behavior (62)

auditbeat-*

hot(11) unmanaged(1)

Incompatible fields	Indices checked	Indices	Docs
13	10	12	29,182

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.6.0-2023.01.17-000001	14,409 (49.4%)	0	hot
--	.ds-auditbeat-8.5.3-2023.01.24-000001	2,857 (9.8%)	--	hot
✅	.ds-auditbeat-8.2.3-2023.01.24-000001	2,246 (7.7%)	0	hot
✅	.ds-auditbeat-8.4.1-2023.01.24-000001	2,179 (7.5%)	0	hot
--	.ds-auditbeat-8.3.3-2023.01.24-000001	1,921 (6.6%)	--	hot
✅	auditbeat-7.16.0-2023.01.17-000001	1,880 (6.4%)	0	hot
✅	.ds-auditbeat-8.1.1-2023.01.24-000001	1,676 (5.7%)	0	hot
✅	.ds-auditbeat-8.2.2-2023.01.24-000001	1,578 (5.4%)	0	hot
✅	.ds-auditbeat-8.0.0-2023.01.24-000001	251 (0.9%)	0	hot
❌	auditbeat-7.10.2-2023.01.24-000001	111 (0.4%)	12	hot
✅	.ds-auditbeat-8.5.0-2023.01.24-000001	74 (0.3%)	0	hot
❌	auditbeat-custom-empty-index-1	0 (0.0%)	1	unmanaged

.ds-auditbeat-8.6.0-2023.01.17-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.6.0-2023.01.17-000001	14,409 (49.4%)	0	hot

Incompatible fields 0 Custom fields 549 ECS compliant fields 1210 All fields 1759

.ds-auditbeat-8.2.3-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.2.3-2023.01.24-000001	2,246 (7.7%)	0	hot

Incompatible fields 0 Custom fields 510 ECS compliant fields 1210 All fields 1720

.ds-auditbeat-8.4.1-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.4.1-2023.01.24-000001	2,179 (7.5%)	0	hot

Incompatible fields 0 Custom fields 509 ECS compliant fields 1210 All fields 1719

auditbeat-7.16.0-2023.01.17-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	auditbeat-7.16.0-2023.01.17-000001	1,880 (6.4%)	0	hot

Incompatible fields 0 Custom fields 523 ECS compliant fields 1111 All fields 1634

.ds-auditbeat-8.1.1-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.1.1-2023.01.24-000001	1,676 (5.7%)	0	hot

Incompatible fields 0 Custom fields 510 ECS compliant fields 1204 All fields 1714

.ds-auditbeat-8.2.2-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.2.2-2023.01.24-000001	1,578 (5.4%)	0	hot

Incompatible fields 0 Custom fields 510 ECS compliant fields 1210 All fields 1720

.ds-auditbeat-8.0.0-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.0.0-2023.01.24-000001	251 (0.9%)	0	hot

Incompatible fields 0 Custom fields 510 ECS compliant fields 1204 All fields 1714

auditbeat-7.10.2-2023.01.24-000001

The auditbeat-7.10.2-2023.01.24-000001 index has mappings or field values that are different than the Elastic Common Schema (ECS), version 8.6.0 definitions.

Result	Index	Docs	Incompatible fields	ILM Phase
❌	auditbeat-7.10.2-2023.01.24-000001	111 (0.4%)	12	hot

Incompatible fields 12 Custom fields 467 ECS compliant fields 602 All fields 1081

12 incompatible fields

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001

Field	ECS mapping type (expected)	Index mapping type (actual)
error.message	match_only_text	text
error.stack_trace	wildcard	keyword
http.request.body.content	wildcard	keyword
http.response.body.content	wildcard	keyword
message	match_only_text	text
process.command_line	wildcard	keyword
process.parent.command_line	wildcard	keyword
registry.data.strings	wildcard	keyword
url.full	wildcard	keyword
url.original	wildcard	keyword
url.path	wildcard	keyword

Incompatible field values - auditbeat-7.10.2-2023.01.24-000001

Field	ECS values (expected)	Document values (actual)
event.kind	alert, enrichment, event, metric, state, pipeline_error, signal	error (1)

.ds-auditbeat-8.5.0-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-auditbeat-8.5.0-2023.01.24-000001	74 (0.3%)	0	hot

Incompatible fields 0 Custom fields 509 ECS compliant fields 1210 All fields 1719

auditbeat-custom-empty-index-1

The auditbeat-custom-empty-index-1 index has mappings or field values that are different than the Elastic Common Schema (ECS), version 8.6.0 definitions.

Result	Index	Docs	Incompatible fields	ILM Phase
❌	auditbeat-custom-empty-index-1	0 (0.0%)	1	unmanaged

Incompatible fields 1 Custom fields 0 ECS compliant fields 0 All fields 0

1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field mappings - auditbeat-custom-empty-index-1

Field	ECS mapping type (expected)	Index mapping type (actual)
@timestamp	date	-

logs-*

hot(2)

Incompatible fields	Indices checked	Indices	Docs
3	2	2	602

Result	Index	Docs	Incompatible fields	ILM Phase
❌	.ds-logs-endpoint.alerts-default-2023.01.17-000001	342 (56.8%)	2	hot
❌	.ds-logs-endpoint.events.process-default-2023.01.17-000001	260 (43.2%)	1	hot

.ds-logs-endpoint.alerts-default-2023.01.17-000001

The .ds-logs-endpoint.alerts-default-2023.01.17-000001 index has mappings or field values that are different than the Elastic Common Schema (ECS), version 8.6.0 definitions.

Result	Index	Docs	Incompatible fields	ILM Phase
❌	.ds-logs-endpoint.alerts-default-2023.01.17-000001	342 (56.8%)	2	hot

Incompatible fields 2 Custom fields 857 ECS compliant fields 675 All fields 1534

2 incompatible fields

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001

Field	ECS mapping type (expected)	Index mapping type (actual)
process.env_vars	keyword	object

Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001

Field	ECS values (expected)	Document values (actual)
event.category	authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, vulnerability, web	behavior (45)

.ds-logs-endpoint.events.process-default-2023.01.17-000001

The .ds-logs-endpoint.events.process-default-2023.01.17-000001 index has mappings or field values that are different than the Elastic Common Schema (ECS), version 8.6.0 definitions.

Result	Index	Docs	Incompatible fields	ILM Phase
❌	.ds-logs-endpoint.events.process-default-2023.01.17-000001	260 (43.2%)	1	hot

Incompatible fields 1 Custom fields 130 ECS compliant fields 304 All fields 435

1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported

Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001

Field	ECS mapping type (expected)	Index mapping type (actual)
process.env_vars	keyword	object

packetbeat-*

hot(2)

Incompatible fields	Indices checked	Indices	Docs
0	2	2	1,372,893

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-packetbeat-8.6.0-2023.01.17-000001	704,062 (51.3%)	0	hot
✅	.ds-packetbeat-8.4.1-2023.01.24-000001	668,831 (48.7%)	0	hot

.ds-packetbeat-8.6.0-2023.01.17-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-packetbeat-8.6.0-2023.01.17-000001	704,062 (51.3%)	0	hot

Incompatible fields 0 Custom fields 604 ECS compliant fields 1209 All fields 1813

.ds-packetbeat-8.4.1-2023.01.24-000001

Result	Index	Docs	Incompatible fields	ILM Phase
✅	.ds-packetbeat-8.4.1-2023.01.24-000001	668,831 (48.7%)	0	hot

Incompatible fields 0 Custom fields 604 ECS compliant fields 1209 All fields 1813

Errors

Some indices were not checked for Data Quality

Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access

The following privileges are required to check an index:

• monitor or manage
• view_index_metadata
• read or read_cross_cluster

Pattern	Index	Error
.alerts-security.alerts-default	--	Error loading stats: Error: Forbidden
auditbeat-*	--	Error loading stats: Error: Forbidden
logs-*	--	Error loading stats: Error: Forbidden
packetbeat-*	--	Error loading stats: Error: Forbidden

See also: https://github.com/elastic/security-team/issues/4559

[andrew-goldstein]

Test plan: https://docs.google.com/document/d/14oTLA85lkpHdjOTm0FG03Vzpsdi5iMRHQMhBOlc6eTI/edit?usp=sharing

[angorayc]

Files owned by explore look good, thanks Andrew!

[andrew-goldstein]

Files by Code Owner

elastic/kibana-operations

• packages/kbn-babel-preset/styled_components_files.js
• packages/kbn-eslint-config/.eslintrc.js
• src/dev/precommit_hook/casing_check_config.js

elastic/security-solution

• x-pack/plugins/security_solution/common/constants.ts
• x-pack/plugins/security_solution/kibana.json
• x-pack/plugins/security_solution/public/app/deep_links/index.ts
• x-pack/plugins/security_solution/public/app/home/home_navigations.ts
• x-pack/plugins/security_solution/public/app/translations.ts
• x-pack/plugins/security_solution/public/common/components/navigation/types.ts
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/snapshots/index.test.tsx.snap
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/index.test.tsx
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/use_navigation_items.tsx
• x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx
• x-pack/plugins/security_solution/public/common/images/data_quality_dashboard_page.png
• x-pack/plugins/security_solution/public/landing_pages/links.ts
• x-pack/plugins/security_solution/public/overview/links.ts
• x-pack/plugins/security_solution/public/overview/pages/data_quality.tsx
• x-pack/plugins/security_solution/public/overview/pages/translations.ts
• x-pack/plugins/security_solution/public/overview/routes.tsx
• x-pack/plugins/security_solution/tsconfig.json

elastic/security-threat-hunting-explore

• x-pack/plugins/security_solution/public/common/components/navigation/types.ts
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/snapshots/index.test.tsx.snap
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/index.test.tsx
• x-pack/plugins/security_solution/public/common/components/navigation/use_security_solution_navigation/use_navigation_items.tsx
• x-pack/plugins/security_solution/public/overview/links.ts
• x-pack/plugins/security_solution/public/overview/pages/data_quality.tsx
• x-pack/plugins/security_solution/public/overview/pages/translations.ts
• x-pack/plugins/security_solution/public/overview/routes.tsx

[michaelolo24]

feel free to ignore, just don't need the ={true}

[michaelolo24]

nit

(${count})

[michaelolo24]

won't be checked

[michaelolo24]

maybe a silly question, but why do we default to showing timestamp as incompatible rather than not showing anything at all?

[michaelolo24]

Would it be worth it adding typeof EcsFlat to the union type for the ecsMetadata field to all of these functions to avoid this casting?

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 64456bb

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3586	3673	+87

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/ecs-data-quality-dashboard	-	1	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.9MB	13.6MB	⚠️ +688.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	52.4KB	52.9KB	+514.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/ecs-data-quality-dashboard	-	9	+9
ecsDataQualityDashboard	-	2	+2
total			+11

ESLint disabled line counts

id	before	after	diff
@kbn/ecs-data-quality-dashboard	-	2	+2
ecsDataQualityDashboard	-	5	+5
total			+7

miscellaneous assets size

id	before	after	diff
securitySolution	3.5MB	3.6MB	+82.4KB

Total ESLint disabled count

id	before	after	diff
@kbn/ecs-data-quality-dashboard	-	2	+2
ecsDataQualityDashboard	-	5	+5
total			+7

History

• 💚 Build #105859 succeeded 0c260c8
• 💚 Build #105427 succeeded 13fa411
• 💛 Build #105416 was flaky 73d1afa
• 💚 Build #105368 succeeded 7a74fe6
• 💚 Build #105345 succeeded 788d65a
• 💚 Build #104960 succeeded 578d147

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein

[nicpenning]

Wow this is amazing 👏 can't wait!!