-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Data Quality Dashboard persistence #175673
[Security Solution] Data Quality Dashboard persistence #175673
Conversation
Pinging @elastic/security-threat-hunting (Team:Threat Hunting) |
Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Manual tested for that race bug and it is gone, woohoo! LGTM, great work @semd
@elasticmachine merge upstream |
@elasticmachine merge upstream |
As agreed offline with @andrew-goldstein, "Last check" column has been added: |
x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts
Show resolved
Hide resolved
@elasticmachine merge upstream |
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/post_results.ts
Outdated
Show resolved
Hide resolved
@elasticmachine merge upstream |
.../security-solution/ecs_data_quality_dashboard/impl/data_quality/use_results_rollup/index.tsx
Show resolved
Hide resolved
Consider if the following screenshot, captured after refreshing the Data Quality dashboard, demonstrates unexpected behavior:
When the following console.log(
`--> getResultsRoute query for pattern ${pattern}`,
JSON.stringify(query, null, 2)
); is added to
the query appears to include the Click meresponse (for the query above, via dev tools){
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 5,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"aggregations": {
"latest": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": ".ds-logs-endpoint.alerts-default-2023.11.26-000002",
"doc_count": 1,
"latest_doc": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
"_id": "KFhPVo0BLR4mG9DskP35",
"_score": null,
"_source": {
"@timestamp": 1706550071544,
"batchId": "16ab03d8-3a83-490a-a938-f5656b9a50e6",
"indexName": ".ds-logs-endpoint.alerts-default-2023.11.26-000002",
"isCheckAll": false,
"checkedAt": 1706550071360,
"docsCount": 1406,
"totalFieldCount": 1582,
"ecsFieldCount": 677,
"customFieldCount": 904,
"incompatibleFieldCount": 1,
"sameFamilyFieldCount": 0,
"sameFamilyFields": [],
"unallowedMappingFields": [],
"unallowedValueFields": [
"event.category"
],
"sizeInBytes": 2560371,
"ilmPhase": "hot",
"markdownComments": [
"""### .ds-logs-endpoint.alerts-default-2023.11.26-000002
""",
"""| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.11.26-000002 | 1,406 (30.8%) | 1 | `hot` | 7.9MB |
""",
"""### **Incompatible fields** `1` **Same family** `0` **Custom fields** `904` **ECS compliant fields** `677` **All fields** `1582`
""",
"""#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
"""
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.11.26-000002
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (141) |
"""
],
"ecsVersion": "8.6.1",
"indexId": "-5OpMG4tSGOidPOGDRYoAg",
"error": null
},
"sort": [
1706550071544
]
}
]
}
}
},
{
"key": ".ds-logs-endpoint.alerts-default-2024.01.08-000003",
"doc_count": 1,
"latest_doc": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
"_id": "11gZVo0BLR4mG9DsLPoE",
"_score": null,
"_source": {
"@timestamp": 1706546506754,
"batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
"indexName": ".ds-logs-endpoint.alerts-default-2024.01.08-000003",
"isCheckAll": true,
"checkedAt": 1706546506575,
"docsCount": 466,
"totalFieldCount": 1582,
"ecsFieldCount": 677,
"customFieldCount": 904,
"incompatibleFieldCount": 1,
"sameFamilyFieldCount": 0,
"sameFamilyFields": [],
"unallowedMappingFields": [],
"unallowedValueFields": [
"event.category"
],
"sizeInBytes": 795964,
"ilmPhase": "hot",
"markdownComments": [
"""### .ds-logs-endpoint.alerts-default-2024.01.08-000003
""",
"""| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.alerts-default-2024.01.08-000003 | 466 (10.2%) | 1 | `hot` | 777.3KB |
""",
"""### **Incompatible fields** `1` **Same family** `0` **Custom fields** `904` **ECS compliant fields** `677` **All fields** `1582`
""",
"""#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
"""
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2024.01.08-000003
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (48) |
"""
],
"ecsVersion": "8.6.1",
"indexId": "frzzVLeRSSSbNHxbNM6FqQ",
"error": null
},
"sort": [
1706546506754
]
}
]
}
}
},
{
"key": ".ds-logs-endpoint.events.process-default-2023.10.27-000001",
"doc_count": 1,
"latest_doc": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
"_id": "1lgZVo0BLR4mG9DsHvr2",
"_score": null,
"_source": {
"@timestamp": 1706546503412,
"batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
"indexName": ".ds-logs-endpoint.events.process-default-2023.10.27-000001",
"isCheckAll": true,
"checkedAt": 1706546503245,
"docsCount": 553,
"totalFieldCount": 446,
"ecsFieldCount": 304,
"customFieldCount": 141,
"incompatibleFieldCount": 1,
"sameFamilyFieldCount": 0,
"sameFamilyFields": [],
"unallowedMappingFields": [],
"unallowedValueFields": [
"event.outcome"
],
"sizeInBytes": 1063145,
"ilmPhase": "hot",
"markdownComments": [
"""### .ds-logs-endpoint.events.process-default-2023.10.27-000001
""",
"""| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.10.27-000001 | 553 (12.1%) | 1 | `hot` | 1MB |
""",
"""### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
"""#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
"""
#### Incompatible field values - .ds-logs-endpoint.events.process-default-2023.10.27-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (491) |
"""
],
"ecsVersion": "8.6.1",
"indexId": "4DeVaFmaTWyXlHQ7IrVvCA",
"error": null
},
"sort": [
1706546503412
]
}
]
}
}
},
{
"key": ".ds-logs-endpoint.events.process-default-2023.11.26-000002",
"doc_count": 1,
"latest_doc": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
"_id": "1VgZVo0BLR4mG9DsEvoz",
"_score": null,
"_source": {
"@timestamp": 1706546500145,
"batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
"indexName": ".ds-logs-endpoint.events.process-default-2023.11.26-000002",
"isCheckAll": true,
"checkedAt": 1706546499983,
"docsCount": 1020,
"totalFieldCount": 446,
"ecsFieldCount": 304,
"customFieldCount": 141,
"incompatibleFieldCount": 1,
"sameFamilyFieldCount": 0,
"sameFamilyFields": [],
"unallowedMappingFields": [],
"unallowedValueFields": [
"event.outcome"
],
"sizeInBytes": 1757231,
"ilmPhase": "hot",
"markdownComments": [
"""### .ds-logs-endpoint.events.process-default-2023.11.26-000002
""",
"""| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.11.26-000002 | 1,020 (22.4%) | 1 | `hot` | 1.7MB |
""",
"""### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
"""#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
"""
#### Incompatible field values - .ds-logs-endpoint.events.process-default-2023.11.26-000002
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (905) |
"""
],
"ecsVersion": "8.6.1",
"indexId": "bFAwi1slQwC0mjQGeSBFIw",
"error": null
},
"sort": [
1706546500145
]
}
]
}
}
},
{
"key": ".ds-logs-endpoint.events.process-default-2024.01.08-000003",
"doc_count": 1,
"latest_doc": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
"_id": "1FgZVo0BLR4mG9DsBfpR",
"_score": null,
"_source": {
"@timestamp": 1706546496847,
"batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
"indexName": ".ds-logs-endpoint.events.process-default-2024.01.08-000003",
"isCheckAll": true,
"checkedAt": 1706546496678,
"docsCount": 333,
"totalFieldCount": 446,
"ecsFieldCount": 304,
"customFieldCount": 141,
"incompatibleFieldCount": 1,
"sameFamilyFieldCount": 0,
"sameFamilyFields": [],
"unallowedMappingFields": [],
"unallowedValueFields": [
"event.outcome"
],
"sizeInBytes": 564632,
"ilmPhase": "hot",
"markdownComments": [
"""### .ds-logs-endpoint.events.process-default-2024.01.08-000003
""",
"""| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2024.01.08-000003 | 333 (7.3%) | 1 | `hot` | 551.4KB |
""",
"""### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
"""#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
"""
#### Incompatible field values - .ds-logs-endpoint.events.process-default-2024.01.08-000003
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (297) |
"""
],
"ecsVersion": "8.6.1",
"indexId": "FILVzAjfS6qQNX5TCH2yOw",
"error": null
},
"sort": [
1706546496847
]
}
]
}
}
}
]
}
}
} I'm wondering, should the |
.../security-solution/ecs_data_quality_dashboard/impl/data_quality/use_results_rollup/index.tsx
Outdated
Show resolved
Hide resolved
Yup, that was not expected. Has been fixed in e0be9c9 The privilege checks have been aligned in this commit as well. Thanks for the thorough testing 🙌 |
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/post_results.ts
Show resolved
Hide resolved
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/get_results.ts
Show resolved
Hide resolved
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/post_results.test.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/post_results.test.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/ecs_data_quality_dashboard/server/routes/results/get_results.ts
Outdated
Show resolved
Hide resolved
@elasticmachine merge upstream |
After discussing offline some unexpected behaviors regarding data stream authorization with @andrew-goldstein, the following behavior has been applied: GET with
POST with
The DQ privilege check in step 2 has been extracted to privileges.ts module. |
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Async chunks
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @semd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @semd for your persistence on persistence! This work unlocks so many new use cases and features! 🙏
✅ Desk tested locally
LGTM 🚀
* main: (224 commits) [Http] Replace `buildNr` with `buildSha` in static asset paths (#175898) [Ops] Fix GCS bucket access for future buildkite agents (#174756) [api-docs] 2024-02-07 Daily api_docs build (#176362) skip flaky suite (#176002) skip failing es promotion suite (#176359) [Cloud Security] [Grouping] Add URL Params support to the grouping components (#175749) chore(NA): update versions after v8.12.2 bump (#176309) chore(NA): update versions after v7.17.19 bump (#176313) skip failing test suite (#176352) [SLO] Enable burn rate alert by default during creation via UI (#176317) [Fleet] Add the uptime capability to observability projects (#176285) [Security Solution][Endpoint] Fix Manifest Manger so that it works with large (>10k) (#174411) [ResponseOps] Alert creation delay based on user definition (#175851) [data views] Default field formatters based on field meta values (#174973) [Cloud Security]Detection Rules counter on Rules Flyout (#176041) [Security Solution] Data Quality Dashboard persistence (#175673) [Ent Search] Connector client copy cleanup (#176290) [ML] Anomaly Detection: Adds actions menu to anomaly markers in Single Metric Viewer chart. (#175556) [ML] Anomaly Detection: Fix `values-dots` colors (#176303) [Fleet] Logstash Output - being compliant to RFC-952 (#176298) ...
## Summary follow-up of elastic#173185 This PR enables the persistence layer implemented in the previous PR, applying the following changes: - Update the mapping to store unitary index results instead of storing the whole pattern with the results in each document. - Change the query to get the stored results by aggregating documents by indexName. The authorized indexNames derived from the `pattern` parameter are retrieved using the `indices.get` request. - A bug involving a race condition with the initialization and the retrieval of stored results, resulting in an unintended reset of the results in the UI, has been fixed. https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4 --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary follow-up of elastic#173185 This PR enables the persistence layer implemented in the previous PR, applying the following changes: - Update the mapping to store unitary index results instead of storing the whole pattern with the results in each document. - Change the query to get the stored results by aggregating documents by indexName. The authorized indexNames derived from the `pattern` parameter are retrieved using the `indices.get` request. - A bug involving a race condition with the initialization and the retrieval of stored results, resulting in an unintended reset of the results in the UI, has been fixed. https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4 --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary follow-up of elastic#173185 This PR enables the persistence layer implemented in the previous PR, applying the following changes: - Update the mapping to store unitary index results instead of storing the whole pattern with the results in each document. - Change the query to get the stored results by aggregating documents by indexName. The authorized indexNames derived from the `pattern` parameter are retrieved using the `indices.get` request. - A bug involving a race condition with the initialization and the retrieval of stored results, resulting in an unintended reset of the results in the UI, has been fixed. https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4 --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary follow-up of elastic#173185 This PR enables the persistence layer implemented in the previous PR, applying the following changes: - Update the mapping to store unitary index results instead of storing the whole pattern with the results in each document. - Change the query to get the stored results by aggregating documents by indexName. The authorized indexNames derived from the `pattern` parameter are retrieved using the `indices.get` request. - A bug involving a race condition with the initialization and the retrieval of stored results, resulting in an unintended reset of the results in the UI, has been fixed. https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4 --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Summary
follow-up of #173185
This PR enables the persistence layer implemented in the previous PR, applying the following changes:
pattern
parameter are retrieved using theindices.get
request.data_quality_dashboard_persistence_demo.mov