Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Data Quality Dashboard persistence #175673

Merged
merged 19 commits into from
Feb 6, 2024

Conversation

semd
Copy link
Contributor

@semd semd commented Jan 26, 2024

Summary

follow-up of #173185

This PR enables the persistence layer implemented in the previous PR, applying the following changes:

  • Update the mapping to store unitary index results instead of storing the whole pattern with the results in each document.
  • Change the query to get the stored results by aggregating documents by indexName. The authorized indexNames derived from the pattern parameter are retrieved using the indices.get request.
  • A bug involving a race condition with the initialization and the retrieval of stored results, resulting in an unintended reset of the results in the UI, has been fixed.
data_quality_dashboard_persistence_demo.mov

@semd semd self-assigned this Jan 26, 2024
@semd semd added Team:Threat Hunting Security Solution Threat Hunting Team release_note:feature Makes this part of the condensed release notes Team:Threat Hunting:Explore v8.13.0 labels Jan 26, 2024
@semd semd marked this pull request as ready for review January 26, 2024 12:23
@semd semd requested a review from a team as a code owner January 26, 2024 12:23
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

Copy link
Contributor

@stephmilovic stephmilovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manual tested for that race bug and it is gone, woohoo! LGTM, great work @semd

@semd
Copy link
Contributor Author

semd commented Jan 26, 2024

@elasticmachine merge upstream

@semd
Copy link
Contributor Author

semd commented Jan 29, 2024

@elasticmachine merge upstream

@semd
Copy link
Contributor Author

semd commented Jan 29, 2024

As agreed offline with @andrew-goldstein, "Last check" column has been added:

screenshot

@semd semd requested a review from a team as a code owner January 29, 2024 14:12
@semd
Copy link
Contributor Author

semd commented Jan 29, 2024

@elasticmachine merge upstream

@andrew-goldstein
Copy link
Contributor

@elasticmachine merge upstream

@andrew-goldstein
Copy link
Contributor

Consider if the following screenshot, captured after refreshing the Data Quality dashboard, demonstrates unexpected behavior:

results_after_check_all

  • The screenshot above appears to illustrate that most results from the previous Check all, (Last checked: an hour ago) were restored from storage, and display a ✅ or ❌ in the Result column of the tables
  • The .ds-logs-endpoint.alerts-default-2023.10.27-000001 index in the logs-* pattern displays a few seconds ago in the Last check column, but a -- in the result

When the following console.log:

          console.log(
            `--> getResultsRoute query for pattern ${pattern}`,
            JSON.stringify(query, null, 2)
          );

is added to getResultsRoute(), the generated query:

--> getResultsRoute query for pattern logs-* {
  "index": ".kibana-data-quality-dashboard-results-default",
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "indexName": [
              ".ds-logs-endpoint.alerts-default-2023.10.27-000001",
              ".ds-logs-endpoint.alerts-default-2023.11.26-000002",
              ".ds-logs-endpoint.alerts-default-2024.01.08-000003",
              ".ds-logs-endpoint.events.process-default-2023.10.27-000001",
              ".ds-logs-endpoint.events.process-default-2023.11.26-000002",
              ".ds-logs-endpoint.events.process-default-2024.01.08-000003"
            ]
          }
        }
      ]
    }
  },
  "aggs": {
    "latest": {
      "terms": {
        "field": "indexName",
        "size": 10000
      },
      "aggs": {
        "latest_doc": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

the query appears to include the .ds-logs-endpoint.alerts-default-2023.10.27-000001 index, but the response from Elasticsearch does not appear to include results for the .ds-logs-endpoint.alerts-default-2023.10.27-000001:

Click me

response (for the query above, via dev tools)

{
"took": 1,
"timed_out": false,
"_shards": {
  "total": 1,
  "successful": 1,
  "skipped": 0,
  "failed": 0
},
"hits": {
  "total": {
    "value": 5,
    "relation": "eq"
  },
  "max_score": null,
  "hits": []
},
"aggregations": {
  "latest": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": ".ds-logs-endpoint.alerts-default-2023.11.26-000002",
        "doc_count": 1,
        "latest_doc": {
          "hits": {
            "total": {
              "value": 1,
              "relation": "eq"
            },
            "max_score": null,
            "hits": [
              {
                "_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
                "_id": "KFhPVo0BLR4mG9DskP35",
                "_score": null,
                "_source": {
                  "@timestamp": 1706550071544,
                  "batchId": "16ab03d8-3a83-490a-a938-f5656b9a50e6",
                  "indexName": ".ds-logs-endpoint.alerts-default-2023.11.26-000002",
                  "isCheckAll": false,
                  "checkedAt": 1706550071360,
                  "docsCount": 1406,
                  "totalFieldCount": 1582,
                  "ecsFieldCount": 677,
                  "customFieldCount": 904,
                  "incompatibleFieldCount": 1,
                  "sameFamilyFieldCount": 0,
                  "sameFamilyFields": [],
                  "unallowedMappingFields": [],
                  "unallowedValueFields": [
                    "event.category"
                  ],
                  "sizeInBytes": 2560371,
                  "ilmPhase": "hot",
                  "markdownComments": [
                    """### .ds-logs-endpoint.alerts-default-2023.11.26-000002
""",
                    """| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.11.26-000002 | 1,406 (30.8%) | 1 | `hot` | 7.9MB |

""",
                    """### **Incompatible fields** `1` **Same family** `0` **Custom fields** `904` **ECS compliant fields** `677` **All fields** `1582`
""",
                    """#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
                    """

#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.11.26-000002


| Field | ECS values (expected) | Document values (actual) | 
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (141) |

"""
                  ],
                  "ecsVersion": "8.6.1",
                  "indexId": "-5OpMG4tSGOidPOGDRYoAg",
                  "error": null
                },
                "sort": [
                  1706550071544
                ]
              }
            ]
          }
        }
      },
      {
        "key": ".ds-logs-endpoint.alerts-default-2024.01.08-000003",
        "doc_count": 1,
        "latest_doc": {
          "hits": {
            "total": {
              "value": 1,
              "relation": "eq"
            },
            "max_score": null,
            "hits": [
              {
                "_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
                "_id": "11gZVo0BLR4mG9DsLPoE",
                "_score": null,
                "_source": {
                  "@timestamp": 1706546506754,
                  "batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
                  "indexName": ".ds-logs-endpoint.alerts-default-2024.01.08-000003",
                  "isCheckAll": true,
                  "checkedAt": 1706546506575,
                  "docsCount": 466,
                  "totalFieldCount": 1582,
                  "ecsFieldCount": 677,
                  "customFieldCount": 904,
                  "incompatibleFieldCount": 1,
                  "sameFamilyFieldCount": 0,
                  "sameFamilyFields": [],
                  "unallowedMappingFields": [],
                  "unallowedValueFields": [
                    "event.category"
                  ],
                  "sizeInBytes": 795964,
                  "ilmPhase": "hot",
                  "markdownComments": [
                    """### .ds-logs-endpoint.alerts-default-2024.01.08-000003
""",
                    """| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.alerts-default-2024.01.08-000003 | 466 (10.2%) | 1 | `hot` | 777.3KB |

""",
                    """### **Incompatible fields** `1` **Same family** `0` **Custom fields** `904` **ECS compliant fields** `677` **All fields** `1582`
""",
                    """#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
                    """

#### Incompatible field values - .ds-logs-endpoint.alerts-default-2024.01.08-000003


| Field | ECS values (expected) | Document values (actual) | 
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (48) |

"""
                  ],
                  "ecsVersion": "8.6.1",
                  "indexId": "frzzVLeRSSSbNHxbNM6FqQ",
                  "error": null
                },
                "sort": [
                  1706546506754
                ]
              }
            ]
          }
        }
      },
      {
        "key": ".ds-logs-endpoint.events.process-default-2023.10.27-000001",
        "doc_count": 1,
        "latest_doc": {
          "hits": {
            "total": {
              "value": 1,
              "relation": "eq"
            },
            "max_score": null,
            "hits": [
              {
                "_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
                "_id": "1lgZVo0BLR4mG9DsHvr2",
                "_score": null,
                "_source": {
                  "@timestamp": 1706546503412,
                  "batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
                  "indexName": ".ds-logs-endpoint.events.process-default-2023.10.27-000001",
                  "isCheckAll": true,
                  "checkedAt": 1706546503245,
                  "docsCount": 553,
                  "totalFieldCount": 446,
                  "ecsFieldCount": 304,
                  "customFieldCount": 141,
                  "incompatibleFieldCount": 1,
                  "sameFamilyFieldCount": 0,
                  "sameFamilyFields": [],
                  "unallowedMappingFields": [],
                  "unallowedValueFields": [
                    "event.outcome"
                  ],
                  "sizeInBytes": 1063145,
                  "ilmPhase": "hot",
                  "markdownComments": [
                    """### .ds-logs-endpoint.events.process-default-2023.10.27-000001
""",
                    """| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.10.27-000001 | 553 (12.1%) | 1 | `hot` | 1MB |

""",
                    """### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
                    """#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
                    """

#### Incompatible field values - .ds-logs-endpoint.events.process-default-2023.10.27-000001


| Field | ECS values (expected) | Document values (actual) | 
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (491) |

"""
                  ],
                  "ecsVersion": "8.6.1",
                  "indexId": "4DeVaFmaTWyXlHQ7IrVvCA",
                  "error": null
                },
                "sort": [
                  1706546503412
                ]
              }
            ]
          }
        }
      },
      {
        "key": ".ds-logs-endpoint.events.process-default-2023.11.26-000002",
        "doc_count": 1,
        "latest_doc": {
          "hits": {
            "total": {
              "value": 1,
              "relation": "eq"
            },
            "max_score": null,
            "hits": [
              {
                "_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
                "_id": "1VgZVo0BLR4mG9DsEvoz",
                "_score": null,
                "_source": {
                  "@timestamp": 1706546500145,
                  "batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
                  "indexName": ".ds-logs-endpoint.events.process-default-2023.11.26-000002",
                  "isCheckAll": true,
                  "checkedAt": 1706546499983,
                  "docsCount": 1020,
                  "totalFieldCount": 446,
                  "ecsFieldCount": 304,
                  "customFieldCount": 141,
                  "incompatibleFieldCount": 1,
                  "sameFamilyFieldCount": 0,
                  "sameFamilyFields": [],
                  "unallowedMappingFields": [],
                  "unallowedValueFields": [
                    "event.outcome"
                  ],
                  "sizeInBytes": 1757231,
                  "ilmPhase": "hot",
                  "markdownComments": [
                    """### .ds-logs-endpoint.events.process-default-2023.11.26-000002
""",
                    """| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.11.26-000002 | 1,020 (22.4%) | 1 | `hot` | 1.7MB |

""",
                    """### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
                    """#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
                    """

#### Incompatible field values - .ds-logs-endpoint.events.process-default-2023.11.26-000002


| Field | ECS values (expected) | Document values (actual) | 
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (905) |

"""
                  ],
                  "ecsVersion": "8.6.1",
                  "indexId": "bFAwi1slQwC0mjQGeSBFIw",
                  "error": null
                },
                "sort": [
                  1706546500145
                ]
              }
            ]
          }
        }
      },
      {
        "key": ".ds-logs-endpoint.events.process-default-2024.01.08-000003",
        "doc_count": 1,
        "latest_doc": {
          "hits": {
            "total": {
              "value": 1,
              "relation": "eq"
            },
            "max_score": null,
            "hits": [
              {
                "_index": ".ds-.kibana-data-quality-dashboard-results-default-2024.01.23-000001",
                "_id": "1FgZVo0BLR4mG9DsBfpR",
                "_score": null,
                "_source": {
                  "@timestamp": 1706546496847,
                  "batchId": "4c6948b4-a493-44a4-9689-6d4567ec1e90",
                  "indexName": ".ds-logs-endpoint.events.process-default-2024.01.08-000003",
                  "isCheckAll": true,
                  "checkedAt": 1706546496678,
                  "docsCount": 333,
                  "totalFieldCount": 446,
                  "ecsFieldCount": 304,
                  "customFieldCount": 141,
                  "incompatibleFieldCount": 1,
                  "sameFamilyFieldCount": 0,
                  "sameFamilyFields": [],
                  "unallowedMappingFields": [],
                  "unallowedValueFields": [
                    "event.outcome"
                  ],
                  "sizeInBytes": 564632,
                  "ilmPhase": "hot",
                  "markdownComments": [
                    """### .ds-logs-endpoint.events.process-default-2024.01.08-000003
""",
                    """| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | .ds-logs-endpoint.events.process-default-2024.01.08-000003 | 333 (7.3%) | 1 | `hot` | 551.4KB |

""",
                    """### **Incompatible fields** `1` **Same family** `0` **Custom fields** `141` **ECS compliant fields** `304` **All fields** `446`
""",
                    """#### 1 incompatible field

Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.

❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
""",
                    """

#### Incompatible field values - .ds-logs-endpoint.events.process-default-2024.01.08-000003


| Field | ECS values (expected) | Document values (actual) | 
|-------|-----------------------|--------------------------|
| event.outcome | `failure`, `success`, `unknown` | `` (297) |

"""
                  ],
                  "ecsVersion": "8.6.1",
                  "indexId": "FILVzAjfS6qQNX5TCH2yOw",
                  "error": null
                },
                "sort": [
                  1706546496847
                ]
              }
            ]
          }
        }
      }
    ]
  }
}
}

I'm wondering, should the Last check for .ds-logs-endpoint.alerts-default-2023.10.27-000001 be a few seconds ago?

@semd
Copy link
Contributor Author

semd commented Jan 30, 2024

@andrew-goldstein

Consider if the following screenshot, captured after refreshing the Data Quality dashboard, demonstrates unexpected behavior

Yup, that was not expected. Has been fixed in e0be9c9

The privilege checks have been aligned in this commit as well. Thanks for the thorough testing 🙌

@semd
Copy link
Contributor Author

semd commented Feb 1, 2024

@elasticmachine merge upstream

@semd
Copy link
Contributor Author

semd commented Feb 1, 2024

After discussing offline some unexpected behaviors regarding data stream authorization with @andrew-goldstein, the following behavior has been applied:

GET with pattern query parameter:

  1. Request the index API (docs) for all indices and data streams associated to the pattern parameter. The internal user is used for this query since privileges for the index API do not match the privileges needed for DQ checks. The features=aliases parameter is added to prevent mappings and settings information from being included.
  2. Check DQ privileges (read and view_index_metadata and (monitor or manage)) for all the indices and the data streams extracted from the previous request. All indices that are data stream backing indices (.ds-*) are excluded from the privilege check since only the authorization for the parent data stream is needed to perform the DQ check on them.
  3. Retrieve the latest results from the storage filtering by authorized indices, including authorized data stream backing indices.

POST with body.indexName payload:

  1. Request the index API (docs) to know if a data stream is associated with the indexName parameter. The internal user is used since privileges for this API do not match the privileges we need for DQ results.
  2. Check DQ privilege (read and view_index_metadata and (monitor or manage)) for the indexName, If the indexName is a data stream backing index, the privilege for the data stream is checked instead.
  3. If the authorization check is succeeded the body result document is indexed.

The DQ privilege check in step 2 has been extracted to privileges.ts module.

@semd
Copy link
Contributor Author

semd commented Feb 5, 2024

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 11.4MB 11.4MB +1.8KB
Unknown metric groups

ESLint disabled line counts

id before after diff
ecsDataQualityDashboard 7 5 -2

Total ESLint disabled count

id before after diff
ecsDataQualityDashboard 7 5 -2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @semd

Copy link
Contributor

@andrew-goldstein andrew-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @semd for your persistence on persistence! This work unlocks so many new use cases and features! 🙏
✅ Desk tested locally
LGTM 🚀

@semd semd merged commit 1c7d089 into elastic:main Feb 6, 2024
35 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Feb 6, 2024
jloleysens added a commit that referenced this pull request Feb 7, 2024
* main: (224 commits)
  [Http] Replace `buildNr` with `buildSha` in static asset paths (#175898)
  [Ops] Fix GCS bucket access for future buildkite agents (#174756)
  [api-docs] 2024-02-07 Daily api_docs build (#176362)
  skip flaky suite (#176002)
  skip failing es promotion suite (#176359)
  [Cloud Security] [Grouping] Add URL Params support to the grouping components (#175749)
  chore(NA): update versions after v8.12.2 bump (#176309)
  chore(NA): update versions after v7.17.19 bump (#176313)
  skip failing test suite (#176352)
  [SLO] Enable burn rate alert by default during creation via UI (#176317)
  [Fleet] Add the uptime capability to observability projects (#176285)
  [Security Solution][Endpoint] Fix Manifest Manger so that it works with large (>10k) (#174411)
  [ResponseOps] Alert creation delay based on user definition (#175851)
  [data views] Default field formatters based on field meta values (#174973)
  [Cloud Security]Detection Rules counter on Rules Flyout (#176041)
  [Security Solution] Data Quality Dashboard persistence (#175673)
  [Ent Search] Connector client copy cleanup (#176290)
  [ML] Anomaly Detection: Adds actions menu to anomaly markers in Single Metric Viewer chart. (#175556)
  [ML] Anomaly Detection: Fix `values-dots` colors (#176303)
  [Fleet] Logstash Output - being compliant to RFC-952 (#176298)
  ...
fkanout pushed a commit to fkanout/kibana that referenced this pull request Feb 7, 2024
## Summary

follow-up of elastic#173185

This PR enables the persistence layer implemented in the previous PR,
applying the following changes:

- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.


https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
## Summary

follow-up of elastic#173185

This PR enables the persistence layer implemented in the previous PR,
applying the following changes:

- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.


https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
## Summary

follow-up of elastic#173185

This PR enables the persistence layer implemented in the previous PR,
applying the following changes:

- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.


https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
fkanout pushed a commit to fkanout/kibana that referenced this pull request Mar 4, 2024
## Summary

follow-up of elastic#173185

This PR enables the persistence layer implemented in the previous PR,
applying the following changes:

- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.


https://github.com/elastic/kibana/assets/17747913/0598606b-c5f4-42b3-901c-f86a3cac65e4

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting release_note:feature Makes this part of the condensed release notes Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team v8.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants