[Security Solution] Data quality dashboard persistence

[semd]

Summary

issue https://github.com/elastic/security-team/issues/7382

Data Stream Adapter

This PR introduces the @kbn/data-stream-adapter package, which is a utility library to facilitate Data Stream creation and maintenance in Kibana, it was inspired by the data stream implementation in the Alerts plugin.
The library has two exports:

• DataStreamSpacesAdapter: to manage space data streams. It uses the name-of-the-data-stream-<spaceId> naming pattern.

• DataStreamAdapter: to manage single (not space-aware) data streams.

Usage examples in the package README

Data Quality Dashboard

The DataStreamSpacesAdapter has been integrated into the data quality dashboard to store all the quality checks users perform. The information stored is the metadata (also used for telemetry) and the actual data rendered in the tables.

FieldMap definition here

Demo

DQDashboard_persistance_demo.mov

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

[semd]

@elasticmachine merge upstream

[TinaHeiligers]

Spaces should be listed under optionalPlugins and you'll have to implement the if spaces do ... pattern.

[TinaHeiligers]

"spaces" can be disabled and should be listed as an optional plugin. See https://github.com/elastic/kibana/blob/main/src/plugins/saved_search/kibana.jsonc

[semd]

however after the next step:

5. Navigating to the Data Quality dashboard

The Data Quality page displays a Error reading saved data quality check results error toaster on page load, and the following errors are logged to the Kibana server log (whenever) the Data Quality dashboard page is loaded:

This is the expected behavior. This data stream is "space-aware", it can not be created on server start, it is created during runtime when we know the space the user is using. If there's an error in the creation it will pop up to the UI.

[semd]

Consider:
Weather or not the POST should return a 200 in this case
Consider displaying a banner in a similar style to the other banners in the Data Quality page, such that it:
Warns the user data quality results won't be persisted
Includes the permissions required to remediate the issue

We discussed that with @dhru42, we don't want to restrict the /results POST or GET from the UI, all users that can use the data quality dashboard should be able to persist the check results and read the persisted checks. We are already preventing data leaks by checking the user's index privileges against the results being retrieved.

This criteria only affects this specific data quality dashboard API route, which is only used to retrieve the latest check results of authorized index patterns. Arbitrary queries to .kibana-data-quality-dashboard-results data stream will require index privileges, as usual.

[andrew-goldstein]

✅ It was possible to recover from all simulated error states (after removing the test code) by restarting Kibana

It appears it may be possible for users to enter a state where it's not possible to recover from a simulated error state via

DELETE /_data_stream/.kibana-data-quality-dashboard-results-default
DELETE /_index_template/.kibana-data-quality-dashboard-results-index-template
DELETE /_component_template/.kibana-data-quality-dashboard-results-mappings
DELETE /_component_template/.kibana-data-quality-dashboard-ecs-mappings

If the following simulated error is thrown in packages/kbn-data-stream-adapter/src/create_or_update_data_stream.ts:

/**
 * Updates the data stream mapping and total field limit setting
 */
const updateDataStreamMappings = async ({
  logger,
  esClient,
  totalFieldsLimit,
  indexNames,
}: UpdateIndexMappingsOpts) => {
  throw new Error('simulated error in updateDataStreamMappings');

  // Update total field limit setting of found indices
  // Other index setting changes are not updated at this time
  // await Promise.all(
  //   indexNames.map((indexName) =>
  //     updateTotalFieldLimitSetting({ logger, esClient, totalFieldsLimit, indexName })
  //   )
  // );
  // // Update mappings of the found indices.
  // await Promise.all(
  //   indexNames.map((indexName) => updateMapping({ logger, esClient, totalFieldsLimit, indexName }))
  // );
};

The error above was simulated after the data stream was created (at startup, on a previous run).

After:

1. Simulating the error above
2. Restoring the deleted code
3. Deleting the previously created data stream via

DELETE /_data_stream/.kibana-data-quality-dashboard-results-default
DELETE /_index_template/.kibana-data-quality-dashboard-results-index-template
DELETE /_component_template/.kibana-data-quality-dashboard-results-mappings
DELETE /_component_template/.kibana-data-quality-dashboard-ecs-mappings

4. Restarting Kibana

Errors are NOT logged at startup:

[2024-01-22T22:16:31.221-05:00][INFO ][root] Kibana is starting
[2024-01-22T22:16:43.490-05:00][INFO ][plugins.ecsDataQualityDashboard] Installing component template .kibana-data-quality-dashboard-ecs-mappings
[2024-01-22T22:16:43.491-05:00][INFO ][plugins.ecsDataQualityDashboard] Installing component template .kibana-data-quality-dashboard-results-mappings
[2024-01-22T22:16:44.642-05:00][INFO ][plugins.ecsDataQualityDashboard] Installing index template .kibana-data-quality-dashboard-results-index-template
[2024-01-22T22:16:44.787-05:00][INFO ][plugins.ecsDataQualityDashboard] Updating data streams - .kibana-data-quality-dashboard-results-*
[2024-01-22T22:16:49.263-05:00][INFO ][status] Kibana is now available

however after the next step:

5. Navigating to the Data Quality dashboard

The Data Quality page displays a Error reading saved data quality check results error toaster on page load, and the following errors are logged to the Kibana server log (whenever) the Data Quality dashboard page is loaded:

[2024-01-22T22:02:13.473-05:00][INFO ][plugins.ecsDataQualityDashboard] Creating data stream - .kibana-data-quality-dashboard-results-default
[2024-01-22T22:02:13.509-05:00][ERROR][plugins.ecsDataQualityDashboard] Error creating data stream .kibana-data-quality-dashboard-results-default - illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]
[2024-01-22T22:02:13.509-05:00][ERROR][plugins.ecsDataQualityDashboard] [GET results] Error retrieving results index name: Failure during installation of create .kibana-data-quality-dashboard-results-default data stream. illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]
[2024-01-22T22:02:13.582-05:00][ERROR][plugins.ecsDataQualityDashboard] [GET results] Error retrieving results index name: Failure during installation of create .kibana-data-quality-dashboard-results-default data stream. illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]

@semd and I paired to debug this, and observed that an index (instead of a data stream) with empty mappings was created around the time the error above started appearing. These observations were made via:

GET .kibana-data-quality-dashboard-results-default/_mapping
GET _cat/indices/.kibana-data-quality-dashboard-results-default?h=h,s,i,id,p,r,dc,dd,ss,creation.date.string

It appears that Elasticsearch created the index as a side effect, but we weren't able to reproduce this behavior.

Resolution

If the Data Quality page displays an Error reading saved data quality check results error toaster on page load, and the following errors are logged to the Kibana server log (whenever) the Data Quality dashboard page is visited:

[2024-01-22T22:02:13.473-05:00][INFO ][plugins.ecsDataQualityDashboard] Creating data stream - .kibana-data-quality-dashboard-results-default
[2024-01-22T22:02:13.509-05:00][ERROR][plugins.ecsDataQualityDashboard] Error creating data stream .kibana-data-quality-dashboard-results-default - illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]
[2024-01-22T22:02:13.509-05:00][ERROR][plugins.ecsDataQualityDashboard] [GET results] Error retrieving results index name: Failure during installation of create .kibana-data-quality-dashboard-results-default data stream. illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]
[2024-01-22T22:02:13.582-05:00][ERROR][plugins.ecsDataQualityDashboard] [GET results] Error retrieving results index name: Failure during installation of create .kibana-data-quality-dashboard-results-default data stream. illegal_state_exception
        Root causes:
                illegal_state_exception: index, alias, and data stream names need to be unique, but the following duplicates were found [data stream [.kibana-data-quality-dashboard-results-default] conflicts with index]

1. Check for an existing index with empty mappings via the Kibana Dev Tools console:

GET .kibana-data-quality-dashboard-results-default/_mapping

2. If a .kibana-data-quality-dashboard-results-default index (instead of a data stream) exists, delete it:

DELETE .kibana-data-quality-dashboard-results-default

3. Delete all the artifacts related to the data stream:

# RESULTS_DATA_STREAM_NAME
DELETE /_data_stream/.kibana-data-quality-dashboard-results-default

# RESULTS_INDEX_TEMPLATE_NAME
DELETE /_index_template/.kibana-data-quality-dashboard-results-index-template

# RESULTS_COMPONENT_TEMPLATE_NAME
DELETE /_component_template/.kibana-data-quality-dashboard-results-mappings

# ECS_COMPONENT_TEMPLATE_NAME
DELETE /_component_template/.kibana-data-quality-dashboard-ecs-mappings

4. Restart Kibana (to re-create the artifacts)

5. Refresh the Data Quality Dashboard page (to create the space-specific instance of the data stream)

[andrew-goldstein]

indices that were previously checked (showing a status of ❌) reverted back to their unchecked -- state

@andrew-goldstein I didn't know it was possible to stop the "check all". I bet the aborted (empty) result is incorrectly persisted. let me check that logic. BTW, the video does not show what happens if we refresh the page on the right side (the one that stated/stopped the check), were both screens consistently showing a -- state?

Yes, both users consistently observed a -- state

[andrew-goldstein]

Consider revising the implementation of this privileges check, because per the following screenshot:

the following privileges are required to check an index:

• monitor or manage
• view_index_metadata
• read

Consider:

• The all privilege is not required
• either monitor or manage is required to check an index
  • In a future release, read may be expanded to be read or read_cross_cluster, if the Elasticsearch stats and _ilm/explain APIs are enhanced to support CCS.

[andrew-goldstein]

Consider an alternative authorization check, because per the following scenario, this check appears to return unexpected results when two users visit the Data Quality page, but have different index privileges for the same pattern.

Example: given a pattern from the Security Solution default, e.g. packetbeat-*, one user may have privileges to view all packetbeat-* data streams and indicies (i.e. .ds-packetbeat-8.9.2-2023.12.06-000001 and packetbeat-7.17.9-2024.01.08-000002), while another user may have privileges for ONLY data streams created by the 8.x version of Packetbeat (i.e. .ds-packetbeat-8.9.2-2023.12.06-000001), but not the 7.x version of Packetbeat.

To demonstrate the unexpected results, first add the following console.log statements to the authorization step, such that the Kibana server logs include the privileges and derived authorizedPatterns:

          // Confirm user has authorization for the requested patterns
          const { patterns } = request.query;
          const userEsClient = services.core.elasticsearch.client.asCurrentUser;
          const privileges = await userEsClient.security.hasPrivileges({
            index: [
              { names: patterns.split(','), privileges: ['all', 'read', 'view_index_metadata'] },
            ],
          });

          console.log('--> privileges', JSON.stringify(privileges, null, 2));

          const authorizedPatterns = Object.keys(privileges.index).filter((pattern) =>
            Object.values(privileges.index[pattern]).some((v) => v === true)
          );

          console.log('--> authorizedPatterns', JSON.stringify(authorizedPatterns, null, 2));

          if (authorizedPatterns.length === 0) {
            return response.ok({ body: [] });
          }

Next, create a new role called ONLY_SOME_PACKETBEAT_ROLE, such that

• The user has access to all auditbeat-* indices
• The user has access to only some packetbeat-* indices

per the following screenshot of the ONLY_SOME_PACKETBEAT_ROLE:

In the screenshot above, the following pattern ONLY matches data streams created by the 8.x version of Packetbeat:

packetbeat-8.*

As a result, users with the ONLY_SOME_PACKETBEAT_ROLE may view (and check), for example, a data stream created by the 8.x version of Packetbeat, with a backing index named:

.ds-packetbeat-8.9.2-2023.12.06-000001

but that same user may NOT check an index created by the 7.x version of Packetbeat, e.g. an index named:

packetbeat-7.17.9-2024.01.08-000002

Next create a user named somepacketbeat, and assign that user the ONLY_SOME_PACKETBEAT_ROLE, per the following screenshot:

Finally, create one more user named alsosuperuser, and assign that user the superuser role, per the screenshot below:

Navigate to Stack Management > Users. Per the screenshot below:

✅ The alsosuperuser user has the superuser role (just like the elastic user)
✅ The somepacketbeat user has the ONLY_SOME_PACKETBEAT_ROLE

Establish a baseline with the elastic user

1. Login to Kibana as the elastic user, who has the superuser role

2. Navigate to Security > Dashboards > Data Qualty

3. Click the Check all button

Note that:

✅ The .alerts-security.alerts-default pattern contains results, per the screenshot below

✅ The auditbeat-* pattern contains results, per previous screenshot above

✅ The packetbeat-* pattern contains 6 pages of results, which include indices from both the 8.x and 7.x versions of Packetbeat, per the screeenshot below:

Also observe the Kibana server logs console.log output for the elastic user:

--> privileges {
  "username": "elastic",
  "has_all_requested": true,
  "cluster": {},
  "index": {
    ".alerts-security.alerts-default": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "auditbeat-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "logs-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "packetbeat-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    }
  },
  "application": {}
}
--> authorizedPatterns [
  ".alerts-security.alerts-default",
  "auditbeat-*",
  "logs-*",
  "packetbeat-*"
]

Compare the alsosuperuser user with the elastic user

1. Login to Kibana as the alsosuperuser user, who also has the superuser role, just like the elastic user

2. Navigate to Security > Dashboards > Data Quality

Note that:

✅ As expected, Last checked: -- is displayed for the alsosuperuser user, because this user has never performed a data quality check

✅ As expected, the .alerts-security.alerts-default pattern contains results, because the alsosuperuser is viewing results created by the elastic user, per the screenshot below:

✅ As expected, the auditbeat-* pattern contains results (from the elastic user), per previous screenshot above

✅ As expected, the packetbeat-* pattern contains 6 pages of results (created by the elastic user), which include indices from both the 8.x and 7.x versions of Packetbeat, per the screeenshot below:

Also observe the Kibana server logs console.log output for the alsosuperuser user:

--> privileges {
  "username": "alsosuperuser",
  "has_all_requested": true,
  "cluster": {},
  "index": {
    ".alerts-security.alerts-default": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "auditbeat-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "logs-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    },
    "packetbeat-*": {
      "all": true,
      "read": true,
      "view_index_metadata": true
    }
  },
  "application": {}
}
--> authorizedPatterns [
  ".alerts-security.alerts-default",
  "auditbeat-*",
  "logs-*",
  "packetbeat-*"
]

So far, the output demonstrates that:

✅ As expected, the alsosuperuser user may view all the persisted results from the elastic user

✅ As expected, the privileges and derived authorizedPatterns are the same for both the alsosuperuser and elastic user

Compare the somepacketbeat user with the elastic user (has unexpected results)

1. Login to Kibana as the somepacketbeat user, who has DIFFERENT permissions compared with the elastic and alsosuperuser users

2. Navigate to Security > Dashboards > Data Quality

Note that:

✅ As expected, Last checked: -- is displayed for the somepacketbeat user, because this user has never performed a data quality check, per the screeenshot below

✅ As expected, the .alerts-security.alerts-default pattern is not displayed, because the somepacketbeat user (unlike the elastic user) does not have access to the .alerts-security.alerts-default pattern

✅ As expected, the auditbeat-* pattern contains results (from the elastic user)

✅ As expected, the packetbeat-* pattern contains 3 pages of indices, (as opposed to 6 for the elastic and alsosuperuser users), because the ONLY_SOME_PACKETBEAT_ROLE role only allows access to data streams created by the 8.x version of Packetbeat

❌ NOT EXPECTED: The packetbeat-* results (created by the elastic user) are NOT shown for the somepacketbeat user, even though the somepacketbeat has access to the 8.x Packetbeat data streams

This unexpected behavior appears to be related to the following unexpected authorizedPatterns derived state in the Kibana server log for the somepacketbeat user:

--> privileges {
  "username": "somepacketbeat",
  "has_all_requested": false,
  "cluster": {},
  "index": {
    "auditbeat-*": {
      "all": false,
      "read": true,
      "view_index_metadata": true
    },
    "packetbeat-*": {
      "all": false,
      "read": false,
      "view_index_metadata": false
    }
  },
  "application": {}
}
--> authorizedPatterns [
  "auditbeat-*"
]

❌ NOT EXPECTED: The authorizedPatterns in the Kibana server log output for the somepacketbeat user does NOT include Packeatbeat, even though the stats API (correctly) returns results for data streams created by the 8.x version of packetbeat.

It appears that Packetbeat is not included in authorizedPatterns because the privileges entry for Packetbeat in the console.log output:

    "packetbeat-*": {
      "all": false,
      "read": false,
      "view_index_metadata": false
    }
  },

has "read": false and "view_index_metadata": false.

It appears both read and view_index_metadata are false for the somepacketbeat user because the packetbeat-* pattern matches data streams created by the 8.x version of Packetbeat (e.g. .ds-packetbeat-8.9.2-2023.12.06-000001), which the user has access to, AND indices created by the 7.x version (e.g. packetbeat-7.17.9-2024.01.08-000002), which the somepacketbeat user does NOT have access to.

Conclusion: it appears that the current implementation of authorization returns incorrect results when the specified pattern is not applicable to ALL data streams and / or indices that match the pattern.

[andrew-goldstein]

Thanks @semd for creating the new reusable Data Stream Adapter package, @kbn/data-stream-adapter, and consuming it to persist results in the Data Quality Dashboard! 🙏

The new package is useful for this use case (Data Quality dashboard persistence), and for creating data streams in future use cases.

Desk testing

🟢 Plugins that create new data streams via the @kbn/data-stream-adapter package at startup (when Kibana server starts) are resilient to errors that may occur.

For example, at least 4 API calls are made (some in parallel) at startup to create the data stream (metadata, without a backing index), index template, and component templates required to persist Data Quality dashboard results.

✅ Any one of the 4 API calls in this example may fail. While desk testing, none of those (simulated) API failures will prevent Kibana from starting. Details of the failure (at startup) are logged to the Kibana log.

✅ For some use cases, like the Data Quality dashboard, the backing indices for a data stream may only be created on demand (when users access a page), because the newly-created indices are space-specific. Errors that occur when attempting to create space-specific backing indices (on demand) are also logged to the Kibana log.

✅ The Data Quality dashboard displays error toasts when errors occur while reading or writing persisted results

🟡 Consider merging this PR to make the @kbn/data-stream-adapter package available to PRs that may consume it, but for now, consider disabling the creation of the data streams required for persisting Data Quality results at startup. This approach would enable the content of the Data Quality checks to evolve in a follow-up PR without the need to migrate mappings and results from their current state (in this PR) to a future state. Similarly, this would allow for revisions to the authorization check that's specific to data quality results.

LGTM 🚀

[semd]

@elasticmachine merge upstream

[semd]

🟡 Consider merging this PR to make the @kbn/data-stream-adapter package available to #173487, but for now, consider disabling the creation of the data streams required for persisting Data Quality results at startup. This approach would enable the #173185 (comment) of the Data Quality checks to evolve in a follow-up PR without the need to migrate mappings and results from their current state (in this PR) to a future state. Similarly, this would allow for revisions to the authorization check that's specific to data quality results.

Sure, I am going to disable the persistence layer of the data quality dashboard so we can merge and make the data stream adapter library available in main.
The persistence for the data quality dashboard will be enabled in a follow-up PR after completing the following tasks:

• Store unitary results on a per-index basis instead of per-pattern, removing unnecessary fields.
• Change the query to get the stored results by aggregating documents with indexName matching the patterns parameter.
• Check indexName authorization of the results returned by the query and then filter the results, instead of filtering the unauthorized patterns parameter before querying.

[angorayc]

Thanks for adding this feature, Sergi, decent work! Very clean code and well structured. 👍

[semd]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: fac328a

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/data-stream-adapter	-	80	+80

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.2MB	11.2MB	+1.7KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/data-stream-adapter	-	1	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	68.4KB	68.9KB	+572.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/data-stream-adapter	-	80	+80

ESLint disabled line counts

id	before	after	diff
@kbn/data-stream-adapter	-	2	+2
ecsDataQualityDashboard	5	7	+2
total			+4

Total ESLint disabled count

id	before	after	diff
@kbn/data-stream-adapter	-	2	+2
ecsDataQualityDashboard	5	7	+2
total			+4

History

• 💔 Build #189002 failed 4edfbd0
• 💔 Build #188969 failed cca0dfe
• 💔 Build #188923 failed 4a136f4
• 💛 Build #188413 was flaky cde29d8
• 💚 Build #188334 succeeded 4517e84

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @semd