[Security Solution] Data quality dashboard persistence (elastic#173185)

## Summary

issue elastic/security-team#7382

### Data Stream Adapter

This PR introduces the `@kbn/data-stream-adapter` package, which is a
utility library to facilitate Data Stream creation and maintenance in
Kibana, it was inspired by the data stream implementation in the Alerts
plugin.
The library has two exports:

- `DataStreamSpacesAdapter`: to manage space data streams. It uses the
`name-of-the-data-stream-<spaceId>` naming pattern.

- `DataStreamAdapter`: to manage single (not space-aware) data streams.

Usage examples in the package
[README](https://github.com/elastic/kibana/blob/450be0369decdef156902d90a5f7292250ebd8cb/packages/kbn-data-stream-adapter/README.md)

### Data Quality Dashboard

The `DataStreamSpacesAdapter` has been integrated into the data quality
dashboard to store all the quality checks users perform. The information
stored is the metadata (also used for telemetry) and the actual data
rendered in the tables.

FieldMap definition
[here](https://github.com/elastic/kibana/blob/450be0369decdef156902d90a5f7292250ebd8cb/x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts)

### Demo


https://github.com/elastic/kibana/assets/17747913/311a0bf5-004b-46d7-8140-52a233361c91

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Sander Philipse <94373878+sphilipse@users.noreply.github.com>
Co-authored-by: JD Kurma <JDKurma@gmail.com>
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com>

.eslintrc.js

@@ -1121,6 +1121,7 @@ module.exports = {
         'x-pack/plugins/security_solution_serverless/**/*.{js,mjs,ts,tsx}',
         'x-pack/plugins/timelines/**/*.{js,mjs,ts,tsx}',
         'x-pack/plugins/cases/**/*.{js,mjs,ts,tsx}',
+        'packages/kbn-data-stream-adapter/**/*.{js,mjs,ts,tsx}',
       ],
       plugins: ['eslint-plugin-node', 'react'],
       env: {
@@ -1218,6 +1219,8 @@ module.exports = {
             'x-pack/plugins/security_solution_ess/**/*.{js,mjs,ts,tsx}',
             'x-pack/plugins/security_solution_serverless/**/*.{js,mjs,ts,tsx}',
             'x-pack/plugins/cases/**/*.{js,mjs,ts,tsx}',
+            'x-pack/plugins/ecs_data_quality_dashboard/**/*.{js,mjs,ts,tsx}',
+            'packages/kbn-data-stream-adapter/**/*.{js,mjs,ts,tsx}',
           ],
           rules: {
             '@typescript-eslint/consistent-type-imports': 'error',

.github/CODEOWNERS

@@ -319,6 +319,7 @@ x-pack/packages/kbn-data-forge @elastic/obs-ux-management-team
 src/plugins/data @elastic/kibana-visualizations @elastic/kibana-data-discovery
 test/plugin_functional/plugins/data_search @elastic/kibana-data-discovery
 packages/kbn-data-service @elastic/kibana-visualizations @elastic/kibana-data-discovery
+packages/kbn-data-stream-adapter @elastic/security-threat-hunting-explore
 src/plugins/data_view_editor @elastic/kibana-data-discovery
 examples/data_view_field_editor_example @elastic/kibana-data-discovery
 src/plugins/data_view_field_editor @elastic/kibana-data-discovery

package.json

@@ -369,6 +369,7 @@
     "@kbn/data-plugin": "link:src/plugins/data",
     "@kbn/data-search-plugin": "link:test/plugin_functional/plugins/data_search",
     "@kbn/data-service": "link:packages/kbn-data-service",
+    "@kbn/data-stream-adapter": "link:packages/kbn-data-stream-adapter",
     "@kbn/data-view-editor-plugin": "link:src/plugins/data_view_editor",
     "@kbn/data-view-field-editor-example-plugin": "link:examples/data_view_field_editor_example",
     "@kbn/data-view-field-editor-plugin": "link:src/plugins/data_view_field_editor",

packages/kbn-data-stream-adapter/README.md

@@ -0,0 +1,69 @@
+# @kbn/data-stream-adapter
+
+Utility library for Elasticsearch data stream management.
+
+## DataStreamAdapter
+
+Manage single data streams. Example:
+
+```
+// Setup
+const dataStream = new DataStreamAdapter('my-awesome-datastream', { kibanaVersion: '8.12.1' });
+
+dataStream.setComponentTemplate({
+    name: 'awesome-component-template',
+    fieldMap: {
+        'awesome.field1: { type: 'keyword', required: true },
+        'awesome.nested.field2: { type: 'number', required: false },
+        // ...
+    },
+});
+
+dataStream.setIndexTemplate({
+    name: 'awesome-index-template',
+    componentTemplateRefs: ['awesome-component-template', 'ecs-component-template'],
+    template: {
+        lifecycle: {
+            data_retention: '5d',
+        },
+    },
+});
+
+// Start
+await dataStream.install({ logger, esClient, pluginStop$ }); // Installs templates and the data stream, or updates existing.
+```
+
+
+## DataStreamSpacesAdapter
+
+Manage data streams per space. Example:
+
+```
+// Setup
+const spacesDataStream = new DataStreamSpacesAdapter('my-awesome-datastream', { kibanaVersion: '8.12.1' });
+
+spacesDataStream.setComponentTemplate({
+    name: 'awesome-component-template',
+    fieldMap: {
+        'awesome.field1: { type: 'keyword', required: true },
+        'awesome.nested.field2: { type: 'number', required: false },
+        // ...
+    },
+});
+
+spacesDataStream.setIndexTemplate({
+    name: 'awesome-index-template',
+    componentTemplateRefs: ['awesome-component-template', 'ecs-component-template'],
+    template: {
+        lifecycle: {
+            data_retention: '5d',
+        },
+    },
+});
+
+// Start
+await spacesDataStream.install({ logger, esClient, pluginStop$ }); // Installs templates and updates existing data streams.
+
+// Create a space data stream on the fly
+await spacesDataStream.installSpace('space2'); // creates 'my-awesome-datastream-space2' data stream if it does not exist.
+```

packages/kbn-data-stream-adapter/index.ts

@@ -0,0 +1,20 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+export { DataStreamAdapter } from './src/data_stream_adapter';
+export { DataStreamSpacesAdapter } from './src/data_stream_spaces_adapter';
+export { retryTransientEsErrors } from './src/retry_transient_es_errors';
+export { ecsFieldMap, type EcsFieldMap } from './src/field_maps/ecs_field_map';
+
+export type {
+  DataStreamAdapterParams,
+  SetComponentTemplateParams,
+  SetIndexTemplateParams,
+  InstallParams,
+} from './src/data_stream_adapter';
+export * from './src/field_maps/types';

packages/kbn-data-stream-adapter/jest.config.js

@@ -0,0 +1,13 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+module.exports = {
+  preset: '@kbn/test',
+  rootDir: '../..',
+  roots: ['<rootDir>/packages/kbn-data-stream-adapter'],
+};

packages/kbn-data-stream-adapter/kibana.jsonc

@@ -0,0 +1,5 @@
+{
+  "type": "shared-common",
+  "id": "@kbn/data-stream-adapter",
+  "owner": "@elastic/security-threat-hunting-explore"
+}

packages/kbn-data-stream-adapter/package.json

@@ -0,0 +1,7 @@
+{
+  "name": "@kbn/data-stream-adapter",
+  "version": "1.0.0",
+  "description": "Utility library for Elasticsearch Data Stream management",
+  "license": "SSPL-1.0 OR Elastic License 2.0",
+  "private": true
+}