Merge branch 'main' into issues/116748

.github/CODEOWNERS

@@ -313,9 +313,11 @@
 /x-pack/plugins/encrypted_saved_objects/ @elastic/kibana-security
 /x-pack/plugins/security/ @elastic/kibana-security
 /x-pack/test/api_integration/apis/security/ @elastic/kibana-security
+/x-pack/test/api_integration/apis/spaces/ @elastic/kibana-security
 /x-pack/test/ui_capabilities/ @elastic/kibana-security
 /x-pack/test/encrypted_saved_objects_api_integration/ @elastic/kibana-security
 /x-pack/test/functional/apps/security/ @elastic/kibana-security
+/x-pack/test/functional/apps/spaces/ @elastic/kibana-security
 /x-pack/test/security_api_integration/ @elastic/kibana-security
 /x-pack/test/security_functional/ @elastic/kibana-security
 /x-pack/test/spaces_api_integration/ @elastic/kibana-security

api_docs/cases.mdx

@@ -12,13 +12,13 @@ import casesObj from './cases.json';
 
 The Case management system in Kibana
 
-Contact [Security Solution Threat Hunting](https://github.com/orgs/elastic/teams/security-threat-hunting) for questions regarding this plugin.
+Contact [ResponseOps](https://github.com/orgs/elastic/teams/response-ops) for questions regarding this plugin.
 
 **Code health stats**
 
-| Public API count  | Any count | Items lacking comments | Missing exports |
-|-------------------|-----------|------------------------|-----------------|
-| 83 | 0 | 57 | 23 |
+| Public API count | Any count | Items lacking comments | Missing exports |
+| ---------------- | --------- | ---------------------- | --------------- |
+| 83               | 0         | 57                     | 23              |
 
 ## Client
 

api_docs/plugin_directory.mdx

@@ -31,7 +31,7 @@ warning: This document is auto-generated and is meant to be viewed inside our ex
 | <DocLink id="kibBannersPluginApi" text="banners"/> | [Kibana Core](https://github.com/orgs/elastic/teams/kibana-core) | - | 9 | 0 | 9 | 0 |
 | <DocLink id="kibBfetchPluginApi" text="bfetch"/> | [App Services](https://github.com/orgs/elastic/teams/kibana-app-services) | Considering using bfetch capabilities when fetching large amounts of data. This services supports batching HTTP requests and streaming responses back. | 76 | 1 | 67 | 2 |
 | <DocLink id="kibCanvasPluginApi" text="canvas"/> | [Kibana Presentation](https://github.com/orgs/elastic/teams/kibana-presentation) | Adds Canvas application to Kibana | 9 | 0 | 8 | 3 |
-| <DocLink id="kibCasesPluginApi" text="cases"/> | [Security Solution Threat Hunting](https://github.com/orgs/elastic/teams/security-threat-hunting) | The Case management system in Kibana | 83 | 0 | 57 | 23 |
+| <DocLink id="kibCasesPluginApi" text="cases"/> | [ResponseOps](https://github.com/orgs/elastic/teams/response-ops) | The Case management system in Kibana | 83 | 0 | 57 | 23 |
 | <DocLink id="kibChartsPluginApi" text="charts"/> | [Vis Editors](https://github.com/orgs/elastic/teams/kibana-vis-editors) | - | 314 | 2 | 281 | 4 |
 | <DocLink id="kibCloudPluginApi" text="cloud"/> | [Kibana Core](https://github.com/orgs/elastic/teams/kibana-core) | - | 22 | 0 | 22 | 0 |
 | <DocLink id="kibConsolePluginApi" text="console"/> | [Stack Management](https://github.com/orgs/elastic/teams/kibana-stack-management) | - | 13 | 0 | 13 | 1 |

docs/CHANGELOG.asciidoc

@@ -134,17 +134,19 @@ The `xpack.apm.autocreateApmIndexPattern` APM setting has been removed. For more
 *Impact* +
 To automatically create data views in APM, use `xpack.apm.autoCreateApmDataView`.
 ====
-      
+
 [discrete]
 [[deprecation-119494]]
-.Updates Fleet API responses for consistency
+.Updates Fleet API to improve consistency
 [%collapsible]
 ====
 *Details* +
-To make sure all Fleet API GET resposes return `items`, the following have been updated:
+The Fleet API has been updated to improve consistency:
 
-* `/api/fleet/enrollment-api-keys`
-* `/api/fleet/agents`
+* Hyphens are changed to underscores in some names.
+* The `pkgkey` path parameter in the packages endpoint is split.
+* The `response` and `list` properties are renamed to `items` or `item` in some
+responses.
 
 For more information, refer to {kibana-pull}119494[#119494].
 
@@ -157,24 +159,30 @@ When you upgrade to 8.0.0, use the following API changes:
 
 * Use `service_tokens` instead of `service-tokens`.
 
-* `check-permissions` is no longer supported.
-
 * Use `/epm/packages/{packageName}/{version}` instead of `/epm/packages/{pkgkey}`.
 
-* Use `items[]` or `item` instead of `response[]` in the following:
-
+* Use `items[]` instead of `response[]` in:
++
 [source,text]
 --
+/api/fleet/enrollment_api_keys
+/api/fleet/agents
 /epm/packages/
-/epm/packages/{pkgkey}
 /epm/categories
 /epm/packages/_bulk
 /epm/packages/limited
+/epm/packages/{packageName}/{version} <1>
 --
+<1> Use `items[]` when the verb is `POST` or `DELETE`. Use `item` when the verb
+is `GET` or `PUT`.
+
+For more information, refer to {fleet-guide}/fleet-api-docs.html[Fleet APIs].
+
 ====
 
-To review the depcrecations in previous versions, refer to the <<deprecations-8.0.0-alpha1,8.0.0-alpha1 release notes>>. 
-
+To review the deprecations in previous versions, refer to the <<deprecations-8.0.0-alpha1,8.0.0-alpha1 release notes>>. 
+
+
 [float]
 [[features-8.0.0-rc1]]
 === Features

docs/apm/troubleshooting.asciidoc

@@ -185,7 +185,7 @@ There are two things you can do to if you'd like to ensure a field is searchable
 1. Index your additional data as {apm-guide-ref}/metadata.html[labels] instead.
 These are dynamic by default, which means they will be indexed and become searchable and aggregatable.
 
-2. Use the {apm-guide-ref}/configuration-template.html[`append_fields`] feature. As an example,
+2. Use the `append_fields` feature. As an example,
 adding the following to `apm-server.yml` will enable dynamic indexing for `http.request.cookies`:
 
 [source,yml]

docs/management/connectors/action-types/email.asciidoc

@@ -16,7 +16,7 @@ NOTE: For emails to have a footer with a link back to {kib}, set the <<server-pu
 Email connectors have the following configuration properties.
 
 Name::      The name of the connector. The name is used to identify a  connector in the management UI connector listing, or in the connector list when configuring an action.
-Sender::    The from address for all emails sent with this connector. This can be specified in `user@host-name` format or as `"human name <user@host-name>"` format. See the https://nodemailer.com/message/addresses/[Nodemailer address documentation] for more information.
+Sender::    The from address for all emails sent with this connector. This must be specified in `user@host-name` format. See the https://nodemailer.com/message/addresses/[Nodemailer address documentation] for more information.
 Service::   The name of the email service. If `service` is one of Nodemailer's https://nodemailer.com/smtp/well-known/[well-known email service providers], the `host`, `port`, and `secure` properties are defined with the default values and disabled for modification. If `service` is `MS Exchange Server`, the `host`, `port`, and `secure` properties are ignored and `tenantId`, `clientId`, `clientSecret` are required instead. If `service` is `other`, the `host` and `port` properties must be defined.
 Host::      Host name of the service provider. If you are using the <<action-settings, `xpack.actions.allowedHosts`>> setting, make sure this hostname is added to the allowed hosts.
 Port::      The port to connect to on the service provider.

packages/elastic-apm-synthtrace/src/scripts/examples/01_simple_trace.ts

@@ -12,14 +12,14 @@ import { getApmWriteTargets } from '../../lib/apm/utils/get_apm_write_targets';
 import { Scenario } from '../scenario';
 import { getCommonServices } from '../utils/get_common_services';
 
-const scenario: Scenario = async ({ target, logLevel }) => {
+const scenario: Scenario = async ({ target, logLevel, scenarioOpts }) => {
   const { client, logger } = getCommonServices({ target, logLevel });
   const writeTargets = await getApmWriteTargets({ client });
 
+  const { numServices = 3 } = scenarioOpts || {};
+
   return {
     generate: ({ from, to }) => {
-      const numServices = 3;
-
       const range = timerange(from, to);
 
       const transactionName = '240rpm/75% 1000ms';

packages/elastic-apm-synthtrace/src/scripts/run.ts

@@ -69,6 +69,12 @@ function options(y: Argv) {
       describe: 'Target to index',
       string: true,
     })
+    .option('scenarioOpts', {
+      describe: 'Options specific to the scenario',
+      coerce: (arg) => {
+        return arg as Record<string, any> | undefined;
+      },
+    })
     .conflicts('to', 'live');
 }
 

packages/elastic-apm-synthtrace/src/scripts/utils/parse_run_cli_flags.ts

@@ -47,7 +47,15 @@ export function parseRunCliFlags(flags: RunCliFlags) {
   }
 
   return {
-    ...pick(flags, 'target', 'workers', 'clientWorkers', 'batchSize', 'writeTarget'),
+    ...pick(
+      flags,
+      'target',
+      'workers',
+      'clientWorkers',
+      'batchSize',
+      'writeTarget',
+      'scenarioOpts'
+    ),
     intervalInMs,
     bucketSizeInMs,
     logLevel: parsedLogLevel,

packages/elastic-apm-synthtrace/src/scripts/utils/start_historical_data_upload.ts

@@ -24,6 +24,7 @@ export async function startHistoricalDataUpload({
   target,
   file,
   writeTarget,
+  scenarioOpts,
 }: RunOptions & { from: number; to: number }) {
   let requestedUntil: number = from;
 
@@ -57,6 +58,7 @@ export async function startHistoricalDataUpload({
       target,
       workers,
       writeTarget,
+      scenarioOpts,
     };
 
     const worker = new Worker(Path.join(__dirname, './upload_next_batch.js'), {

packages/elastic-apm-synthtrace/src/scripts/utils/start_live_data_upload.ts

@@ -24,6 +24,7 @@ export async function startLiveDataUpload({
   logLevel,
   workers,
   writeTarget,
+  scenarioOpts,
 }: RunOptions & { start: number }) {
   let queuedEvents: ElasticsearchOutput[] = [];
   let requestedUntil: number = start;
@@ -41,6 +42,7 @@ export async function startLiveDataUpload({
     target,
     workers,
     writeTarget,
+    scenarioOpts,
   });
 
   function uploadNextBatch() {

packages/elastic-apm-synthtrace/src/scripts/utils/upload_next_batch.ts

@@ -17,6 +17,7 @@ export interface WorkerData {
   bucketFrom: number;
   bucketTo: number;
   file: string;
+  scenarioOpts: Record<string, any> | undefined;
   logLevel: LogLevel;
   clientWorkers: number;
   batchSize: number;
@@ -39,6 +40,7 @@ const {
   workers,
   target,
   writeTarget,
+  scenarioOpts,
 } = workerData as WorkerData;
 
 async function uploadNextBatch() {
@@ -63,6 +65,7 @@ async function uploadNextBatch() {
     target,
     workers,
     writeTarget,
+    scenarioOpts,
   });
 
   const events = logger.perf('execute_scenario', () =>

packages/kbn-es-query/src/filters/build_filters/query_string_filter.ts

@@ -39,11 +39,9 @@ export const isQueryStringFilter = (filter: Filter): filter is QueryStringFilter
  *
  * @public
  */
-export const buildQueryFilter = (query: QueryStringFilter['query'], index: string, alias: string) =>
-  ({
-    query,
-    meta: {
-      index,
-      alias,
-    },
-  } as QueryStringFilter);
+export const buildQueryFilter = (
+  query: QueryStringFilter['query'],
+  index: string,
+  alias?: string,
+  meta: QueryStringFilterMeta = {}
+) => ({ query, meta: { index, alias, ...meta } });

packages/kbn-typed-react-router-config/src/types/index.ts

@@ -257,6 +257,18 @@ type MapRoutes<TRoutes, TParents extends Route[] = []> = TRoutes extends [Route]
       MapRoute<TRoutes[8], TParents> &
       MapRoute<TRoutes[7], TParents> &
       MapRoute<TRoutes[9], TParents>
+  : TRoutes extends [Route, Route, Route, Route, Route, Route, Route, Route, Route, Route, Route]
+  ? MapRoute<TRoutes[0], TParents> &
+      MapRoute<TRoutes[1], TParents> &
+      MapRoute<TRoutes[2], TParents> &
+      MapRoute<TRoutes[3], TParents> &
+      MapRoute<TRoutes[4], TParents> &
+      MapRoute<TRoutes[5], TParents> &
+      MapRoute<TRoutes[6], TParents> &
+      MapRoute<TRoutes[8], TParents> &
+      MapRoute<TRoutes[7], TParents> &
+      MapRoute<TRoutes[9], TParents> &
+      MapRoute<TRoutes[10], TParents>
   : {};
 
 // const element = null as any;

src/core/server/elasticsearch/client/cluster_client.test.ts

@@ -144,13 +144,13 @@ describe('ClusterClient', () => {
       });
     });
 
-    it('creates a scoped facade with filtered auth headers', () => {
+    it('does not filter auth headers', () => {
       const config = createConfig({
         requestHeadersWhitelist: ['authorization'],
       });
       getAuthHeaders.mockReturnValue({
         authorization: 'auth',
-        other: 'nope',
+        other: 'yep',
       });
 
       const clusterClient = new ClusterClient(config, logger, 'custom-type', getAuthHeaders);
@@ -160,7 +160,12 @@ describe('ClusterClient', () => {
 
       expect(scopedClient.child).toHaveBeenCalledTimes(1);
       expect(scopedClient.child).toHaveBeenCalledWith({
-        headers: { ...DEFAULT_HEADERS, authorization: 'auth', 'x-opaque-id': expect.any(String) },
+        headers: {
+          ...DEFAULT_HEADERS,
+          authorization: 'auth',
+          other: 'yep',
+          'x-opaque-id': expect.any(String),
+        },
       });
     });
 
@@ -170,7 +175,7 @@ describe('ClusterClient', () => {
       });
       getAuthHeaders.mockReturnValue({
         authorization: 'auth',
-        other: 'nope',
+        other: 'yep',
       });
 
       const clusterClient = new ClusterClient(config, logger, 'custom-type', getAuthHeaders);
@@ -184,7 +189,12 @@ describe('ClusterClient', () => {
 
       expect(scopedClient.child).toHaveBeenCalledTimes(1);
       expect(scopedClient.child).toHaveBeenCalledWith({
-        headers: { ...DEFAULT_HEADERS, authorization: 'auth', 'x-opaque-id': expect.any(String) },
+        headers: {
+          ...DEFAULT_HEADERS,
+          authorization: 'auth',
+          other: 'yep',
+          'x-opaque-id': expect.any(String),
+        },
       });
     });
 

src/core/server/elasticsearch/client/cluster_client.ts

@@ -54,8 +54,6 @@ export interface ICustomClusterClient extends IClusterClient {
 export class ClusterClient implements ICustomClusterClient {
   public readonly asInternalUser: KibanaClient;
   private readonly rootScopedClient: KibanaClient;
-  private readonly allowListHeaders: string[];
-
   private isClosed = false;
 
   constructor(
@@ -72,8 +70,6 @@ export class ClusterClient implements ICustomClusterClient {
       getExecutionContext,
       scoped: true,
     });
-
-    this.allowListHeaders = ['x-opaque-id', ...this.config.requestHeadersWhitelist];
   }
 
   asScoped(request: ScopeableRequest) {
@@ -95,14 +91,15 @@ export class ClusterClient implements ICustomClusterClient {
   private getScopedHeaders(request: ScopeableRequest): Headers {
     let scopedHeaders: Headers;
     if (isRealRequest(request)) {
-      const requestHeaders = ensureRawRequest(request).headers;
+      const requestHeaders = ensureRawRequest(request).headers ?? {};
       const requestIdHeaders = isKibanaRequest(request) ? { 'x-opaque-id': request.id } : {};
-      const authHeaders = this.getAuthHeaders(request);
+      const authHeaders = this.getAuthHeaders(request) ?? {};
 
-      scopedHeaders = filterHeaders(
-        { ...requestHeaders, ...requestIdHeaders, ...authHeaders },
-        this.allowListHeaders
-      );
+      scopedHeaders = {
+        ...filterHeaders(requestHeaders, this.config.requestHeadersWhitelist),
+        ...requestIdHeaders,
+        ...authHeaders,
+      };
     } else {
       scopedHeaders = filterHeaders(request?.headers ?? {}, this.config.requestHeadersWhitelist);
     }

src/core/server/http/router/headers.ts

@@ -61,7 +61,7 @@ export function filterHeaders(
   headers: Headers,
   fieldsToKeep: string[],
   fieldsToExclude: string[] = []
-) {
+): Headers {
   const fieldsToExcludeNormalized = fieldsToExclude.map(normalizeHeaderField);
   // Normalize list of headers we want to allow in upstream request
   const fieldsToKeepNormalized = fieldsToKeep

src/core/server/saved_objects/migrations/integration_tests/batch_size_bytes.test.ts

@@ -67,7 +67,7 @@ describe('migration v2', () => {
         es: {
           license: 'basic',
           dataArchive: Path.join(__dirname, 'archives', '7.14.0_xpack_sample_saved_objects.zip'),
-          esArgs: ['http.max_content_length=1715275b'],
+          esArgs: ['http.max_content_length=1715329b'],
         },
       },
     }));
@@ -85,7 +85,7 @@ describe('migration v2', () => {
   });
 
   it('completes the migration even when a full batch would exceed ES http.max_content_length', async () => {
-    root = createRoot({ maxBatchSizeBytes: 1715275 });
+    root = createRoot({ maxBatchSizeBytes: 1715329 });
     esServer = await startES();
     await root.preboot();
     await root.setup();
@@ -109,7 +109,7 @@ describe('migration v2', () => {
     await root.preboot();
     await root.setup();
     await expect(root.start()).rejects.toMatchInlineSnapshot(
-      `[Error: Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715274 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.]`
+      `[Error: Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715329 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.]`
     );
 
     await retryAsync(
@@ -122,7 +122,7 @@ describe('migration v2', () => {
         expect(
           records.find((rec) =>
             rec.message.startsWith(
-              `Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715274 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.`
+              `Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715329 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.`
             )
           )
         ).toBeDefined();