[DataView] Enrich data view field data by ECS information (when available)

[kertal]

Description

This is a follow up of a POC evaluation (#180543) enriching data view field data by ECS information if requested. First we tried to add ECS information on Browser side, but this would add up to our bundle size, so we evaluated the approach to use the Kibana server to enrich field data provided by Elasticsearch. This approach has no impact to browser bundle size

Adding ECS field information to data view fields provides a great way to show more information to the user (description), help with prioritization, allow us to gain insights (Telemetry about ECS usage). This issue is focusing on extending the API for this purpose, out of scope is the consumption.

Extension of DataViews: _field_for_wildcard endpoint

So the current FIELDS_FOR_WILDCARD_PATH, /internal/data_views/_fields_for_wildcard, should provide functionality to return ECS information, if requested.

Those are the parameters we currently use

Server

kibana/src/plugins/data_views/server/rest_api_routes/internal/fields_for.ts

Lines 49 to 60 in ed29b97

	export interface IQuery {
	pattern: string;
	meta_fields: string | string[];
	type?: string;
	rollup_index?: string;
	allow_no_index?: boolean;
	include_unmapped?: boolean;
	fields?: string | string[];
	allow_hidden?: boolean;
	field_types?: string | string[];
	include_empty_fields?: boolean;
	}

Browser

kibana/src/plugins/data_views/server/fetcher/lib/es_api.ts

Lines 43 to 52 in ed29b97

	interface FieldCapsApiParams {
	callCluster: ElasticsearchClient;
	indices: string[] | string;
	fieldCapsOptions?: { allow_no_indices: boolean; include_unmapped?: boolean };
	indexFilter?: QueryDslQueryContainer;
	fields?: string[];
	expandWildcards?: ExpandWildcard;
	fieldTypes?: string[];
	includeEmptyFields?: boolean;
	}

I therefore should offer a new parameter includeEcsMeta(Browser) and include , which can be used in the following ways

1. if includeEcsMeta is not set, nothing happens, everything works like before
2. if includeEcsMeta is set to all, all available ECS data for a given field should be returned
3. if includeEcsMeta is set to a comma separated array, like type, description,... the ECS fields matching this list should be returned

Here is an example of ECS meta information which can be added to a field returned by the endpoint:

'agent.version': {
    dashed_name: 'agent-version',
    description: 'Version of the agent.',
    example: '6.0.0-rc2',
    flat_name: 'agent.version',
    ignore_above: 1024,
    level: 'core',
    name: 'version',
    normalize: [],
    short: 'Version of the agent.',
    type: 'keyword',
  }, 

Here is what should be returned when includeEcsMeta=all and a field name matches an ECS name

{
    "name": "agent.version",
    "type": "string",
    "esTypes": [
        "text"
    ],
    "searchable": true,
    "aggregatable": false,
    "readFromDocValues": false,
    "metadata_field": true,
    "ecs": {
       "dashed_name": "agent-version",
      "description": "Version of the agent.",
      "example": "6.0.0-rc2",
      "flat_name": "agent.version",
      "ignore_above": 1024,
      "level": "core",
      "name": "version",
      "normalize": [],
      "short": "Version of the agent.",
      "type": 'keyword',
   }
}

Here is what should be returned when includeEcsMeta=short,description and a field name matches an ECS name

{
    "name": "agent.version",
    "type": "string",
    "esTypes": [
        "text"
    ],
    "searchable": true,
    "aggregatable": false,
    "readFromDocValues": false,
    "metadata_field": true,
    "ecs":  {
      "description": "Version of the agent.",
      "short": "Version of the agent.",
   },
}

This is provided by the @elastic/ecs package (https://github.com/elastic/ecs-typescript), more information about ECS https://www.elastic.co/guide/en/ecs/master/ecs-reference.html

Consumption

This API extension allows us to make use of ECS meta data in a flexible way, a few examples.

Showing ECS descriptions on demand

For this just a single field should be requested when it should be displayed, and just the description information would be needed.

Prioritize fields based on ecs information

#181036

Improved telemetry

We could gain more insights about the usage of ECS fields

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[tonyghiani]

@kertal FYI I'm going to work on this https://github.com/elastic/observability-dev/issues/3331 in a couple of weeks, this work might be a good consumer of the service I had in mind to expose!

[kertal]

@tonyghiani thx for the heads up, yes this is definitely related and we have the same goals, so we should aim to see what we can share or consume!

[kertal]

Closing since it will be solved differently due to merge of #183806