Support custom metadata on Data Streams #59195

cjcenizal · 2020-07-07T21:07:24Z

The intention behind this feature is to enable the UI to identify the data streams created by Ingest Manager. We'll need to implement a solution that allows index templates to define the metadata that is applied to the data streams that they create.

elasticmachine · 2020-07-07T21:07:26Z

Pinging @elastic/es-core-features (:Core/Features/Data streams)

afharo · 2020-07-15T13:47:12Z

We need this as well for the changes in elastic/kibana#71419.
Ideally, the _meta info in the Data Streams should be copied over to the underlying indices it creates.

dakrone · 2020-07-15T15:54:36Z

Ideally, the _meta info in the Data Streams should be copied over to the underlying indices it creates.

We first need to support _meta on indices themselves, which is probably a separate issue

afharo · 2020-07-15T16:34:49Z

@dakrone what do you mean with support _meta on indices themselves?

I thought it is already supported. Kibana uses that feature in the .kibana index and AFAIK, Beats does as well. I'm talking about mappings._meta though. Maybe that's my confusion?

dakrone · 2020-07-15T16:55:27Z

Mappings metadata is more thought of as metadata about the mappings themselves, not the index (at least that's they way I think about it). It would work for now, but I don't think we should use it as a general purpose metadata storage for the index.

afharo · 2020-07-16T10:10:39Z

@dakrone I agree! Thank you for explaining!

yuliacech · 2020-10-12T14:12:12Z

@dakrone Hi Lee, I was wondering if there is any work planned on adding metadata to data streams? I see now that we have

"_meta": {
    "package": {
      "name": "system"
    },
    "managed_by": "ingest-manager",
    "managed": true
  }

in both index templates mappings and index mappings if they were created by ingest manager.

dakrone · 2020-10-13T21:13:46Z

@yuliacech it's still something we'd like to do, but I don't think it has anyone currently working on adding it.

martijnvg · 2020-10-19T11:13:52Z

I think this is something that we can pickup in the near future.

When a data stream is created then custom data stream metadata should be sourced from the matching composable index template and be added to the data stream under the _meta key. This custom metadata can then be retrieved via the get data stream api.

I do have a question about from where the data stream custom metadata is sourced. Should all the keys under the current _meta json object on a composable index template be added to a data stream's custom metadata? Or only a subset of keys? Or should the data_stream object on a composable template have its own _meta object?

Also is there a need to update a data stream's custom metadata after its creation? Currently there isn't a data stream update api or something similar.

yuliacech · 2020-10-20T13:12:57Z

Hi @martijnvg, thanks a lot for getting back to me on this!
I can tell that in Index Management we want to show which data streams are managed by Ingest Manager, so probably using "managed_by"and "managed"keys. Currently, "package" is probably not needed but might be in the future for more info in the UI.
There is no UI for updating data stream at the moment, not sure about planning for it though, @cjcenizal maybe has more info on this?

martijnvg · 2020-10-20T13:44:04Z

@yuliacech Besides the mentioned keys, would it be okay if other keys also appear on a data stream? More specifically i'm wondering whether there is a need for custom meta keys to only be associated with a data stream or if it is okay if all custom meta data keys from the composable template appear on a data stream and in other places.

yuliacech · 2020-10-20T14:08:56Z

@martijnvg additional keys should not be a problem for UI, I think we can safely ignore them and only use the ones for managed info.

martijnvg · 2020-10-20T14:38:04Z

@yuliacech that is good to know. That will simplify the change that we need to make.

cjcenizal · 2020-10-20T20:51:48Z

Also is there a need to update a data stream's custom metadata after its creation? Currently there isn't a data stream update api or something similar.

I'm not aware of a need at this time.

@timestamp

Composable index template may hold custom metadata. This change adds behaviour that when a data stream gets created the custom metadata of the matching composable index template is copied to new data stream. The get data stream api can then be used to view the custom metadata. Example: ``` PUT /_index_template/my-logs-template { "index_patterns": [ "logs-*" ], "data_stream": { }, "template": { "settings": { "index.number_of_replicas": 0 } }, "_meta": { "managed": true } } PUT /_data_stream/logs-myapp GET /_data_stream ``` The get data stream api then yields the following response: ``` { "data_streams": [ { "name": "logs-myapp", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-logs-myapp-000001", "index_uuid": "3UaBxM3mQXuHR6qx0IDVCw" } ], "generation": 1, "_meta": { "managed": true }, "status": "GREEN", "template": "my-logs-template" } ] } ``` Closes elastic#59195

@timestamp

Composable index template may hold custom metadata. This change adds behaviour that when a data stream gets created the custom metadata of the matching composable index template is copied to new data stream. The get data stream api can then be used to view the custom metadata. Example: ``` PUT /_index_template/my-logs-template { "index_patterns": [ "logs-*" ], "data_stream": { }, "template": { "settings": { "index.number_of_replicas": 0 } }, "_meta": { "managed": true } } PUT /_data_stream/logs-myapp GET /_data_stream ``` The get data stream api then yields the following response: ``` { "data_streams": [ { "name": "logs-myapp", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-logs-myapp-000001", "index_uuid": "3UaBxM3mQXuHR6qx0IDVCw" } ], "generation": 1, "_meta": { "managed": true }, "status": "GREEN", "template": "my-logs-template" } ] } ``` Closes #59195

@timestamp

Composable index template may hold custom metadata. This change adds behaviour that when a data stream gets created the custom metadata of the matching composable index template is copied to new data stream. The get data stream api can then be used to view the custom metadata. Example: ``` PUT /_index_template/my-logs-template { "index_patterns": [ "logs-*" ], "data_stream": { }, "template": { "settings": { "index.number_of_replicas": 0 } }, "_meta": { "managed": true } } PUT /_data_stream/logs-myapp GET /_data_stream ``` The get data stream api then yields the following response: ``` { "data_streams": [ { "name": "logs-myapp", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-logs-myapp-000001", "index_uuid": "3UaBxM3mQXuHR6qx0IDVCw" } ], "generation": 1, "_meta": { "managed": true }, "status": "GREEN", "template": "my-logs-template" } ] } ``` Closes elastic#59195

@timestamp

Composable index template may hold custom metadata. This change adds behaviour that when a data stream gets created the custom metadata of the matching composable index template is copied to new data stream. The get data stream api can then be used to view the custom metadata. Example: ``` PUT /_index_template/my-logs-template { "index_patterns": [ "logs-*" ], "data_stream": { }, "template": { "settings": { "index.number_of_replicas": 0 } }, "_meta": { "managed": true } } PUT /_data_stream/logs-myapp GET /_data_stream ``` The get data stream api then yields the following response: ``` { "data_streams": [ { "name": "logs-myapp", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-logs-myapp-000001", "index_uuid": "3UaBxM3mQXuHR6qx0IDVCw" } ], "generation": 1, "_meta": { "managed": true }, "status": "GREEN", "template": "my-logs-template" } ] } ``` Closes #59195

cjcenizal added >enhancement :Data Management/Data streams Data streams and their lifecycles Team:Deployment Management Meta label for Management Experience - Deployment Management team labels Jul 7, 2020

elasticmachine added the Team:Data Management Meta label for data/management team label Jul 7, 2020

cjcenizal mentioned this issue Jul 7, 2020

Identify data streams created by Ingest Manager with metadata elastic/kibana#69573

Closed

afharo mentioned this issue Jul 15, 2020

[Telemetry] Data: Report dataset info only if there is known metadata elastic/kibana#71419

Merged

2 tasks

martijnvg mentioned this issue Oct 21, 2020

Add custom metadata support to data steams. #63991

Merged

martijnvg closed this as completed in #63991 Oct 26, 2020

martijnvg mentioned this issue Oct 26, 2020

Add custom metadata support to data steams. #64169

Merged

stevejgordon mentioned this issue Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom metadata on Data Streams #59195

Support custom metadata on Data Streams #59195

cjcenizal commented Jul 7, 2020 •

edited

Loading

elasticmachine commented Jul 7, 2020

afharo commented Jul 15, 2020

dakrone commented Jul 15, 2020

afharo commented Jul 15, 2020

dakrone commented Jul 15, 2020

afharo commented Jul 16, 2020

yuliacech commented Oct 12, 2020

dakrone commented Oct 13, 2020

martijnvg commented Oct 19, 2020

yuliacech commented Oct 20, 2020

martijnvg commented Oct 20, 2020

yuliacech commented Oct 20, 2020

martijnvg commented Oct 20, 2020

cjcenizal commented Oct 20, 2020

Support custom metadata on Data Streams #59195

Support custom metadata on Data Streams #59195

Comments

cjcenizal commented Jul 7, 2020 • edited Loading

elasticmachine commented Jul 7, 2020

afharo commented Jul 15, 2020

dakrone commented Jul 15, 2020

afharo commented Jul 15, 2020

dakrone commented Jul 15, 2020

afharo commented Jul 16, 2020

yuliacech commented Oct 12, 2020

dakrone commented Oct 13, 2020

martijnvg commented Oct 19, 2020

yuliacech commented Oct 20, 2020

martijnvg commented Oct 20, 2020

yuliacech commented Oct 20, 2020

martijnvg commented Oct 20, 2020

cjcenizal commented Oct 20, 2020

cjcenizal commented Jul 7, 2020 •

edited

Loading