[Bug] _metadata rename into Metadata

[redxeagle]

Hi,

currently, we play around with the usage of the schema. We have trouble with the metadata field.

Is there any reason why the Metadata field is named _metadata?
Because also Elasticsearch mentioned here: "Field names beginning with _ are not supported"

There is another feature request to allow structured data on the root level. This would also be a solution. But we need a way to extend the logging with our own fields for business requirements. The schema does not fit here for all cases (e.g. we want to have all headers of the request logged)

[RowlandBanks]

I've also suffered from the same behaviour. One easy workaround is to add an ingestion pipeline with a rename processor to rename the _metadata field to something without an underscore (we used app for example).

First, create a new pipeline with the rename processor:

// Create a pipeline to rename the _metadata field to app (fields starting with an underscore cannot be indexed by Elastic)
// We ignore failures because a failure to rename will be very obvious - there will be no app field.
PUT /_ingest/pipeline/application_pipeline
{
  "description": "",
  "processors": [
	{
	  "rename": {
		"field": "_metadata",
		"target_field": "app",
		"ignore_failure": true,
		"tag": "Rename _metadata to app"
	  }
	}
  ]
}

Then, when creating your index, give it a default_pipeline, for example:

PUT /application-000001
{
  "settings": {
    "default_pipeline": "application_pipeline"
  },
  "aliases": {
    "application-write":{
      "is_write_index": true 
    }
  }
}

If you are using index templates, make sure to set the default pipeline on the template, not the index, or it will be lost as soon as your index rolls over:

PUT /_template/application-index-template
{
  "index_patterns":["application-*"],
  "settings" : {
    "index" : {
      "lifecycle" : {
        "name" : "application-ilm-policy",
        "rollover_alias" : "application-write"
      },
      "default_pipeline": "application_pipeline"
    }
  },
  // <snip>

[RascoApps]

Is this related to a schema change, why is it incorrect?

[RowlandBanks]

@RascoApps the problem with _metadata is that Elastic won't index fields starting with an underscore.

So all the application specific bits of logging one is are doing aren't being indexed by Elastic, so you can't search or filter on them.

I don't think there's a schema change that has caused this - it seems to have been the behaviour for a long time.

[kwsorensen]

I made a pull request to changed _metadata to metadata.

[RascoApps]

Maybe it should be changed to _meta? https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-meta-field.html

[RowlandBanks]

Maybe it should be changed to _meta?

@RascoApps The problem with _meta is it still starts with an underscore, so will suffer from the same "Elastic won't index it" problem.

I like metadata as a solution - ideally as a configurable default.

[yorondevops]

Hi, commenting here to see if it's possible to get the ball moving on this issue.

I'm using the NLog ECS Layout and want to serialize key-value pairs from the MDLC as top-level fields into my log messages.

Looks like I'm running into the same issue as everyone else here where the ECS Layout prepends _metadata to the field names in the messages, making them unindexable.

Does anyone know a way to get around this problem?

[ghost]

Are you still going to implement this?

I am using the NLog component and noticed "_metadata" gets ignored in the most recent version of Elastic.

I created a subclass for EcsLayout which will rename "_metadata" to "metadata" upon serialization. However, I don't want to conflict with any planned future use-case for the "metadata" node.

If so, I can use just "data" that aligns with the Azure Event Grid schema for custom data.

[russcam]

This has been addressed in #138 by renaming _metadata to metadata.

What would be most useful to folks:

1. Release a new 1.5.x version now that includes this fix
2. Wait on a 1.6.x version that updates to ECS 1.6 and includes this fix. This requires some changes in the generator to accommodate changes in the ECS schema and will take a bit more time than a 1.5.x release.

Please vote on this comment with a 👍 for 1 and a ❤️ for 2.

Renaming the output of a field fixes the issue, but could be a breaking change for folks that might be expecting the _metadata field to be present

[russcam]

Thanks for voting! I will release a new 1.5.x version shortly, followed by a 1.6.x version

[russcam]

1.5.3 has now been released to Nuget that includes the rename from "_metadata" to "metadata".

Thank you for your patience!