Allow 'metadata' to an event that is not sent on output.

[jordansissel]

Originally from: https://logstash.jira.com/browse/LOGSTASH-1798

It would be great to have arbitrary metadata for an event, which isn't passed through the output and specifically not made available to the encoding/serialization phase.

Example use case:

• Fields like index_name, which are derived by a filter, and used by the elasticsearch output, but probably shouldn't be part of the event sent to Elasticsearch.

Proposed syntax is simply a fieldref namespace "[@metadata]". Anything under this is considered metadata and not show up normally through JSON or other serialization.

---

Example use (once #1644 is merged)

input {
  elasticsearch {
    host => "localhost"
    # Store ES document metadata (index, type, id, etc) in metadata
    docinfo_target => "@metadata"
  }
}

filter {
  ...
}

output {
  elasticsearch {
    action => "update"
    document_id => "%{[@metadata][id]}"
    index => "%{[@metadata][index]}"
    type => "%{[@metadata][type]}"
  }
}

[untergeek]

+1 to @metadata as the field name. I think it fits our schema naming quite well, and will not likely collide with anything else.

[untergeek]

And the use case for me is [@metadata][document_id] and [@metadata][action] for determining output behavior for the elasticsearch output plugin, with special regard to replacing rivers.

[torrancew]

👍 to @metadata here as well. In general, I've got my colleagues fairly educated on avoiding field names matching "@*" as a policy when generating custom logs, as the existing reserved fields follow that convention.

[jordansissel]

@torrancew I'm open to possibly allowing logstash to do that policing for you. That is, having Logstash validate any attempts to use @-named fields and warn or otherwise do something for you to validate.

[colinsurprenant]

+1 on the metadata idea and @metadata seems right.

[avleen]

Really like this.
The example is spot on with what I had in mind too.
I'd use this for index name, cluster name, all kinds of stuff.

Grok is another place where this could make a difference. Currently we extract a my_timestamp field like this:
%{TIMESTAMP:my_timestamp}. If grok can take %{TIMESTAMP:[@metadata][my_timestamp]}, it means we wouldn't have to do a remove_field in every date{} filter.

[jordansissel]

@avleen +1, there's some performance and configuration-complexity benefits here for users, I think, because you can use @metadata as a sort of scratch space to store things that are important to the event but do not represent the event itself.