String expansion on document_id setting

[ggrossetie]

I'm using string expansion (i.e. %{foo}) on document_id setting:

elasticsearch {
  document_id => "%{foo}"
}

The current behavior is:

• if the event has foo == "bar" then this string: "%{foo}" will return "bar"
• if a %{foo} value does not exist, then no substitution occurs.

As defined in https://github.com/elastic/logstash/blob/master/lib/logstash/event.rb#L224

I think it's safer to let ElasticSearch generates a document id rather than index a document with id %{foo}. One major issue is that ElasticSearch will overwrite existing entries with the same ID %{foo} 😞

IMHO the default behavior for document_id should be:

• if the event has foo == "bar" then this string: "%{foo}" will return "bar"
• if a %{foo} value does not exist, then return nil (and let ElasticSearch generates a document id)

(or at least make this behavior configurable)

Thanks

[KlavsKlavsen]

I just hit this exact issue as well :(

[KlavsKlavsen]

since I can't guarantee all documents has %{foo} - it means I can't use document_id setting at all :(

[ggrossetie]

@KlavsKlavsen you can but the configuration is really verbose:

if [log-id] {
  elasticsearch {
    protocol => "transport"
    cluster => "cluster"
    host => "localhost"
    document_id => "%{log-id}"
  }
} else {
  elasticsearch {
    protocol => "transport"
    cluster => "cluster"
    host => "localhost"
  }
}

[KlavsKlavsen]

ahh.. ofcourse.. thank you for the tip..

I still think document_id setting should handle this more nicely though.. but a fine workaround it is.

[KlavsKlavsen]

could one use the metadata feature to make this "prettier"?

I'm not sure if the "nil" part of it will work though.. ?

filter {
if [message_id] {
mutate { add_field => { "[@metadata][id]" => "%{message_id}" } }
} else {
mutate { add_field => { "[@metadata][id]" => nil } }
}
}

output {
elasticsearch {
manage_template => false
protocol => http
port => 9200
document_id => [@metadata][id]
}
}

[KlavsKlavsen]

I got this working.. it was document_id => %{[@metadata][id]}
I don't know if nil works.. I choose to generate message_id using ruby section, doing md5 hash on message field if the field is not there.

[jsvd]

this is a limitation of how the event sprintf works, we can't detect in the output since the template maybe have mixed interpolations and real text like "my-id-%{id}-and-type-%{type}" and it becomes expensive checking every interpolated string for the presence of %{ }

As suggested above, data should be sanitized before reaching the output, so I'm closing this for now.
Please open a new issue if you see a related problem that is specific to the es output, otherwise an issue can be created in github.com/elastic/logstash