This could benefit all people using ELK.
Our logs are rotated with daily fashion and this should match elasticsearch index. But when we do it now we got holes and duplicates :( Deleting index and just feeding logstash with data should be fine and dandy but there is mismatch timezone now.
There could be configuration option to set default timezone and the indexes would follow the log rotation.
All the best!
Sorry @3h4x but I don't understand what you mean. Could you explain in more detail please?
When I delete index from elasticsearch I got hole in Kibana from 2:00 -> 2:00 not midnight -> midnight. We got different timezone than UTC.
Every midnight logrotate save new file on servers. When we want for whatever reason to delete elasticsearch index, recreate index and forward logs one more time then I need to grep like crazy because logrotated log isn't fitting the index.
OK - more details still :)
Are you talking about index names? the @timestamp field? what? Any reason why you don't convert your timestamps (or index names or whatever) to UTC in logstash?
Everyday our elasticsearch creates index ( logstash-2014.08.22 ). I would like it to create daily index with regard to timezone so when I delete one index I will see empty one day in Kibana. Now because we are not in UTC i see empty day but from 2:00 to 2:00 next day.
I don't wanna manipulate @timestamp field. UTC is completely fine there.
I would like to see a configuration option in elasticsearch like "timezone offset index". So index starts at midnight in our timezone and ends at midnight.
Yes there is a reason why I don't manipulate @timestamp. It takes resources grok and mutate. I want to keep it simple. Just timezone offset in index creation would be lovely, is there any option like it or is it very unlikely to happen?
@3h4x To ES, the index name is arbitrary. It's logstash that decides how to name the indices and it does so in UTC. It seems the logstash team has already discussed this and has voted not to implement. Perhaps you can share your use case there and see whether it helps getting this implemented? https://logstash.jira.com/browse/LOGSTASH-973
Since this is not really an ES issue, I'll close it for now.
@bleskes thanks for your input. I'll try to lobby it there ✌️
But I guess it is something different if you need this functionality for dynamic filenames when using csv output module.
I'd love to use dynamic filenames in the csv output module to get proper date values that matches with the records in the csv.
If there is an intention that logstash should or can be used for other purposes as for ES than it would be great to support features like that and outside the ES buble.