Elasticsearch doesn't care about timezone and creates indexes with UTC #7375

Closed
3h4x opened this Issue Aug 21, 2014 · 7 comments

Projects

None yet

4 participants

@3h4x
3h4x commented Aug 21, 2014

This could benefit all people using ELK.

Our logs are rotated with daily fashion and this should match elasticsearch index. But when we do it now we got holes and duplicates :( Deleting index and just feeding logstash with data should be fine and dandy but there is mismatch timezone now.

There could be configuration option to set default timezone and the indexes would follow the log rotation.

All the best!

@clintongormley
Member

Sorry @3h4x but I don't understand what you mean. Could you explain in more detail please?

@3h4x
3h4x commented Aug 22, 2014

No worries.
When I delete index from elasticsearch I got hole in Kibana from 2:00 -> 2:00 not midnight -> midnight. We got different timezone than UTC.

Every midnight logrotate save new file on servers. When we want for whatever reason to delete elasticsearch index, recreate index and forward logs one more time then I need to grep like crazy because logrotated log isn't fitting the index.

@clintongormley
Member

OK - more details still :)

Are you talking about index names? the @timestamp field? what? Any reason why you don't convert your timestamps (or index names or whatever) to UTC in logstash?

@3h4x
3h4x commented Aug 22, 2014

Okie dokie.
Everyday our elasticsearch creates index ( logstash-2014.08.22 ). I would like it to create daily index with regard to timezone so when I delete one index I will see empty one day in Kibana. Now because we are not in UTC i see empty day but from 2:00 to 2:00 next day.
I don't wanna manipulate @timestamp field. UTC is completely fine there.
I would like to see a configuration option in elasticsearch like "timezone offset index". So index starts at midnight in our timezone and ends at midnight.

Yes there is a reason why I don't manipulate @timestamp. It takes resources grok and mutate. I want to keep it simple. Just timezone offset in index creation would be lovely, is there any option like it or is it very unlikely to happen?

@bleskes
Member
bleskes commented Aug 25, 2014

@3h4x To ES, the index name is arbitrary. It's logstash that decides how to name the indices and it does so in UTC. It seems the logstash team has already discussed this and has voted not to implement. Perhaps you can share your use case there and see whether it helps getting this implemented? https://logstash.jira.com/browse/LOGSTASH-973

Since this is not really an ES issue, I'll close it for now.

@bleskes bleskes closed this Aug 25, 2014
@3h4x
3h4x commented Aug 25, 2014

@bleskes thanks for your input. I'll try to lobby it there ✌️

@nean-and-i

But I guess it is something different if you need this functionality for dynamic filenames when using csv output module.
I'd love to use dynamic filenames in the csv output module to get proper date values that matches with the records in the csv.
If there is an intention that logstash should or can be used for other purposes as for ES than it would be great to support features like that and outside the ES buble.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment