Default Logstash index pattern should be "[logstash-]YYYY.MM.DD", not "logstash-*"

[cjchand]

In the course of troubleshooting poor ES performance for one of our ELK stacks, I ultimately figured out that the default index pattern for Logstash is very sub-optimal.

In the Kibana code, it is set as this in multiple places: logstash-*

Because of this, even if I search just the past 15 minutes all queries search across all indices. This can cause alot of unneeded churn in ES, leading to heap bloat and eventually alot of GC.

If you look at this default in Kibana's UI, then click the "Use event times to create index names" checkbox, the pattern changes to what it really should default to: [logstash-]YYYY.MM.DD.

How I happened to come across this is that ES was puking, citing indices that had nothing to do with the timeframe I was querying. Once I made the change above, performance improved dramatically and resource usage by ES was much lower.

This is basically the same thing as the timestamping setting for Kibana 3 dashboards, which I recall seeing somewhere as a best practice, for the exact same reasons as above.

In short: If you're gonna pre-populate a default to work with Logstash, make it a good default :)

Hope that helps and thanks in advance!

[epixa]

@cjchand Kibana 4.3.0 should address this for you: it automatically optimizes wildcard index patterns such as logstash-* in the same way that you could previously only achieve by manually configuring a time-based index pattern name that matches your underlying indexing scheme (e.g. [logstash-]YYYY.MM.DD).

If you want more details about this, check out the original ticket #4342. There is a comment linked from the main description that goes into more detail about the underlying changes.

Thanks for the in-depth ticket!

[cjchand]

Crud. I tried searching before I opened this, but didn't find any matches.

Sorry about that and thanks for the quick reply!