Elastic v2 support and elastic expr refactor #1561

Merged
merged 1 commit into from Jan 20, 2016

Projects

None yet

5 participants

@kylebrandt
Member

No description provided.

@kylebrandt kylebrandt changed the title from WIP: Estype to WIP: Elastic v3 support and elastic expr refactor Jan 15, 2016
@kylebrandt kylebrandt changed the title from WIP: Elastic v3 support and elastic expr refactor to Elastic v2 support and elastic expr refactor Jan 19, 2016
@Dieterbe Dieterbe and 1 other commented on an outdated diff Jan 19, 2016
docs/expressions.md
@@ -176,6 +178,68 @@ lstat returns various summary stats per bucket for the specified `field`. The fi
* As of January 15, 2015 - logstash functionality is new so these functions may change a fair amount based on experience using them in alerts.
* Alerts using this information likely want to set ignoreUnknown, since only "groups" that appear in the time frame are in the results.
+## Elastic Query Functions
+
+Elasitc replaces the deprecated logstash (ls) functions. It only works with Elastic v2+. It introduces two new types to allow for greater flexibility in querying. The ESIndexer type generates indexes to query (based on the date range). There are now different functions to generate indexers for people with different configurations. The ESQuery type is generates elastic queries so you can filter your results. By making these new types, new Indexers and Elastic queries can be added over time.
+
@Dieterbe
Dieterbe Jan 19, 2016 Contributor

generate indexes? do you mean generate index names [for which indexes to be queried] ?

@kylebrandt
kylebrandt Jan 19, 2016 Member

Yes, index names, will update. Thanks!

@Dieterbe Dieterbe and 1 other commented on an outdated diff Jan 19, 2016
docs/expressions.md
@@ -176,6 +178,68 @@ lstat returns various summary stats per bucket for the specified `field`. The fi
* As of January 15, 2015 - logstash functionality is new so these functions may change a fair amount based on experience using them in alerts.
* Alerts using this information likely want to set ignoreUnknown, since only "groups" that appear in the time frame are in the results.
+## Elastic Query Functions
+
+Elasitc replaces the deprecated logstash (ls) functions. It only works with Elastic v2+. It introduces two new types to allow for greater flexibility in querying. The ESIndexer type generates indexes to query (based on the date range). There are now different functions to generate indexers for people with different configurations. The ESQuery type is generates elastic queries so you can filter your results. By making these new types, new Indexers and Elastic queries can be added over time.
@Dieterbe
Dieterbe Jan 19, 2016 Contributor

so now we can also use these ES functions for any ES querying, without being tied to particular logstash schema? can this be clarified? are there any known limitations that we may want to document?

@kylebrandt
kylebrandt Jan 19, 2016 Member

@Dieterbe Right, this meant to make it so we don't tie to a particular schema. There are elastic queries beyond what I created functions for, but if people request them they can just be new funcs that return ESQuery. Same goes for the index naming schema.

Regexes might not all make it through the parser - there is an open issue for that. But I think that would be fixed independently of this.

One limitation I will add is that if your fields is analyzed, you get multiple histograms based on the terms. But turning an analyzed field into a tag key doesn't really make sense in any scenario I can think of.

@kylebrandt
kylebrandt Jan 19, 2016 Member

@Dieterbe Ah you made me think of one:

elastic.NewDateHistogramAggregation().Field("@timestamp").

@timestamp is hardcoded, maybe I will make that an argument for the indexer...

@Dieterbe
Contributor

Nice work Kyle! Thanks.
Looking forward to play with this at some point.

@deanefrati

this looks great @kylebrandt! Can't wait to try it (hopefully this week). Quick question are you planning to handle the scenario mentioned in #1160 as part of this work?

@kylebrandt
Member

@deanefrati I looked at this and didn't want to do it in this PR, but will make sure it is on my radar for soon. I think that issue is impacting us at stack well to make elastic usable for alerts. I just had enough to wrap my head around with elastic's API changing, the elastic go library decided that was a good time to do some refactoring since elastic changed, and then I decided it was a good time to do it in bosun too. That is about as much as I could hold in my head :-P

@deanefrati

makes total sense. thanks again

@captncraig captncraig and 1 other commented on an outdated diff Jan 20, 2016
cmd/bosun/conf/conf.go
@@ -67,6 +67,7 @@ type Conf struct {
GraphiteHost string // Graphite query host: foo.bar.baz
GraphiteHeaders []string // extra http headers when querying graphite.
LogstashElasticHosts expr.LogstashElasticHosts // CSV Elastic Hosts (All part of the same cluster) that stores logstash documents, i.e http://ny-elastic01:9200
+ ElasticHosts expr.ElasticHosts // CSV Elastic Hosts (All part of the same cluster), i.e http://ny-elastic01:9200
@captncraig
captncraig Jan 20, 2016 Contributor

I would like this to be more clear which is for v1 and which is for v2

@kylebrandt
kylebrandt Jan 20, 2016 Member

@captncraig updated comments and the documentation.

@captncraig captncraig and 1 other commented on an outdated diff Jan 20, 2016
cmd/bosun/expr/elastic.go
+ Return: parse.TypeSeriesSet,
+ Tags: elasticTagQuery,
+ F: ESStat,
+ },
+
+ // Funcs to create elastic index names (ESIndexer type)
+ "esindices": {
+ Args: []parse.FuncType{parse.TypeString, parse.TypeString},
+ VArgs: true,
+ VArgsPos: 1,
+ Return: parse.TypeESIndexer,
+ F: ESIndicies,
+ },
+ "esdaily": {
+ Args: []parse.FuncType{parse.TypeString, parse.TypeString, parse.TypeString},
+ VArgsPos: 1,
@captncraig
captncraig Jan 20, 2016 Contributor

VArgs: false and VargsPos: 1? I'm not sure what that combination means.

@kylebrandt
kylebrandt Jan 20, 2016 Member

Is a mistake

@kylebrandt kylebrandt merged commit c94cd3f into master Jan 20, 2016

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@awesensepaul

should be lowercase first, elasticHosts

@captncraig captncraig deleted the estype branch Feb 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment