cmd/bosun: Azure Monitor Datasource #2283

kylebrandt · 2018-07-20T15:36:46Z

No description provided.

kylebrandt · 2018-07-31T12:50:06Z

For review except the metric metadata function which can be ignored for now. Also will vendor later. cc @mhenderson-so . ~~Note: the azure datasource commit by itself has some cache edits by accident and doesn't build, so I will probably just squash all this into one when time ready for master.~~ (fixed in rebase and force push)

kylebrandt · 2018-07-31T14:23:50Z

cmd/bosun/conf/system.go

+	DebugResponse  bool
+}
+
+// Valid returns if the configuration for the AzureMonitor


fix comment

kylebrandt · 2018-07-31T14:25:57Z

cmd/bosun/conf/system.go

+	return allClients
+}
+
+func azureLogRequest() autorest.PrepareDecorator {


Comment this function and the following, add comment to attribute source of readme from the go azure sdk

Are these even required long term?

I don't think they hurt, and since the documentation is marking the experimental for now they could be handy. I would probably be better if more datasources had this I think

kylebrandt · 2018-07-31T14:26:48Z

cmd/bosun/conf/system.go

+			if err != nil {
+				slog.Warningf("failure to dump azure request: %v", err)
+			}
+			dump, _ := httputil.DumpRequestOut(r, true)


check + log err

kylebrandt · 2018-07-31T14:26:56Z

cmd/bosun/conf/system.go

+			if err != nil {
+				slog.Warningf("failure to dump azure request: %v", err)
+			}
+			dump, _ := httputil.DumpResponse(r, true)


check + log err

kylebrandt · 2018-07-31T14:27:37Z

cmd/bosun/expr/azure.go

+	},
+}
+
+// Tag function for the "az" expression function


comment: azTags is

kylebrandt · 2018-07-31T14:27:49Z

cmd/bosun/expr/azure.go

+	return azureTags(args[2])
+}
+
+// Tag function for the "azmulti" expression function


comment: azMultiTags is

kylebrandt · 2018-07-31T14:28:32Z

cmd/bosun/expr/azure.go

+
+// azureTags adds tags for the csv argument along with the "name" and "rsg" tags
+func azureTags(arg parse.Node) (parse.Tags, error) {
+	tags := parse.Tags{"name": struct{}{}, "rsg": struct{}{}}


make name and rsg constants and change in all places where referenced

kylebrandt · 2018-07-31T14:28:57Z

cmd/bosun/expr/azure.go

+
+const azTimeFmt = "2006-01-02T15:04:05"
+
+func azResourceURI(subscription, resourceGrp, Namespace, Resource string) string {


comment func

kylebrandt · 2018-07-31T14:33:14Z

cmd/bosun/expr/azure.go

+	return
+}
+
+// AzureQuery queries an Azure monitor metric for the given resource and returns a series set tagged by


finish this func comment

kylebrandt · 2018-07-31T14:33:51Z

cmd/bosun/expr/azure.go

+	// Verify prefix is a defined resource and fetch the collection of clients
+	cc, clientFound := e.Backends.AzureMonitor[prefix]
+	if !clientFound {
+		return r, fmt.Errorf("azure client with name %v not defined", prefix)


... "%v" ...

kylebrandt · 2018-07-31T14:34:44Z

cmd/bosun/expr/azure.go

+	st := e.now.Add(time.Duration(-sd)).Format(azTimeFmt)
+	en := e.now.Add(time.Duration(-ed)).Format(azTimeFmt)
+
+	// Set Dimensions (tag) keys for metrics that support them by building a filter


expand on comment , link to https://docs.microsoft.com/en-us/rest/api/monitor/filter-syntax

kylebrandt · 2018-07-31T14:35:40Z

cmd/bosun/expr/azure.go

+		tg = azureIntervalToTimegrain(interval)
+	}
+
+	// Set azure aggregation method


Find various "azure" comments and replace with "Azure" (casing)

kylebrandt · 2018-07-31T14:38:22Z

cmd/bosun/expr/azure.go

+	if err != nil {
+		slog.Errorf("failure to parse remaning reads from azure response")
+	} else {
+		collect.Sample("azure.remaining_reads", opentsdb.TagSet{"prefix": prefix}, float64(readsRemaining))


add comment explaining why this is sampled

kylebrandt · 2018-07-31T14:39:14Z

cmd/bosun/expr/expr.go

+// collectCache is a helper function for collecting metrics on
+// the expression cache
+func collectCacheHit(cacheName, qType string, hit bool) {
+	tags := opentsdb.TagSet{"query_type": qType, "name": cacheName}


add metadata for these metrics

kylebrandt · 2018-07-31T14:40:32Z

cmd/bosun/expr/azure.go

+				}
+				series := make(Series)
+				tags := make(opentsdb.TagSet)
+				tags["rsg"] = rsg


ref constant here

kylebrandt · 2018-07-31T14:41:11Z

cmd/bosun/expr/azure.go

+					}
+				}
+				for _, mValue := range *dataContainer.Data {
+					exValue := azureExtractMetricValue(&mValue, aggLong)


add comment

kylebrandt · 2018-07-31T14:42:22Z

cmd/bosun/expr/azure.go

+	for res := range resCh {
+		queryResults = append(queryResults, res)
+	}
+	// Merge the query results into a single seriesSet


skip merge if length of set is not gt than 1

kylebrandt · 2018-07-31T14:44:02Z

cmd/bosun/expr/azure.go

+// or tags associated with that resource
+func AzureFilterResources(e *State, T miniprofiler.Timer, resources AzureResources, filter string) (r *Results, err error) {
+	r = new(Results)
+	bqf, err := boolq.Parse(filter)


add comment

kylebrandt · 2018-07-31T14:45:03Z

cmd/bosun/expr/azure.go

+	if len(sp) != 2 {
+		return false, fmt.Errorf("bad filter, filter must be in k:v format, got %v", filter)
+	}
+	key := strings.ToLower(sp[0]) // Make key case insensitive


note case insensitivity in expression docs

kylebrandt · 2018-07-31T14:45:33Z

cmd/bosun/expr/azure.go

+	key := strings.ToLower(sp[0]) // Make key case insensitive
+	value := sp[1]
+	switch key {
+	case "name":


consts again for name, rsg

kylebrandt · 2018-07-31T14:46:06Z

cmd/bosun/expr/azure.go

+		if re.MatchString(ar.Name) {
+			return true, nil
+		}
+	case "rsg", "resourcegroup":


remove "resourcegroup" alias, no real point, update expr docs to reflect

kylebrandt · 2018-07-31T14:46:43Z

cmd/bosun/expr/azure.go

+	case string(insights.Maximum):
+		v = mv.Maximum
+	case string(insights.Total):
+		v = mv.Total


add missing None aggregation

nm, None isn't a field in the sdk

kylebrandt · 2018-07-31T14:47:05Z

cmd/bosun/expr/azure.go

+		return string(insights.Maximum), nil
+	case "total":
+		return string(insights.Total), nil
+	case "count":


add missing none aggregation here as well

captncraig

Seems pretty good to me. Just finish out todos and comments.

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/conf/system.go

@@ -84,6 +94,7 @@ func (sc *SystemConf) EnabledBackends() EnabledBackends {
 	b.Influx = sc.InfluxConf.URL != ""
 	b.Elastic = len(sc.ElasticConf["default"].Hosts) != 0
 	b.Annotate = len(sc.AnnotateConf.Hosts) != 0
+	b.AzureMonitor = sc.AzureMonitorConf["default"].ClientId != ""


This feels like a weak thing to check. Are they required to have a "default"? Is ClientID required? (assume thats a credential of some kind?)

Agreed will make something better, this is left from my first pass

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/conf/system.go

+		clients.MetricsClient = insights.NewMetricsClient(conf.SubscriptionId)
+		clients.MetricDefinitionsClient = insights.NewMetricDefinitionsClient(conf.SubscriptionId)
+		clients.ResourcesClient = resources.NewClient(conf.SubscriptionId)
+		if conf.DebugRequest {


Do we really need these debug things long term? I'd probably prefer not having a million different config options if we can avoid it.

Just two options. Could help with people debugging issues, in particular since I saw what I think is a bug in the SDK about error being missed.

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/conf/system.go

+		if err != nil {
+			// Should not hit this since we check for authorizer errors in Validation
+			// This is checked before because this method is not called until the an expression is called
+			slog.Fatal("Azure conf: ", err)


Fatal feels wrong here. Does this method get invoked every time an expression runs? If azure is down intermittently, could this crash bosun?

I don't think this actually make any API calls and is just a validity thing. So it should be caught by Valid() method, but added this here in case I'm wrong.

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/conf/system.go

+	return allClients
+}
+
+func azureLogRequest() autorest.PrepareDecorator {


Are these even required long term?

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/expr/azure.go

+		F:             AzureMultiQuery,
+		PrefixEnabled: true,
+	},
+	"azmd": { // TODO Finish and document this func


captncraig · 2018-07-31T20:11:47Z

cmd/bosun/expr/azure.go

+		F:             AzureResourcesByType,
+		PrefixEnabled: true,
+	},
+	"azrf": {


azrf or azfr? Inner function is "FilterResources", so maybe azfr matches better? I kinda hate short names.

I think like azrf because it shares the azr prefix with azrt

captncraig · 2018-07-31T20:11:47Z

cmd/bosun/expr/azure.go

+}
+
+// AzureMetricDefinitions fetches metric information for a specific resource and metric tuple
+// TODO make this return and not fmt.Printf


captncraig · 2018-07-31T20:11:47Z

cmd/bosun/expr/azure.go

+			return
+		}
+	}
+	st := e.now.Add(time.Duration(-sd)).Format(azTimeFmt)


"startTime" and "endTime" would not be unreasonable names.

When this is done I want to go back and make a method on the State that does since we use it all the time.

this includes an incomplete azure metadata function with TODOs to be completed in a future commit.

so metrics can be collected

…ommit)

kylebrandt force-pushed the ads branch 2 times, most recently from e22f47c to 1fc3f64 Jul 30, 2018

kylebrandt requested a review from captncraig Jul 31, 2018

kylebrandt changed the title ~~cmd/bosun: (WIP) Azure Monitor Datasource~~ cmd/bosun: Azure Monitor Datasource Jul 31, 2018

kylebrandt force-pushed the ads branch from 1fc3f64 to d07cf32 Jul 31, 2018