Add azure module/monitor metricset #13196

narph · 2019-08-08T11:07:43Z

Azure Monitor collects and aggregates logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting.

The metrics are numerical values that describe some aspect of a system at a particular point in time. They are collected at regular intervals and are identified with a timestamp, a name, a value, and one or more defining labels. Metrics can be aggregated using a variety of algorithms, compared to other metrics, and analyzed for trends over time .

We can access the azure monitor metrics:

from a command line using PowerShell cmdlets
from custom application using REST API.
from a command line using CLI.
if metrics are forwarded to an event hub then a custom solution to read data from the event hub

PR is in progress and subject to changes, initial approach using the rest API to retrieve metric definitions and metrics, currently using the azure sdk for go.

Current azure module/monitor metricset configuration:

- module: azure
  metricsets:
  - monitor
  enabled: true
  period: 60s
  client_id: ''
  client_secret: ''
  tenant_id: ''
  subscription_id: ''
  refresh_list_interval: 120s
  resources:
  - resource_id: "subscriptions/xxxxx/resourceGroups/obs-infrastructure/providers/Microsoft.ApiManagement/service/obsapimanagement"
    metrics:
    - name: "Requests"
      namespace: "Microsoft.ApiManagement/service"
      aggregations: ["Maximum"]
      dimensions:
      - name: "Hostname"
        value: "obsapimanagement.azure-api.net"
    - name: ["Capacity", "Requests"]
      namespace: "Microsoft.ApiManagement/service"
      aggregations: ["Average"]
      dimensions:
      - name: "Location"
        value: "West Europe"
  - resource_group: "obs-infrastructure"
    resource_type: "Microsoft.Compute/virtualMachines"
    metrics:
    - name: "Percentage CPU"
      namespace: "Microsoft.Compute/virtualMachines"
      aggregations: ["Average", "Minimum", "Maximum"]
    - name: ["Disk Read Bytes", "Data Disk Write Operations/Sec", "Network Out"]
      namespace: "Microsoft.Compute/virtualMachines"
      aggregations: ["Average", "Maximum"]
  - resource_query: "resourceType eq 'Microsoft.Storage/storageAccounts' and resourceGroup eq 'obs-infrastructure'"
    metrics:
    - name: ["*"]
      namespace: "Microsoft.Storage/storageAccounts"
      aggregations:  ["Total", "Average"]

Example output:

        "_index" : "metricbeat-8.0.0-2019.08.07-000001",
        "_type" : "_doc",
        "_id" : "Tj85cGwBFNllmdJdQHXi",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2019-08-08T07:54:50.905Z",
          "azure" : {
            "monitor" : {
              "resource" : {
                "name" : "obsinfrastructurediag",
                "location" : "westeurope",
                "type" : "Microsoft.Storage/storageAccounts"
              },
              "namespace" : "Microsoft.Storage/storageAccounts",
              "subscriptionId" : "xxxxxx",
              "metrics" : {
                "successe2elatency.avg" : 34.54545454545455,
                "availability.avg" : 100,
                "transactions.avg" : 1,
                "ingress.avg" : 35056.5,
                "egress.avg" : 8167.5,
                "successserverlatency.avg" : 22.181818181818183
              }
            }
          },
          "event" : {
            "module" : "azure",
            "duration" : 3039980500,
            "dataset" : "azure.monitor"
          },
          "metricset" : {
            "name" : "monitor"
          },
          "service" : {
            "type" : "azure"
          },
          "ecs" : {
            "version" : "1.0.1"
          .....
        },

STILL TO DO

Clean up/improve code
Implement validation on resources, metric names/ aggregations configured
Create tests for the client and helper methods
Move location value to cloud.region

x-pack/metricbeat/module/azure/monitor/client.go

x-pack/metricbeat/module/azure/azure.go

x-pack/metricbeat/module/azure/monitor/client_test.go

x-pack/metricbeat/module/azure/monitor/azure_service_interface.go

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

ruflin

Skimmed through the code and left some comments.

One thing I'm especially curious about is how exactly the calculation and gathering of metrics happens. What happens if there are 2 metrics in the queried timeframe? What will the API return? If someone selects 1h for the selection and we get a metric, do we know when it happened or is it calculated on demand? We had quite a few discussions around this also for the AWS side. Maybe @kaiyan-sheng would be good to have a look at this PR.

For some of the long methods it would be great to get a bit more details in the method description on what it does which would make reviewing easier.

One part I didn't fully get yet: Will azure.monitors contain logs and metrics? Could you provide an example on how a log even looks like?

metricbeat/docs/modules_list.asciidoc

vendor/vendor.json

x-pack/metricbeat/module/azure/monitor/_meta/data.json

x-pack/metricbeat/module/azure/monitor/_meta/fields.yml

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

x-pack/metricbeat/module/azure/monitor/data.go

x-pack/metricbeat/module/azure/monitor/monitor.go

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

vendor/vendor.json

kaiyan-sheng

Not sure if https://github.com/Azure-Samples/azure-sdk-for-go-samples/blob/master/insights/insights.go will be helpful. I see ListMetricDefinitions and GetMetricsData here in the code might be handy.

x-pack/metricbeat/metricbeat.reference.yml

kaiyan-sheng · 2019-08-20T03:07:39Z

x-pack/metricbeat/module/azure/azure.go

+	ClientSecret        string           `config:"client_secret" validate:"required"`
+	TenantID            string           `config:"tenant_id" validate:"required"`
+	SubscriptionID      string           `config:"subscription_id" validate:"required"`
+	Period              time.Duration    `config:"period"`


Maybe add validate:"nonzero,required" here? Same for other required params.

x-pack/metricbeat/modules.d/azure.yml

x-pack/metricbeat/module/azure/monitor/monitor.go

kaiyan-sheng · 2019-08-20T04:23:48Z

Please ignore this comment if I missed it, I think you also need to add TestFetch and TestData functions: https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch_integration_test.go

jsoriano

Great job here! I have added some extra comments.

And one more comment I didn't know how to add: metricbeat/modules.d/system.yml shouldn't be renamed.

vendor/vendor.json

jsoriano · 2019-08-21T16:42:57Z

x-pack/metricbeat/module/azure/monitor/_meta/fields.yml

+      type: keyword
+      description: >
+        The subscription ID
+    - name: metrics.*.*


Cloudwatch metrics are stored in aws.metrics.*.*, should we have Azure metrics in azure.metrics.*.*?

I thought they are inside cloudwatch https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/aws/cloudwatch/_meta/fields.yml. Correct me if I am wrong.

It will be prefixed by aws.cloudwatch.*. fields.yml fields are concatenated together. Best double check in the resulting template to confirm.

I thought they are inside cloudwatch https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/aws/cloudwatch/_meta/fields.yml. Correct me if I am wrong.

Umm, in https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/aws/cloudwatch/_meta/data.json they are in aws.metrics.*, @kaiyan-sheng could you confirm?

@jsoriano Sorry I missed this comment. cloudwatch metricset stores data in aws.metrics.* now. Because with using light module to create other metricsets, we don't want aws.cloudwatch being the prefix for other services, such as elb or ebs.

Thanks, I think we should use the same pattern here for consistency. @narph what do you think? It'd mean to store azure metrics in azure.metrics.* instead of azure.monitor.metrics.*.
I think that other common fields like the namespace or the subscription ID could be placed also at the module level.

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

jsoriano · 2019-08-21T18:46:37Z

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

+	}
+	if query != "" {
+		resourceQuery = query
+	}


If query is set, group, rtype and ID will be ignored. And also if group is set, ID will be ignored. This can be counterintuitive, should we check for these cases and return errors?

Or maybe better we should check for these cases implementing a Validate() method for ResourceConfig.

@narph should we explicitly validate that a valid combination of options is provided (e.g. if ID and query are provided, return an error)?

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

x-pack/metricbeat/metricbeat.reference.yml

x-pack/metricbeat/module/azure/azure.go

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go

houndci-bot · 2019-09-18T16:16:16Z

x-pack/metricbeat/module/azure/compute_vm/compute_vm_test.go

+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package compute_vm


don't use an underscore in package name

houndci-bot · 2019-09-18T16:16:17Z

x-pack/metricbeat/module/azure/compute_vm/client_helper_test.go

+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package compute_vm


don't use an underscore in package name

x-pack/metricbeat/module/azure/client_utils.go

houndci-bot · 2019-09-18T16:16:20Z

x-pack/metricbeat/module/azure/compute_vm_scaleset/compute_vm_scaleset_test.go

+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package compute_vm_scaleset


don't use an underscore in package name

houndci-bot · 2019-09-18T16:16:20Z

x-pack/metricbeat/module/azure/compute_vm_scaleset/compute_vm_scaleset_test.go

+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package compute_vm_scaleset


don't use an underscore in package name

jsoriano

@narph huge effort here, let's see if we can merge it soon.

I have added some additional comments, but nothing specially serious. I think that we should merge it soon because it is starting to be quite big and difficult to review. Further fixes and refactors can be addressed in future smaller PRs.

Also as all the code is located in a new specific module there shouldn't be any risk of regressions, and it is beta, what would help to set expectations.

There is only one thing that I think we should clarify before merging this. I think that field paths should be coherent with the cloudwatch metricset. In my opinion metrics should be under azure.metrics. And other fields coming from information that is not specific of the monitor service should be placed at the module level.

We should also generate proper data.json files, but we need to review docs in any case, so this can be left for a future change.

jsoriano · 2019-09-19T10:42:51Z

metricbeat/docs/modules/azure.asciidoc

+=== Example configuration
+
+The azure module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:


More than a big example it may be good to have smaller examples with more explanations in the docs. We may revisit docs in follow-ups in any case.

jsoriano · 2019-09-19T10:53:00Z

x-pack/metricbeat/module/azure/client.go

+	"time"
+
+	"github.com/Azure/azure-sdk-for-go/services/preview/monitor/mgmt/2019-06-01/insights"
+


Nit. Empty line.

Suggested change

I usually make the changes but every time I run mage fmt update I get a different order, any other commands that should be ran here?

jsoriano · 2019-09-19T10:54:38Z

x-pack/metricbeat/module/azure/client.go

+	Config              Config
+	Resources           ResourceConfiguration
+	Log                 *logp.Logger
+}


For a follow up, move common code to configure and initialize clients to some common place so it can be also used in other beats.

x-pack/metricbeat/module/azure/client.go

x-pack/metricbeat/module/azure/compute_vm_scaleset/client_helper_test.go

x-pack/metricbeat/module/azure/data.go

jsoriano · 2019-09-19T15:38:54Z

x-pack/metricbeat/module/azure/monitor/_meta/fields.yml

+      type: keyword
+      description: >
+        The subscription ID
+    - name: metrics.*.*


Thanks, I think we should use the same pattern here for consistency. @narph what do you think? It'd mean to store azure metrics in azure.metrics.* instead of azure.monitor.metrics.*.
I think that other common fields like the namespace or the subscription ID could be placed also at the module level.

x-pack/metricbeat/module/azure/monitor/client_helper.go

jsoriano · 2019-09-19T15:44:29Z

x-pack/metricbeat/module/azure/monitor/monitor.go

+func (m *MetricSet) Fetch(report mb.ReporterV2) error {
+	err := m.client.InitResources(report)
+	if err != nil {
+		return err


@narph please review these return err to see if more context can be provided with errors.Wrap.

jsoriano

I would say to merge this as is, and continue with refactors, documentation and dashboards in follow ups.

Great job here!

narph · 2019-09-23T07:27:53Z

jenkins test this

narph · 2019-09-23T09:57:37Z

jenkins test this

narph added 8 commits May 22, 2019 10:57

Progress commit

d5bbbc4

Progress on azure module

52f709e

Staging

774b609

Progres on POC

cbf3f77

Work on client

bb47fd2

Update event format, implement refresh interval

d5936c3

Merge branch 'master' into azure-module

c2e79b7

Work on importing dependencies

6dc4433

houndci-bot reviewed Aug 8, 2019

View reviewed changes

narph self-assigned this Aug 8, 2019

narph added the Team:Integrations Label for the Integrations team label Aug 8, 2019

Fix houndbot notifications

6a081f7

andresrc added the [zube]: In Progress label Aug 12, 2019

Start work on unittests

b6514c5

houndci-bot reviewed Aug 14, 2019

View reviewed changes

narph added 2 commits August 14, 2019 15:01

Work on unittests

45dde4d

Work on unittests

b257446

ruflin reviewed Aug 15, 2019

View reviewed changes

Adding unittests

9fdd3dd

kaiyan-sheng reviewed Aug 20, 2019

View reviewed changes

vendor/vendor.json Show resolved Hide resolved

kaiyan-sheng reviewed Aug 20, 2019

View reviewed changes

narph added 4 commits August 20, 2019 16:05

Fixing issues

5b686b0

Generate fields

1b32171

Small changes

2bed2a2

Merge branch 'master' into azure-module

69bdb22

jsoriano reviewed Aug 21, 2019

View reviewed changes

narph added 2 commits August 23, 2019 09:40

Work on removing double records

c6b8e23

Work on error handling and regenrate files

ddd3b27

houndci-bot reviewed Aug 28, 2019

View reviewed changes

x-pack/metricbeat/module/azure/monitor/azure_monitor_service.go Outdated Show resolved Hide resolved

houndci-bot reviewed Sep 18, 2019

View reviewed changes

run fmt update

0da8ac7

narph marked this pull request as ready for review September 18, 2019 17:28

narph requested a review from a team as a code owner September 18, 2019 17:28

narph added [zube]: In Review and removed [zube]: In Progress labels Sep 18, 2019

narph added 3 commits September 18, 2019 20:11

Merge branch 'master' into azure-module

e746103

Generate notice

6a075e3

Work on monitor service

a17d73e

narph requested a review from jsoriano September 19, 2019 08:08

narph added 2 commits September 19, 2019 14:26

Work on failing test

d001e84

Wotk on test

e1f19f2

jsoriano reviewed Sep 19, 2019

View reviewed changes

narph added 2 commits September 20, 2019 13:44

Work on feedback

bd47d97

Work on event format

beac3e6

jsoriano approved these changes Sep 20, 2019

View reviewed changes

narph merged commit 2f5b816 into elastic:master Sep 23, 2019

zube bot added [zube]: Done and removed [zube]: In Review labels Sep 23, 2019

narph deleted the azure-module branch September 23, 2019 14:45

narph added [zube]: (7.5) Planned for release v7.5.0 labels Sep 24, 2019

andresrc removed [zube]: (7.5) Planned for release [zube]: Done labels Sep 24, 2019

jsoriano mentioned this pull request Oct 3, 2019

Small improvements on the azure module #13859

Merged

narph added the Metricbeat Metricbeat label Oct 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add azure module/monitor metricset #13196

Add azure module/monitor metricset #13196

narph commented Aug 8, 2019 •

edited by sorantis

Loading

ruflin left a comment

kaiyan-sheng left a comment

kaiyan-sheng Aug 20, 2019

kaiyan-sheng commented Aug 20, 2019

jsoriano left a comment

jsoriano Aug 21, 2019

narph Aug 28, 2019 •

edited

Loading

ruflin Sep 2, 2019

jsoriano Sep 2, 2019

kaiyan-sheng Sep 19, 2019

jsoriano Sep 19, 2019

jsoriano Aug 21, 2019

jsoriano Sep 2, 2019

houndci-bot Sep 18, 2019

houndci-bot Sep 18, 2019

houndci-bot Sep 18, 2019

houndci-bot Sep 18, 2019

jsoriano left a comment

jsoriano Sep 19, 2019

jsoriano Sep 19, 2019

narph Sep 20, 2019

jsoriano Sep 19, 2019

jsoriano Sep 19, 2019

jsoriano Sep 19, 2019

jsoriano left a comment

narph commented Sep 23, 2019

narph commented Sep 23, 2019

		"time"

		"github.com/Azure/azure-sdk-for-go/services/preview/monitor/mgmt/2019-06-01/insights"

Add azure module/monitor metricset #13196

Add azure module/monitor metricset #13196

Conversation

narph commented Aug 8, 2019 • edited by sorantis Loading

ruflin left a comment

Choose a reason for hiding this comment

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaiyan-sheng commented Aug 20, 2019

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narph Aug 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

narph commented Sep 23, 2019

narph commented Sep 23, 2019

narph commented Aug 8, 2019 •

edited by sorantis

Loading

narph Aug 28, 2019 •

edited

Loading