Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add azure storage metricset #15342

Merged
merged 18 commits into from Jan 13, 2020
Merged

Add azure storage metricset #15342

merged 18 commits into from Jan 13, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented Jan 6, 2020 •
edited

Should handle:
#14548
#14549
#14551
#14552

Configuration:

- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_group: [""]
    service_type: ["blob", "file"]
  - resource_id: [""]

Output:

 {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "JRT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "version" : "8.0.0"
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "name" : "DESKTOP-RFOOE09"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9280652400
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts",
            "storage" : {
              "successe2elatency" : {
                "avg" : 26.516666666666666
              },
              "availability" : {
                "avg" : 100
              },
              "transactions" : {
                "total" : 310
              },
              "ingress" : {
                "total" : 7723808.0
              },
              "egress" : {
                "total" : 1473483.0
              },
              "successserverlatency" : {
                "avg" : 16.883333333333333
              }
            },
            "resource" : {
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure",
              "name" : "obsdiaglinux"
            }
          }
        },
        "sort" : [
          1578332520000
        ]
      },
      {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "KBT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "version" : "8.0.0",
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9281658700
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts/tableServices",
            "storage" : {
                "successserverlatency" : {
                  "avg" : 115.76666666666667
                },
                "successe2elatency" : {
                  "avg" : 118.63333333333334
                },
                "availability" : {
                  "avg" : 100
                },
                "transactions" : {
                  "total" : 30
                },
                "ingress" : {
                  "total" : 2248122.0
                },
                "egress" : {
                  "total" : 1209276.0
              }
            },
            "resource" : {
              "name" : "obsdiagwin",
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure"
            }
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "name" : "DESKTOP-RFOOE09",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "os" : {
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          }
        },
        "sort" : [
          1578332520000
        ]
      },

PR also contains:

  • fix on generated fields and modified dashboard
  • change on the way metric values are retrieved and events are sent. The events are sent now after all metric values are retrieved for each resource. In the past, events were sent only after all metric value calls are returned for all resources in the config (this took some time if the number of resources is larger)

@narph narph requested a review from a team as a code owner January 6, 2020 18:43
@narph narph self-assigned this Jan 6, 2020
@narph narph added Metricbeat Metricbeat Team:Integrations Label for the Integrations team [zube]: In Progress labels Jan 6, 2020
ChrsMark
ChrsMark reviewed Jan 13, 2020
View reviewed changes
x-pack/metricbeat/module/azure/client.go
// select period to collect metrics, will double the interval value in order to retrieve any missing values
//if timegrain is larger than intervalx2 then interval will be assigned the timegrain value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion here: wouldn't it be better to be calculated in a different helping method? :)

@narph narph requested a review from ChrsMark January 13, 2020 11:50
@narph narph added needs_backport PR is waiting to be backported to other branches. v7.6.0 labels Jan 13, 2020
kaiyan-sheng
kaiyan-sheng approved these changes Jan 13, 2020
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, data.json and the code looks good to me. But I haven't tested this PR locally.

@narph
Copy link
Contributor Author

narph commented Jan 13, 2020

Overall, data.json and the code looks good to me. But I haven't tested this PR locally.

thanks for the review, the PR will be in the test plan.

@narph narph merged commit ad8c503 into elastic:master Jan 13, 2020
@narph narph deleted the azure-storage branch January 13, 2020 20:22
narph added a commit to narph/beats that referenced this pull request Jan 13, 2020
@narph
Add azure storage metricset (elastic#15342)
312daaa 
* create metricset

* work in prog

* Work on metricset

* Add serviceType config option

* Fix tests

* Add func to retrieve on all dimensions

* work on different intervals

* revert custom event format

* work on reducing api calls

* fix fields and dashboards, work on reducing amount of api calls

* Add json example

* refactor

* refactor

* fix tests

* fix tests

* feedback

(cherry picked from commit ad8c503)
@narph narph mentioned this pull request Jan 13, 2020
Merged
narph added a commit that referenced this pull request Jan 14, 2020
@narph
Cherry-pick #15342 to 7.x: Add azure storage metricset (#15514)
4d0bacd 
* Add azure storage metricset (#15342)

* create metricset

* work in prog

* Work on metricset

* Add serviceType config option

* Fix tests

* Add func to retrieve on all dimensions

* work on different intervals

* revert custom event format

* work on reducing api calls

* fix fields and dashboards, work on reducing amount of api calls

* Add json example

* refactor

* refactor

* fix tests

* fix tests

* feedback

(cherry picked from commit ad8c503)

* Fix changelog
@narph narph added test-plan Add this PR to be manual test plan and removed needs_backport PR is waiting to be backported to other branches. labels Jan 14, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Jan 15, 2020
This was referenced Jan 21, 2020
Closed
Closed
Closed
Closed
@kaiyan-sheng kaiyan-sheng added the test-plan-ok This PR passed manual testing label Jan 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@houndci-bot houndci-bot houndci-bot left review comments

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

@ChrsMark ChrsMark Awaiting requested review from ChrsMark

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
Metricbeat Metricbeat release-highlight Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan test-plan-ok This PR passed manual testing v7.6.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@narph @houndci-bot @ChrsMark @kaiyan-sheng @urso @andresrc