|
| 1 | +--- |
| 2 | +mapped_pages: |
| 3 | + - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-azure-storage.html |
| 4 | +applies_to: |
| 5 | + stack: ga |
| 6 | +--- |
| 7 | + |
| 8 | +% This file is generated! See scripts/docs_collector.py |
| 9 | + |
| 10 | +# Azure storage metricset [metricbeat-metricset-azure-storage] |
| 11 | + |
| 12 | +This is the storage metricset of the module azure. |
| 13 | + |
| 14 | +This metricset allows users to retrieve all metrics from specified storage accounts. |
| 15 | + |
| 16 | +## Metricset-specific configuration notes [_metricset_specific_configuration_notes_11] |
| 17 | + |
| 18 | +`refresh_list_interval` |
| 19 | +: Resources will be retrieved at each fetch call (`period` interval), this means a number of Azure REST calls will be executed each time. This will be helpful if the azure users will be adding/removing resources that could match the configuration options so they will not added/removed to the list. To reduce on the number of API calls we are executing to retrieve the resources each time, users can configure this setting and make sure the list or resources will not be refreshed as often. This is also beneficial for performance and rate/ cost reasons ([https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits)). |
| 20 | + |
| 21 | +`resources` |
| 22 | +: This will contain all options for identifying resources and configuring the desired metrics |
| 23 | + |
| 24 | +### Config options to identify resources [_config_options_to_identify_resources_11] |
| 25 | + |
| 26 | +`resource_id` |
| 27 | +: (*[]string*) The fully qualified ID’s of the resource, including the resource name and resource type. Has the format `/subscriptions/{{guid}}/resourceGroups/{{resource-group-name}}/providers/{{resource-provider-namespace}}/{resource-type}/{{resource-name}}`. Should return a list of resources. |
| 28 | + |
| 29 | +`resource_group` |
| 30 | +: (*[]string*) This option will return all storage accounts inside the resource group. |
| 31 | + |
| 32 | +`service_type` |
| 33 | +: (*[]string*) This configuration key can be used with any of the 2 options above, for example: |
| 34 | + |
| 35 | +``` |
| 36 | +resources: |
| 37 | + - resource_id: "" |
| 38 | + service_type: ["blob", "table"] |
| 39 | + - resource_group: "" |
| 40 | + service_type: ["queue", "file"] |
| 41 | +``` |
| 42 | + |
| 43 | +It will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here [https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts). The service type values allowed are `blob`, `table`, `queue`, `file` based on the namespaces `Microsoft.Storage/storageAccounts/blobServices`,`Microsoft.Storage/storageAccounts/tableServices`,`Microsoft.Storage/storageAccounts/fileServices`,`Microsoft.Storage/storageAccounts/queueServices`. If no service_type is specified all values are applied. |
| 44 | + |
| 45 | +Also, if the `resources` option is not specified, then all the storage accounts from the entire subscription will be selected. The primary aggregation value will be retrieved for all the metrics contained in the namespaces. The aggregation options are `avg`, `sum`, `min`, `max`, `total`, `count`. |
| 46 | + |
| 47 | +A default non configurable timegrain of 5 min is set so users are advised to configure an interval of 300s or a multiply of it. |
| 48 | + |
| 49 | +`default_timegrain`: |
| 50 | +: (*string*) Sets the default time grain to use when collecting storage account metrics. Defaults to PT5M. |
| 51 | + |
| 52 | +To collect storage account metrics with a PT1M time grain, we recommend using one of the following configurations: |
| 53 | + |
| 54 | +```yaml |
| 55 | +# (1) With `period: 60s` and `default_timegrain: "PT1M"`, the metricset |
| 56 | +# collects 1 data point every 60s. |
| 57 | +- module: azure |
| 58 | + metricsets: |
| 59 | + - storage |
| 60 | + enabled: true |
| 61 | + period: 60s |
| 62 | + client_id: '${AZURE_CLIENT_ID:""}' |
| 63 | + client_secret: '${AZURE_CLIENT_SECRET:""}' |
| 64 | + tenant_id: '${AZURE_TENANT_ID:""}' |
| 65 | + subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' |
| 66 | + refresh_list_interval: 3600s # 1h |
| 67 | + enable_batch_api: true |
| 68 | + default_timegrain: "PT1M" |
| 69 | +``` |
| 70 | +
|
| 71 | +```yaml |
| 72 | +# (2) With `period: 300s` and `default_timegrain: "PT1M"`, the metricset |
| 73 | +# collects 5 data points every 300s (5 minutes) — one for each minute, |
| 74 | +# but all data points arrive after 5 minutes |
| 75 | +- module: azure |
| 76 | + metricsets: |
| 77 | + - storage |
| 78 | + enabled: true |
| 79 | + period: 300s |
| 80 | + client_id: '${AZURE_CLIENT_ID:""}' |
| 81 | + client_secret: '${AZURE_CLIENT_SECRET:""}' |
| 82 | + tenant_id: '${AZURE_TENANT_ID:""}' |
| 83 | + subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' |
| 84 | + refresh_list_interval: 3600s # 1h |
| 85 | + enable_batch_api: true |
| 86 | + default_timegrain: "PT1M" |
| 87 | +``` |
| 88 | +
|
| 89 | +These two configurations trade off scalability and freshness. Configuration (1) prioritizes freshness over scalability, while configuration (2) prioritizes scalability over freshness. |
| 90 | +
|
| 91 | +Suggested changes: |
| 92 | +
|
| 93 | +- `enable_batch_api: true`: Retrieves metric values for multiple Azure resources in one API call, supporting more storage accounts. |
| 94 | +- `refresh_list_interval: 3600s`: Looks for new storage accounts every 60 minutes instead of 10 minutes, helping to avoid or reduce gaps when monitoring many storage accounts. |
| 95 | + |
| 96 | +Note: By setting the collection `period: 1m`, the metricset only has 60s to collect all metric values instead of 300s, so it can handle fewer storage accounts. Keep in mind that the storage accounts metricset collects metrics for five different namespaces (storage account, blob, file, queue, and table). |
| 97 | + |
| 98 | +## Fields [_fields] |
| 99 | + |
| 100 | +For a description of each field in the metricset, see the [exported fields](/reference/metricbeat/exported-fields-azure.md) section. |
| 101 | + |
| 102 | +Here is an example document generated by this metricset: |
| 103 | + |
| 104 | +```json |
| 105 | +{ |
| 106 | + "@timestamp": "2017-10-12T08:05:34.853Z", |
| 107 | + "azure": { |
| 108 | + "namespace": "Microsoft.Storage/storageAccounts/queueServices", |
| 109 | + "resource": { |
| 110 | + "group": "obs-infrastructure", |
| 111 | + "type": "Microsoft.Storage/storageAccounts" |
| 112 | + }, |
| 113 | + "storage": { |
| 114 | + "queue_capacity": { |
| 115 | + "avg": 0 |
| 116 | + }, |
| 117 | + "queue_count": { |
| 118 | + "avg": 0 |
| 119 | + }, |
| 120 | + "queue_message_count": { |
| 121 | + "avg": 0 |
| 122 | + } |
| 123 | + }, |
| 124 | + "subscription_id": "fd675b6f-b5e5-426e-ac45-d1f876d0ffa6", |
| 125 | + "timegrain": "PT1H" |
| 126 | + }, |
| 127 | + "cloud": { |
| 128 | + "instance": { |
| 129 | + "id": "/subscriptions/fd675b6f-b5e5-426e-ac45-d1f876d0ffa6/resourceGroups/obs-infrastructure/providers/Microsoft.Storage/storageAccounts/urcbyscmrkbygsawinvm/queueServices/default", |
| 130 | + "name": "urcbyscmrkbygsawinvm" |
| 131 | + }, |
| 132 | + "provider": "azure", |
| 133 | + "region": "westeurope" |
| 134 | + }, |
| 135 | + "event": { |
| 136 | + "dataset": "azure.storage", |
| 137 | + "duration": 115000, |
| 138 | + "module": "azure" |
| 139 | + }, |
| 140 | + "metricset": { |
| 141 | + "name": "storage", |
| 142 | + "period": 10000 |
| 143 | + }, |
| 144 | + "service": { |
| 145 | + "type": "azure" |
| 146 | + } |
| 147 | +} |
| 148 | +``` |
0 commit comments