Switch branches/tags
Nothing to show
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
lib/logstash/inputs logstash-input-azurewadtable: Add Integration Tests (#2) Jan 18, 2018
spec/inputs logstash-input-azurewadtable: Add Integration Tests (#2) Jan 18, 2018
CHANGELOG.md Bump up the gem version number and update the CHANGELOG.md. Jun 27, 2016
Gemfile Fix loop issue for Azure WAD Table plug-in. Also some minor documenta… May 2, 2016
README.md Updating README.md Mar 7, 2018
Rakefile First commit - adding Logstash plug-ins for Azure. Mar 1, 2016
logstash-input-azurewadtable.gemspec logstash-input-azurewadtable: Add Integration Tests (#2) Jan 18, 2018


Logstash input plugin for Azure diagnostics data from Storage Tables


This plugin reads Azure diagnostics data from specified Azure Storage Table and parses the data for output.


You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:

logstash-plugin install logstash-input-azurewadtable

For more information, see Logstash reference Working with plugins.


Required Parameters


The Azure Storage account name.


The access key to the storage account.


The storage table to pull data from.

Optional Parameters


The plugin queries and processes table entities in a loop, this parameter is to specify the maximum number of entities it should query and process per loop. The default value is 100.


Specifies the point of time after which the entities created should be included in the query results. The default value is when the plugin gets initialized:



True to pretty print ETW files, otherwise False. The default value is False.


Specifies the seconds to wait between each processing loop. The default value is 15.


Specifies the endpoint of Azure environment. The default value is "core.windows.net".

past_queries_count Specifies the number of past queries to run so the plugin doesn't miss late arriving data. By default this is 5


        account_name => "mystorageaccount"
        access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
        table_name => "WADWindowsEventLogsTable"

Partition Key Format

When fetching data from Azure storage, this plugin assumes the data was produced by the Windows Azure Diagnostics (WAD) agent and queries according to its partition key format. The format differs depending on the eventVolume parameter in WAD configuration. Here is a short explanation of the format, not meant to be a full explanation though.

Small (default)


Medium or Large


For small eventVolume, the partition key is just the timestamp. This timestamp is a count of 100 nanoseconds since Jan 1st, 0001. The logic for computing this in the plugin is here.

For medium and large eventVolume, three '_' and a partition id is prepended to the timestamp. (Example above: 0000000000000000001). This partition id allows Azure storage to further distribute the data so it can reach better throughput. For medium eventVolume, this number can be between 0 and 9. For large eventVolume, this number can be between 0 and 99.

More information

The source code of this plugin is hosted in GitHub repo Microsoft Azure Diagnostics with ELK. We welcome you to provide feedback and/or contribute to the project.

Please also see Analyze Diagnostics Data with ELK template for quick deployment of ELK to Azure.