Initial commit

ChangeLog.md

@@ -0,0 +1,3 @@
+## 0.1.0
+
+* Inital Release

Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-azuresearch.gemspec
+gemspec

README.md

@@ -1,2 +1,157 @@
 # fluent-plugin-azure-loganalytics
-Azure Log Analytics output plugin for Fluentd
+[Azure Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-overview) output plugin for Fluentd. The plugin aggregates semi-structured data in real-time and writes the buffered data via HTTPS request to Azure Log Analytics.
+
+![fluent-plugin-azure-loganalytics overview](https://github.com/yokawasa/fluent-plugin-azure-loganalytics/raw/master/img/Azure-LogAnalytics-Fluentd.png)
+
+## Installation
+```
+$ gem install fluent-plugin-azure-loganalytics
+```
+
+## Configuration
+
+### Azure Log Analytics
+To start running with Log Analytics in the Microsoft Operations Management Suite (OMS), You need to create either an OMS workspace using the OMS website or Log Analytics workspace using your Azure subscription. Workspaces created either way are functionally equivalent. Here is an instruction:
+
+ * [Get started with Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-get-started)
+
+Once you have the workspace, get Workspace ID and Shared Key (either Primary Key or Secondary Key), which are needed by [Log Analytics HTTP Data Collector API](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api) to post the data to Log Analytics.
+
+
+### Fluentd - fluent.conf
+
+```
+<match azure-loganalytics.**>
+    @type azure-loganalytics
+    customer_id CUSTOMER_ID   # Customer ID aka WorkspaceID String
+    shared_key KEY_STRING     # The primary or the secondary Connected Sources client authentication key
+    log_type EVENT_TYPE_NAME  # The name of the event type. ex) ApacheAccessLog
+    add_time_field true
+    time_field_name mytime
+    time_format %s
+    localtime true
+    add_tag_field true
+    tag_field_name mytag
+</match>
+```
+
+ * **customer\_id (required)** - Your Operations Management Suite workspace ID
+ * **shared\_key (required)** - The primary or the secondary Connected Sources client authentication key
+ * **log\_type (required)** - The name of the event type that is being submitted to Log Analytics
+ * **add\_time\_field (optional)** - Default:true. This option allows to insert a time field to record
+ * **time\_field\_name (optional)** - Default:time. This is required only when add_time_field is true
+ * **localtime (optional)** - Default:false. Time record is inserted with UTC (Coordinated Universal Time) by default. This option allows to use local time if you set localtime true. This is valid only when add_time_field is true
+ * **time\_format (optional)** -  Default:%s. Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example. This is valid only when add_time_field is true.
+ * **add\_tag\_field (optional)** - Default:false. This option allows to insert a tag field to record
+ * **tag\_field\_name (optional)** - Default:tag. This is required only when add_time_field is true
+
+
+## Configuration examples
+
+fluent-plugin-azure-loganalytics adds **time** and **tag** attributes by default if **add_time_field** and **add_tag_field** are true respectively. Below are two types of the plugin configurations - Default and All options configuration.
+
+### (1) Default Configuration (No options)
+<u>fluent.conf</u>
+```
+<source>
+    @type tail                         # input plugin
+    path /var/log/apache2/access.log   # monitoring file
+    pos_file /tmp/fluentd_pos_file     # position file
+    format apache                      # format
+    tag azure-loganalytics.access      # tag
+</source>
+
+<match azure-loganalytics.**>
+    @type azure-loganalytics
+    customer_id 818f7bbc-8034-4cc3-b97d-f068dd4cd658
+    shared_key ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksCzvBmQXHw==(dummy)
+    log_type ApacheAccessLog
+</match>
+```
+
+### (2) Configuration with All Options
+<u>fluent.conf</u>
+```
+<source>
+    @type tail                         # input plugin
+    path /var/log/apache2/access.log   # monitoring file
+    pos_file /tmp/fluentd_pos_file     # position file
+    format apache                      # format
+    tag azure-loganalytics.access      # tag
+</source>
+
+<match azure-loganalytics.**>
+    @type azure-loganalytics
+    customer_id 818f7bbc-8034-4cc3-b97d-f068dd4cd658
+    shared_key ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksCzvBmQXHw==(dummy)
+    log_type ApacheAccessLog
+    add_time_field true
+    time_field_name mytime
+    time_format %s
+    localtime true
+    add_tag_field true
+    tag_field_name mytag
+</match>
+```
+
+## Sample inputs and expected records
+
+An expected output record for sample input will be like this:
+
+<u>Sample Input (apache access log)</u>
+```
+125.212.152.166 - - [17/Jan/2016:05:03:25 +0000] "GET /foo/bar/test.html HTTP/1.1" 304 179 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
+```
+
+<u>Output Record</u>
+
+![fluent-plugin-azure-loganalytics output image](https://github.com/yokawasa/fluent-plugin-azure-loganalytics/raw/master/img/Azure-LogAnalytics-Output-Image.png)
+
+
+## Tests
+### Running test code
+```
+$ git clone https://github.com/yokawasa/fluent-plugin-azure-loganalytics.git
+$ cd fluent-plugin-azure-loganalytics
+
+# edit CONFIG params of test/plugin/test_azure_loganalytics.rb
+$ vi test/plugin/test_azure_loganalytics.rb
+
+# run test
+$ rake test
+```
+
+### Creating package, running and testing locally
+```
+$ rake build
+$ rake install:local
+
+# running fluentd with your fluent.conf
+$ fluentd -c fluent.conf -vv &
+
+# send test apache requests for testing plugin ( only in the case that input source is apache access log )
+$ ab -n 5 -c 2 http://localhost/foo/bar/test.html
+```
+
+## Change log
+* [Changelog](ChangeLog.md)
+
+## Links
+
+* https://rubygems.org/gems/fluent-plugin-azure-loganalytics
+* https://rubygems.org/gems/azure-loganalytics-datacollector-api
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/fluent-plugin-azure-loganalytics.
+
+## Copyright
+
+<table>
+  <tr>
+    <td>Copyright</td><td>Copyright (c) 2016- Yoichi Kawasaki</td>
+  </tr>
+  <tr>
+    <td>License</td><td>Apache License, Version 2.0</td>
+  </tr>
+</table>

Rakefile

@@ -0,0 +1,15 @@
+#!/usr/bin/env rake
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default do
+  system("rake -T")
+end
+

VERSION

@@ -0,0 +1 @@
+0.1.0

examples/fluent_1.conf

@@ -0,0 +1,14 @@
+<source>
+    @type tail                         # input plugin
+    path /var/log/apache2/access.log   # monitoring file
+    pos_file /tmp/fluentd_pos_file     # position file
+    format apache                      # format
+    tag azure-loganalytics.access      # tag
+</source>
+
+<match azure-loganalytics.**>
+    @type azure-loganalytics
+    customer_id CUSTOMER_ID   # Customer ID aka WorkspaceID String
+    shared_key KEY_STRING     # The primary or the secondary Connected Sources client authentication key
+    log_type EVENT_TYPE_NAME  # The name of the event type. ex) ApacheAccessLog
+</match>

examples/fluent_2.conf

@@ -0,0 +1,20 @@
+<source>
+    @type tail                         # input plugin
+    path /var/log/apache2/access.log   # monitoring file
+    pos_file /tmp/fluentd_pos_file     # position file
+    format apache                      # format
+    tag azure-loganalytics.access      # tag
+</source>
+
+<match azure-loganalytics.**>
+    @type azure-loganalytics
+    customer_id CUSTOMER_ID   # Customer ID aka WorkspaceID String
+    shared_key KEY_STRING     # The primary or the secondary Connected Sources client authentication key
+    log_type EVENT_TYPE_NAME  # The name of the event type. ex) ApacheAccessLog
+    add_time_field true
+    time_field_name mytime
+    time_format %s
+    localtime true
+    add_tag_field true
+    tag_field_name mytag
+</match>

fluent-plugin-azure-loganalytics.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |gem|
+  gem.name          = "fluent-plugin-azure-loganalytics"
+  gem.version       = File.read("VERSION").strip
+  gem.authors       = ["Yoichi Kawasaki"]
+  gem.email         = ["yoichi.kawasaki@outlook.com"]
+  gem.summary       = %q{Azure Functions output plugin for Fluentd}
+  gem.description   = gem.summary
+  gem.homepage      = "http://github.com/yokawasa/fluent-plugin-azure-loganalytics"
+  gem.license       = "Apache-2.0"
+  gem.has_rdoc       = false
+
+  gem.files         = `git ls-files`.split("\n")
+  gem.executables   = gem.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|gem|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_dependency "fluentd", [">= 0.10.58", "< 2"]
+  gem.add_dependency "rest-client"
+  gem.add_dependency "azure-loganalytics-datacollector-api", [">= 0.1.1"]
+  gem.add_development_dependency "bundler", "~> 1.11"
+  gem.add_development_dependency "rake", "~> 10.0"
+  gem.add_development_dependency "test-unit"
+end
+

lib/fluent/plugin/out_azure_loganalytics.rb

@@ -0,0 +1,85 @@
+# -*- coding: utf-8 -*-
+
+module Fluent
+  class AzureLogAnalyticsOutput < BufferedOutput
+    Plugin.register_output('azure-loganalytics', self)
+
+    def initialize
+      super
+      require 'msgpack'
+      require 'time'
+      require "azure/loganalytics/datacollectorapi/client"
+    end
+
+    config_param :customer_id, :string,
+                 :desc => "Your Operations Management Suite workspace ID"
+    config_param :shared_key, :string, :secret => true,
+                 :desc => "The primary or the secondary Connected Sources client authentication key"
+    config_param :log_type, :string, default: nil,
+                 :desc => "The name of the event type that is being submitted to Log Analytics"
+    config_param :add_time_field, :bool, :default => true,
+                 :desc => "This option allows to insert a time field to record"
+    config_param :time_field_name, :string, :default => "time",
+                 :desc => "This is required only when add_time_field is true"
+    config_param :time_format, :string, :default => "%s",
+                 :desc => "Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example. This is valid only when add_time_field is true."
+    config_param :localtime, :bool, :default => false,
+                 :desc => "Time record is inserted with UTC (Coordinated Universal Time) by default. This option allows to use local time if you set localtime true. This is valid only when add_time_field is true."
+    config_param :add_tag_field, :bool, :default => false,
+                 :desc => "This option allows to insert a tag field to record"
+    config_param :tag_field_name, :string, :default => "tag",
+                 :desc => "This is required only when add_time_field is true"
+
+    def configure(conf)
+      super
+      raise ConfigError, 'no customer_id' if @customer_id.empty?
+      raise ConfigError, 'no shared_key' if @shared_key.empty?
+      raise ConfigError, 'no log_type' if @log_type.empty?
+      if @add_time_field and @time_field_name.empty?
+        raise ConfigError, 'time_field_name must be set if add_time_field is true'
+      end
+      if @add_tag_field and @tag_field_name.empty?
+        raise ConfigError, 'tag_field_name must be set if add_tag_field is true'
+      end
+      @timef = TimeFormatter.new(@time_format, @localtime)
+    end
+
+    def start
+      super
+      # start
+      @client=Azure::Loganalytics::Datacollectorapi::Client::new(@customer_id,@shared_key)
+    end
+
+    def shutdown
+      super
+      # destroy
+    end
+
+    def format(tag, time, record)
+      if @add_time_field
+        record[@time_field_name] = @timef.format(time)
+      end
+      if @add_tag_field
+        record[@tag_field_name] = tag
+      end
+      record.to_msgpack
+    end
+
+    def write(chunk)
+      records = []
+      chunk.msgpack_each { |record|
+        records.push(record)
+      }
+      begin
+        res = @client.post_data(@log_type, records)
+        if not Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
+          $log.fatal "DataCollector API request failure: error code: " 
+                  + "#{res.code}, data=>" + records.to_json
+        end
+      rescue Exception => ex
+        $log.fatal "Exception occured in posting to DataCollector API: " 
+                  + "'#{ex}', data=>" + records.to_json
+      end
+    end
+  end
+end

test/helper.rb

@@ -0,0 +1,29 @@
+require 'rubygems'
+require 'bundler'
+
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'test/unit'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'fluent/test'
+unless ENV.has_key?('VERBOSE')
+  nulllogger = Object.new
+  nulllogger.instance_eval {|obj|
+    def method_missing(method, *args)
+      # pass
+    end
+  }
+  $log = nulllogger
+end
+
+require 'fluent/plugin/out_azure_loganalytics'
+
+class Test::Unit::TestCase
+end