Permalink
Browse files

initial commit

  • Loading branch information...
0 parents commit 97ff11e889ddd52ba9812a30b021936e4a19c187 @kentaro committed Jun 30, 2012
Showing with 593 additions and 0 deletions.
  1. +18 −0 .gitignore
  2. +4 −0 Gemfile
  3. +2 −0 LICENSE
  4. +178 −0 README.md
  5. +9 −0 Rakefile
  6. +18 −0 fluent-plugin-rewrite.gemspec
  7. +99 −0 lib/fluent/plugin/rewrite.rb
  8. +233 −0 test/plugin/test_out_rewrite.rb
  9. +32 −0 test/test_helper.rb
@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+vendor/bundler
@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-rewrite.gemspec
+gemspec
@@ -0,0 +1,2 @@
+Copyright (c) 2012 Kentaro Kuribayashi
+Apache License, Version 2.0
@@ -0,0 +1,178 @@
+# fluent-plugin-rewrite
+
+## Component
+
+### RewriteOutput
+
+Output plugin to rewrite messages' tags/values along with pattern
+matching and re-emit them.
+
+## Synopsis
+
+```
+<match apache.log.**>
+ type rewrite
+
+ remove_prefix apache.log
+ add_prefix filtered
+
+ <rule>
+ key path
+ pattern \\?.+$
+ replace
+ </rule>
+ <rule>
+ key path
+ pattern (/[^/]+)\\?([^=]+)=(\\d)
+ replace \\1/\\2/\\3
+ </rule>
+ <rule>
+ key status
+ pattern ^500$
+ ignore true
+ </rule>
+ <rule>
+ key path
+ pattern ^\/(users|entries)
+ append_to_tag true
+ fallback others
+ </rule>
+</match>
+```
+
+## Configuration
+
+### remove_prefix / add_prefix
+
+```
+remove_prefix apache.log
+add_prefix filtered
+```
+
+- remove_prefix: removes the string from a prefix of tag.
+- add_prefix: prepend the string to a tag.
+
+### rule: replace
+
+For example, if you want to filter out query string form URL string:
+
+```
+<rule>
+ key path
+ pattern \\?.+$
+ replace
+</rule>
+```
+
+It executes pattern matching against a value related with `key` and replaces it with empty string if it matches.
+
+```
+/foo?bar=baz -> /foo
+```
+
+This time, if you want to rewrite path string along with some pattern:
+
+```
+<rule>
+ key path
+ pattern (/[^/]+)\\?([^=]+)=(\\d)
+ replace \\1/\\2/\\3
+</rule>
+```
+
+It executes pattern matching against a value related with `key` and replaces it with `replace` if it matches.
+
+```
+/foo?bar=1 -> /foo/bar/1
+```
+
+### rule: ignore
+
+For example, if you want to skip a message which matches some pattern:
+
+```
+<rule>
+ key status
+ pattern ^500$
+ ignore true
+</rule>
+```
+
+It executes pattern matching against a value related with `key` and skip emitting the message if it matches.
+
+### rule: append_to_tag
+
+```
+<rule>
+ key path
+ pattern ^\/(users|entries)
+ append_to_tag true
+</rule>
+```
+
+It executes pattern matching against a value related with `key` and append mathed strings to message tag. For example:
+
+```
+apache.log { "path" : "/users/antipop" }
+```
+
+the messabe above will be re-emmited as the message below:
+
+```
+apache.log.users { "path" : "/users/antipop" }
+```
+
+If you set `fallback` option like below:
+
+```
+<rule>
+ key path
+ pattern ^\/(users|entries)
+ append_to_tag true
+ fallback others
+</rule>
+```
+
+the value of `fallback` option will be appended to message tag.
+
+```
+apache.log { "path" : "/foo/bar" }
+```
+
+This time, the messabe above will be re-emmited as the message below:
+
+```
+apache.log.others { "path" : "/foo/bar" }
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+ gem 'fluent-plugin-rewrite'
+
+And then execute:
+
+ $ bundle
+
+Or install it yourself as:
+
+ $ gem install fluent-plugin-rewrite
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Copyright
+
+### Copyright
+
+Copyright (c) 2012- Kentaro Kuribayashi (@kentaro)
+
+### License
+
+Apache License, Version 2.0
@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |test|
+ test.libs << 'lib' << 'test'
+ test.pattern = 'test/**/test_*.rb'
+ test.verbose = true
+end
@@ -0,0 +1,18 @@
+Gem::Specification.new do |gem|
+ gem.name = "fluent-plugin-rewrite"
+ gem.version = '0.0.1'
+ gem.authors = ["Kentaro Kuribayashi"]
+ gem.email = ["kentarok@gmail.com"]
+ gem.homepage = "http://github.com/kentaro/fluent-plugin-rewrite"
+ gem.description = %q{Fluentd plugin to rewrite tags/values along with pattern matching and re-emit them.}
+ gem.summary = %q{Fluentd plugin to rewrite tags/values along with pattern matching and re-emit them.}
+
+ gem.files = `git ls-files`.split($\)
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
+ gem.require_paths = ["lib"]
+
+ gem.add_development_dependency "rake"
+ gem.add_development_dependency "fluentd"
+ gem.add_runtime_dependency "fluentd"
+end
@@ -0,0 +1,99 @@
+module Fluent
+ class RewriteOutput < Output
+ Fluent::Plugin.register_output('rewrite', self)
+
+ config_param :remove_prefix, :string, :default => nil
+ config_param :add_prefix, :string, :default => nil
+
+ attr_reader :rules
+
+ def configure(conf)
+ super
+
+ if @remove_prefix
+ @removed_prefix_string = @remove_prefix + '.'
+ @removed_length = @removed_prefix_string.length
+ end
+ if @add_prefix
+ @added_prefix_string = @add_prefix + '.'
+ end
+
+ @rules = conf.elements.select { |element|
+ element.name == 'rule'
+ }.each { |element|
+ if element.has_key?("pattern")
+ element["regex"] = Regexp.new(element["pattern"])
+ end
+ }
+ end
+
+ def start
+ super
+ end
+
+ def shutdown
+ super
+ end
+
+ def emit(tag, es, chain)
+ es.each do |time, record|
+ tag, record = rewrite(tag, record)
+ Engine.emit(tag, time, record) if tag && record
+ end
+
+ chain.next
+ end
+
+ def length_will_be_removed
+ return 0 unless @remove_prefix
+ (@remove_prefix + '.').length
+ end
+
+ def rewrite(tag, record)
+ if @remove_prefix and
+ ((tag.start_with?(@removed_prefix_string) && tag.length > @removed_length) ||
+ tag == @remove_prefix)
+ tag = tag[@removed_length..-1]
+ end
+
+ if @add_prefix
+ tag = tag && tag.length > 0 ? @added_prefix_string + tag : @add_prefix
+ end
+
+ rules.each do |rule|
+ tag, record = apply_rule(rule, tag, record)
+ return if !tag && !record
+ end
+
+ [tag, record]
+ end
+
+ def apply_rule(rule, tag, record)
+ tag = rule["append_to_tag"] ? tag.dup : tag
+ key = rule["key"]
+ pattern = rule["pattern"]
+
+ return [tag, record] if !key || !record.has_key?(key)
+ return [tag, record] unless pattern
+
+ if matched = record[key].match(rule["regex"])
+ return if rule["ignore"]
+
+ if rule["replace"]
+ replace = rule["replace"].to_s
+ record[key] = record[key].gsub(rule["regex"], replace)
+ end
+
+ if rule["append_to_tag"]
+ matched.captures.each { |m| tag << ".#{m}" }
+ end
+ else
+ if rule["append_to_tag"] && rule["fallback"]
+ tag << ".#{rule["fallback"]}"
+ end
+ end
+
+ [tag, record]
+ end
+ end
+end
Oops, something went wrong. Retry.

0 comments on commit 97ff11e

Please sign in to comment.