Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

v0.1.2 (with LTSV and ignore_start_check_error option)

  • Loading branch information...
commit c3cc7a5cc636aa43ea4a4343cdb69afa8268f5c6 1 parent 0b9bd90
@tagomoris tagomoris authored
Showing with 28 additions and 3 deletions.
  1. +25 −0 README.md
  2. +3 −3 fluent-plugin-webhdfs.gemspec
View
25 README.md
@@ -5,6 +5,7 @@ Fluentd output plugin to write data into Hadoop HDFS over WebHDFS/HttpFs.
WebHDFSOutput slices data by time (specified unit), and store these data as hdfs file of plain text. You can specify to:
* format whole data as serialized JSON, single attribute or separated multi attributes
+ * or LTSV, labeled-TSV (see http://ltsv.org/ )
* include time as line header, or not
* include tag as line header, or not
* change field separator (default: TAB)
@@ -25,6 +26,16 @@ To store data by time,tag,json (same with 'type file') over WebHDFS:
path /path/on/hdfs/access.log.%Y%m%d_%H.log
</match>
+To store data as LTSV without time and tag over WebHDFS:
+
+ <match access.**>
+ type webhdfs
+ host namenode.your.cluster.local
+ port 50070
+ path /path/on/hdfs/access.log.%Y%m%d_%H.log
+ output_data_type ltsv
+ </match>
+
With username of pseudo authentication:
<match access.**>
@@ -88,6 +99,20 @@ Or with random filename (to avoid duplicated file name only):
With configurations above, you can handle all of files of '/log/access/20120820/*' as specified timeslice access logs.
+### For unstable Namenodes
+
+With default configuration, fluent-plugin-webhdfs checks HDFS filesystem status and raise error for inacive NameNodes.
+
+If you were usging unstable NameNodes and have wanted to ignore NameNode errors on startup of fluentd, enable `ignore_start_check_error` option like below:
+
+ <match access.**>
+ type webhdfs
+ host namenode.your.cluster.local
+ port 50070
+ path /log/access/%Y%m%d/${hostname}.log
+ ignore_start_check_error true
+ </match>
+
## TODO
* patches welcome!
View
6 fluent-plugin-webhdfs.gemspec
@@ -1,7 +1,7 @@
# -*- encoding: utf-8 -*-
Gem::Specification.new do |gem|
gem.name = "fluent-plugin-webhdfs"
- gem.version = "0.1.1"
+ gem.version = "0.1.2"
gem.authors = ["TAGOMORI Satoshi"]
gem.email = ["tagomoris@gmail.com"]
gem.summary = %q{Fluentd plugin to write data on HDFS over WebHDFS, with flexible formatting}
@@ -15,11 +15,11 @@ Gem::Specification.new do |gem|
gem.add_development_dependency "rake"
gem.add_development_dependency "fluentd"
- gem.add_development_dependency "fluent-mixin-plaintextformatter", '>= 0.2.0'
+ gem.add_development_dependency "fluent-mixin-plaintextformatter", '>= 0.2.1'
gem.add_development_dependency "fluent-mixin-config-placeholders"
gem.add_development_dependency "webhdfs", '>= 0.5.1'
gem.add_runtime_dependency "fluentd"
- gem.add_runtime_dependency "fluent-mixin-plaintextformatter", '>= 0.2.0'
+ gem.add_runtime_dependency "fluent-mixin-plaintextformatter", '>= 0.2.1'
gem.add_runtime_dependency "fluent-mixin-config-placeholders"
gem.add_runtime_dependency "webhdfs", '>= 0.5.1'
end
Please sign in to comment.
Something went wrong with that request. Please try again.