No description, website, or topics provided.
Clone or download
Latest commit bdedb41 Dec 17, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/fluent/plugin Add arrow_file buffer plugin Dec 16, 2018
test Support nested record Dec 1, 2018
.gitignore init Nov 28, 2018
Gemfile init Nov 28, 2018
LICENSE init Nov 28, 2018
README.md Update README Nov 28, 2018
Rakefile init Nov 28, 2018
fluent-plugin-arrow.gemspec v0.0.3 Dec 17, 2018

README.md

fluent-plugin-arrow

Fluentd buffer plugin to output Apache Arrow and Parquet format.

Prerequisite

Installation

RubyGems

$ gem install fluent-plugin-arrow

Bundler

Add following line to your Gemfile:

gem "fluent-plugin-arrow"

And then execute:

$ bundle

Configuration

You can generate configuration template:

<match arrow>
  @type file

  path arrow_test

  <buffer>
    @type arrow_memory
    arrow_format arrow # or parquet

    schema [
      {"name": "key1", "type": "string"},
      {"name": "key2", "type": "uint64"},
      {"name": "key3", "type": "timestamp", "unit": "milli"},
      {"name": "key4", "type": "list", "value_type": {"name": "value", "type": "uint64"}},
      {"name": "key5", "type": "struct", "fields": [
        {"name": "bar1", "type": "uint64"},
        {"name": "bar2", "type": "list", "value_type": {"name": "value", "type": "string"}}
      ]}
    ]
  </buffer>

  <format>
    @type arrow
  </format>
</match>

You can copy and paste generated documents here.

Copyright

  • Copyright(c) 2018- joker1007
  • License
    • Apache License, Version 2.0