Skip to content
Implementation of ClickHouse output for logstash
Ruby
Branch: master
Clone or download

Latest commit

Krzysztof Majk
Latest commit 269d039 Nov 29, 2019

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/logstash return src on empty mutations Nov 29, 2019
.gitignore initial Mar 15, 2017
Gemfile initial Mar 15, 2017
LICENSE initial Mar 15, 2017
README.md fix mutate doc Jul 28, 2018
Rakefile initial Mar 15, 2017
logstash-output-clickhouse.gemspec Update logstash-output-clickhouse.gemspec Apr 15, 2019

README.md

Logstash Plugin

This plugin is a modified version of the Lucidworks logstash json_batch. That plugin is available here.

It has been modified to support ClickHouse JSON Format, but also supports fault tolerance.

Usage

Please note that the name of the plugin when used is clickhouse, it only supports json in its current form. If further output formats are added in the future, this might change back to json_batch.

output {
  clickhouse {
    headers => ["Authorization", "Basic YWRtaW46cGFzc3dvcmQxMjM="]
    http_hosts => ["http://your.clickhouse1/", "http://your.clickhouse2/", "http://your.clickhouse3/"]
    table => "table_name"
    mutations => {
      "to1" => "from1"
      "to2" => [ "from2", "(.)(.)", '\1\2' ]
    }
  }
}

Other custom options

  • save_on_failure (default: true) - enable / disable request body save on failure
  • save_dir (default: /tmp) - directory where failed request body will be saved
  • automatic_retries (default: 1) - number of connect retry attempts to each host in http_hosts
  • request_tolerance (default: 5) - number of http request send retry attempts if response status code is not 200
  • backoff_time (default: 3) - time to wait in seconds for next retry attempt of connect or request

Default batch size is 50, with a wait of at most 5 seconds per send. These can be tweaked with the parameters flush_size and idle_flush_time respectively.

Installation

The easiest way to use this plugin is by installing it through rubygems like any other logstash plugin. To get the latest versio installed, you should run the following command: bin/logstash-plugin install logstash-output-clickhouse

Building the gem and installing a local version

To build the gem yourself, use gem build logstash-output-clickhouse.gemspec in the root of this repository. Alternatively, you can download a built version of the gem from the dist branch of this repository.

To install, run the following command, assuming the gem is in the local directory: $LOGSTASH_HOME/bin/plugin install logstash-output-clickhouse-X.Y.Z.gem

You can’t perform that action at this time.