Skip to content
Collects many small insterts to ClickHouse and send in big inserts
Branch: master
Clone or download
Nikolay Pavlovich
Nikolay Pavlovich update docker go version
Latest commit 2478ca2 Apr 16, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore config.json ignore May 1, 2017
.goreleaser.yml build binaries Mar 17, 2018
.travis.yml update go version Apr 16, 2019
Dockerfile update docker go version Apr 16, 2019
LICENSE Initial commit Apr 29, 2017
Makefile
README.md Update README.md Mar 5, 2019
clickhouse.go lint fixes & version command Mar 21, 2018
clickhouse_test.go fix query parse bug + added coverage Mar 22, 2018
collector.go collector: use RWMutex Feb 18, 2019
collector_test.go fix query parse bug + added coverage Mar 22, 2018
config.sample.json fixed lost data when stop signal; Dec 12, 2017
doc.go fixed lost data when stop signal; Dec 12, 2017
go.mod add support for go modules Feb 19, 2019
go.sum add support for go modules Feb 19, 2019
main.go added docker image and some env variables Apr 25, 2018
main_test.go fix query parse bug + added coverage Mar 22, 2018
sender.go fixed lost data when stop signal; Dec 12, 2017
server.go
server_test.go tests: remove accidentally added line Feb 19, 2019
utils.go lint fixes & version command Mar 21, 2018

README.md

ClickHouse-Bulk

Build Status codecov download binaries Go Report Card godoc

Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.

Installation

Download binary for you platorm

or

Use docker image

or from sources (Go 1.11+):

git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build

Features

  • Group n requests and send to any of ClickHouse server
  • Sending collected data by interval
  • Tested with VALUES, TabSeparated formats
  • Supports many servers to send
  • Supports query in query parameters and in body
  • Supports other query parameters like username, password, database
  • Supports basic authentication

For example:

INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')

sends as

INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')

Options

  • -config - config file (json); default config.json

Configuration file

{
  "listen": ":8124", 
  "flush_count": 10000, // check by \n char
  "flush_interval": 1000, // milliseconds
  "debug": false, // log incoming requests
  "dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
  "clickhouse": {
    "down_timeout": 300, // wait if server in down (seconds)
    "servers": [
      "http://127.0.0.1:8123"
    ]
  }
}

Environment variables (used for docker image)

  • CLICKHOUSE_SERVERS - comma separated list of servers
  • CLICKHOUSE_FLUSH_COUNT - count of rows for insert
  • CLICKHOUSE_FLUSH_INTERVAL - insert interval

Quickstart

./clickhouse-bulk and send queries to :8124

Tips

For better performance words FORMAT and VALUES must be uppercase.

You can’t perform that action at this time.