clickhouse_sinker

clickhouse_sinker is a sinker program that transfer kafka message into ClickHouse.

Features

• Easy to use and deploy, you don't need write any hard code, just care about the configuration file
• Custom parser support.
• Support multiple sinker tasks, each runs on parallel.
• Support multiply kafka and ClickHouse clusters.
• Bulk insert (by config bufferSize and flushInterval).
• Loop write (when some node crashes, it will retry write the data to the other healthy node)
• Uses Native ClickHouse client-server TCP protocol, with higher performance than HTTP.

Install && Run

By binary files (suggested)

Download the binary files from release, choose the executable binary file according to your env, modify the conf files, then run ./clickhouse_sinker -conf conf

By source

• Install Golang

• Go Get

go get -u github.com/housepower/clickhouse_sinker/...

• Build && Run

make build
## modify the config files, set the configuration directory, then run it
./dist/clickhouse_sinker -conf conf

Examples

• look at the integration test.
• there is a simple tutorial in Chinese which created by user @taiyang-li.

Support parsers

• Json
• Csv

Supported data types

• UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
• Float32, Float64
• String
• FixedString
• DateTime(UInt32), Date(UInt16)
• Array(UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64)
• Array(Float32, Float64)
• Array(String)
• Array(FixedString)
• Array(DateTime(UInt32), Date(UInt16))
• Nullable
• ElasticDateTime => Int64 (2019-12-16T12:10:30Z => 1576498230)

Configuration

See config example

Custom metric parser

• You just need to implement the parser interface on your own

type Parser interface {
	Parse(bs []byte) model.Metric
}

See json parser

Debugging

echo '{"date": "2019-07-11T12:10:30Z", "level": "info", "message": "msg4"}' | kafkacat -b 127.0.0.1:9093 -P -t logstash

clickhouse-client -q 'select * from default.logstash' 
2019-12-16	info	msg4
2019-07-11	info	msg4
2015-05-11	info	msg4
2019-07-11	info	msg4