Kinesis connector applications written in Go
With the new release of Kinesis Firehose I'd recommend using the Lambda Streams to Firehose project for loading data directly into S3 and Redshift.
Inspired by the Amazon Kinesis Connector Library. This library is intended to be a lightweight wrapper around the Kinesis API to handle batching records, setting checkpoints, respecting ratelimits, and recovering from network errors.
The consumer expects a handler func that will process a buffer of incoming records.
func main() {
var(
app = flag.String("app", "", "The app name")
stream = flag.String("stream", "", "The stream name")
)
flag.Parse()
// create new consumer
c := connector.NewConsumer(*app, *stream)
// override default values
c.Set("maxBatchCount", 200)
c.Set("pollInterval", "3s")
// start consuming records from the queues
c.Start(connector.HandlerFunc(func(b connector.Buffer) {
fmt.Println(b.GetRecords())
// process the records
}))
select {}
}
Get the package source:
$ go get github.com/harlow/kinesis-connectors
Install gvt
:
$ export GO15VENDOREXPERIMENT=1
$ go get github.com/FiloSottile/gvt
Install dependencies into ./vendor/
:
$ gvt restore
Use the seed stream code to put sample data onto the stream.
Default logging is handled by go-kit package log. Applications can override the default loging behaviour by implementing the Logger interface.
connector.SetLogger(NewCustomLogger())
Please see CONTRIBUTING.md for more information. Thank you, contributors!
Copyright (c) 2015 Harlow Ward. It is free software, and may be redistributed under the terms specified in the LICENSE file.