Streaming reference architecture built around Kafka.
Scala Other
Latest commit 98f5332 Jan 19, 2017 @stheppi stheppi updating the release notes
Permalink
Failed to load latest commit information.
bin minor coap fixes, removed timer Jan 9, 2017
conf add bloomberg authentication mode, Fixes #22 Jan 13, 2017
gradle/wrapper coap test and project wide clean up (cosmetic) Dec 30, 2016
images update readme, add quickstart props Oct 31, 2016
kafka-connect-blockchain fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-bloomberg coap test and project wide clean up (cosmetic) Dec 30, 2016
kafka-connect-cassandra Hazelcast extend datastructures (#116) Jan 18, 2017
kafka-connect-coap adding coap sink config quickstart and clean up config settings Jan 10, 2017
kafka-connect-druid clean up imports and switch of untested coap discover Jan 4, 2017
kafka-connect-elastic fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-ftp Update README.md Jan 9, 2017
kafka-connect-hazelcast Hazelcast extend datastructures (#116) Jan 18, 2017
kafka-connect-hbase fixing the support for logical types Jan 15, 2017
kafka-connect-influxdb Merge pull request #115 from datamountaineer/feature/influx_tags Jan 17, 2017
kafka-connect-jms fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-kudu fix kudu autoevlove test Jan 17, 2017
kafka-connect-mongodb fixing the unrecognized schema for standard avro types Jan 16, 2017
kafka-connect-mqtt fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-redis fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-rethink Hazelcast extend datastructures (#116) Jan 18, 2017
kafka-connect-voltdb fix default fields on kudu, also log timer Jan 12, 2017
kafka-connect-yahoo fix default fields on kudu, also log timer Jan 12, 2017
kafka-socket-streamer clean up imports and switch of untested coap discover Jan 4, 2017
style coap test and project wide clean up (cosmetic) Dec 30, 2016
.gitignore coap test and project wide clean up (cosmetic) Dec 30, 2016
.travis.yml removing travis debug Jul 4, 2016
LICENSE Added LICENSE Feb 24, 2016
README.md updating the release notes Jan 19, 2017
build.gradle Merge pull request #115 from datamountaineer/feature/influx_tags Jan 17, 2017
gradle.properties Adding FTP and HTTP connector (#109) Jan 6, 2017
gradlew updating wrapper Sep 8, 2016
gradlew.bat updating wrapper Sep 8, 2016
settings.gradle Adding FTP and HTTP connector (#109) Jan 6, 2017

README.md

Build Status Documentation Status

Join us on slack

Alt text

Stream Reactor

Streaming reference architecture built around Kafka.

Alt text

A collection of components to build a real time ingestion pipeline.

Connectors

Connector Type Description Docs
BlockChain Source Kafka connect Blockchain source to subscribe to Blockchain streams and write to Kafka. Docs
Bloomberg Source Kafka connect Blockchain source to subscribe to Blockchain streams and write to Kafka. Docs
Cassandra Source Kafka connect Cassandra source to read Cassandra and write to Kafka. Docs
Coap Source Kafka connect Coap source to read from IoT Coap endpoints using Californium. Docs
Coap Sink Kafka connect Coap sink to write kafka topic payload to IoT Coap endpoints using Californium. Docs
*DSE Cassandra Sink Certified DSE Kafka connect Cassandra sink task to write Kafka topic payloads to Cassandra. Docs
Druid Sink Kafka connect Druid sink to write Kafka topic payloads to Druid. Docs
Elastic Sink Kafka connect Elastic Search sink to write Kafka topic payloads to Elastic Search. Docs
FTP/HTTP Source Kafka connect FTP and HTTP source to write file data into Kafka topics. Docs
HBase Sink Kafka connect HBase sink to write Kafka topic payloads to HBase. Docs
Hazelcast Sink Kafka connect Hazelcast sink to write Kafka topic payloads to Hazelcast. Docs
Kudu Sink Kafka connect Kudu sink to write Kafka topic payloads to Kudu. Docs
InfluxDb Sink Kafka connect InfluxDb sink to write Kafka topic payloads to InfluxDb. Docs
JMS Sink Kafka connect JMS sink to write Kafka topic payloads to JMS. Docs
MongoDB Sink Kafka connect MongoDB sink to write Kafka topic payloads to MongoDB. Docs
MQTT Source Kafka connect MQTT source to write data from MQTT to Kafka. Docs
Redis Sink Kafka connect Redis sink to write Kafka topic payloads to Redis. Docs
ReThinkDB Source Kafka connect RethinkDb source subscribe to ReThinkDB changefeeds and write to Kafka. Docs
ReThinkDB Sink Kafka connect RethinkDb sink to write Kafka topic payloads to RethinkDb. Docs
Yahoo Finance Source Kafka connect Yahoo Finance source to write to Kafka. Docs
VoltDB Sink Kafka connect Voltdb sink to write Kafka topic payloads to Voltdb. Docs

Kafka-Socket-Streamer

Akka Http and Reactive Kafka with Websocket and Server Send Event support. Supports limited SQL statements to stream and select from Kafka topics in real time.

Release Notes

0.2.4 (In progress)

  • Added FTP and HTTO Source.
  • Added InfluxDB tag support. KCQL: INSERT INTO targetdimension SELECT * FROM influx-topic WITHTIMESTAMP sys_time() WITHTAG(field1, CONSTANT_KEY1=CONSTANT_VALUE1, field2,CONSTANT_KEY2=CONSTANT_VALUE1)
  • Added InfluxDb consistency level. Default is ALL. Use connect.influx.consistency.level to set it to ONE/QUORUM/ALL/ANY
  • InfluxDb connect.influx.sink.route.query was renamed to connect.influx.sink.kcql

0.2.3 (5 Jan 2017)

  • Added CoAP Source and Sink.
  • Added MongoDB Sink.
  • Added MQTT Source.
  • Hazelcast support for ring buffers.
  • Redis support for Sorted Sets.
  • Added start scripts.
  • Added Kafka Connect and Schema Registry CLI.
  • Kafka Connect CLI now supports pause/restart/resume; checking connectors on the classpath and validating configuration of connectors.
  • Support for Struct, Schema.STRING and Json with schema in the Cassandra, ReThinkDB, InfluxDB and MongoDB sinks.
  • Rename export.query.route to sink.kcql.
  • Rename import.query.route to source.kcql.
  • Upgrade to KCQL 0.9.5 - Add support for STOREAS so specify target sink types, e.g. Redis Sorted Sets, Hazelcast map, queues, ringbuffers.

Building

Requires gradle 3.0 to build.

To build

gradle compile

To test

gradle test

To create a fat jar

gradle shadowJar

You can also use the gradle wrapper

./gradlew shadowJar

To view dependency trees

gradle dependencies # or
gradle :kafka-connect-cassandra:dependencies