feat(contrib): add Kafka plugin

Makefile

@@ -174,5 +174,6 @@ plugin:
 plugins:
 	GOARCH=amd64 PLUGIN=twitter make plugin
 	GOARCH=amd64 PLUGIN=mastodon make plugin
+	GOARCH=amd64 PLUGIN=kafka make plugin
 	GOARCH=amd64 PLUGIN=rake make plugin
 .PHONY: plugins

contrib/kafka/README.md

@@ -0,0 +1,47 @@
+# Kafka plugin for Feedpushr
+
+Send new articles to Kafka.
+
+## Configuration
+
+You have to provides Kafka configuration in order to use this plugin:
+
+| Property | Description |
+|----------|-------------|
+| `brokers` | Comma separated list of broker host |
+| `topic` | Target topic |
+| `format` | Payload [format](https://github.com/ncarlier/feedpushr#output-format) (internal JSON format if not provided) | 
+
+## Installation
+
+Copy the `feedpushr-kafka.so` file into your Feedpushr working directory.
+
+## Usage
+
+```bash
+$ feedpushr --log-pretty --plugin ./feedpushr-kafka.so
+```
+## Testing
+
+Start Kafka Docker stack:
+
+```bash
+$ docker-compose up
+```
+
+Consume `test` topic with [kaf](https://github.com/birdayz/kaf):
+
+```bash
+$ go get github.com/birdayz/kaf/cmd/kaf
+$ kaf consume test
+```
+
+Start Feedpushr and configure Kafka output:
+
+- Brokers: `localhost:9092`
+- Topic: `test`
+
+Kaf should consume new records.
+
+---
+

contrib/kafka/docker-compose.yml

@@ -0,0 +1,29 @@
+---
+version: '2'
+services:
+  zookeeper:
+    image: confluentinc/cp-zookeeper:5.4.0
+    hostname: zookeeper
+    container_name: zookeeper
+    ports:
+      - "2181:2181"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_TICK_TIME: 2000
+
+  broker:
+    image: confluentinc/cp-kafka:5.4.0
+    hostname: broker
+    container_name: broker
+    depends_on:
+      - zookeeper
+    ports:
+      - "29092:29092"
+      - "9092:9092"
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

contrib/kafka/go.mod

@@ -0,0 +1,10 @@
+module github.com/ncarlier/feedpushr/v3/contrib/twitter
+
+require (
+	github.com/ncarlier/feedpushr/v3 v3.0.0-00010101000000-000000000000
+	github.com/segmentio/kafka-go v0.3.6
+)
+
+replace github.com/ncarlier/feedpushr/v3 => ../..
+
+go 1.13