Skip to content
This repository has been archived by the owner on Jun 12, 2019. It is now read-only.

ovhlabs/queue-examples

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

59 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OVH Queue - Kafka examples

Build Status

Here are Kafka client examples using SASL/SSL in several languages and tested on OVH Queue.

Each example can be run to consume messages or produce messages from STDIN and requires four flags:

--broker    the Kafka broker address
--username  the SASL username
--password  the SASL password
--topic     the Kafka topic

When using OVH Queue:

  • the topic must be prefixed by the namespace (ex: --topic collector.events where the namespace is collector)
  • the consumer group must be prefixed by the username (ex: --username collector.admin --consumer-group collector.admin.group).

Set the following environment variables to execute the commands shown below: HOST, SASL_USERNAME, SASL_PASSWORD, TOPIC.

Go

Requirements
  • Go >= 1.5
Setup

Build binary:

cd go
go build -o kafka-client
Consume
./kafka-client consume \
    --brokers $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC --consumer-group $SASL_USERNAME.group-go
Produce
./kafka-client produce \
    --brokers $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC

Python

Requirements
  • Python >= 2.7
Setup

Install dependencies:

cd python
pip install --upgrade kafka-python==1.3.1
Consume
python client.py consume \
    --broker $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC --consumer-group $SASL_USERNAME.group-python
Produce
python client.py produce \
    --broker $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC

Java

Requirements
  • Java >= 1.8
  • Maven >= 3
Setup

The Java client needs a jaas.conf file containing SASL login information.

KafkaClient {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="<YOUR_SASL_USERNAME>"
   password="<YOUR_SASL_PASSWORD>";
};

Build the JAR file:

cd java
mvn compile package
Consume
java \
	-cp target/kafka-example-jar-with-dependencies.jar \
	-Djava.security.auth.login.config=jaas.conf \
	ovh.queue.Main consume \
	--broker $HOST:9093 \
	--topic $TOPIC --consumer-group $SASL_USERNAME.group-java
Produce
java \
    -cp target/kafka-example-jar-with-dependencies.jar \
    -Djava.security.auth.login.config=jaas.conf \
    ovh.queue.Main produce \
    --broker $HOST:9093 \
    --topic $TOPIC

Node.js

Requirements
  • Node.js >= 4.x
  • NPM >= 2.x
Setup

Install dependencies:

cd nodejs
npm install
Consume
node client.js consume \
    --broker $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC --consumer-group $SASL_USERNAME.group-node
Produce
node client.js produce \
    --broker $HOST:9093 \
    --username $SASL_USERNAME --password $SASL_PASSWORD \
    --topic $TOPIC