Kafka Streams test

Goal of this project is to generate Random data in order to insert them into different services of CDP platform.

How to run it ?

First, go to pom.xml and change cdp version to yours, change also if required, individual versions of each component.
Then Package the program:

mvn clean package

Then you can run it using this java command:

java -jar kafka-streams-test.jar

It is also possible to launch it on a platform using script: src/main/resources/launchToPlatform.sh. (Adapt it to your needs by changing user & machine of the platform)

To run it on YARN, check project yarn-submit .

Where to code ?

1. In Main file, change KStream types to be the one that your application will use

2. Change KafkaConfig file to add more configs if needed

3. Change config.properties to setup your cluster properties

4. Code Treatment you want in file Treatment (and others if you want)

To Debug

export KAFKA_OPTS="-Djava.security.auth.login.config=/home/root/kafka-streams-test/kafka-jaas-streamtest.config"
cd /opt/cloudera/parcels/CDH/lib/kafka/bin
./kafka-console-consumer.sh --bootstrap-server ccycloud-4.maryland.root.hwx.site:9092 --from-beginning  --consumer.config consumer.properties --topic words