spark-submit-kafka

Exercise

Write spark structured streaming application that will:

• run on command line (spark-submit)
• read records from topic (args(0) - from kafka to console)
• write it down the screen

1. Run zookeeper:

    bin/zookeeper-server-start.sh config/zookeeper.properties

2. Run kafka server:

    bin/kafka-server-start.sh config/server.properties

3. Run kafka console producer:

    bin/kafka-console-producer.sh --topic <topic-name> --bootstrap-server :9092

4. Run spark server:

    bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.1 <path/to/project.jar>

Write messages from kafka producer console and watch displayed data frame in spark server console. Enjoy!