Install the following
- Spark
- Kafka
- MongoDB
Run the following commands in different terminals or push them to background
# navigate to kafka location firstly
sudo bin/zookeeper-server-start.sh config/zookeeper.properties # start zookeeper
sudo bin/kafka-server-start.sh config/server.properties # start kafka
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092 # describe topic
python3 stream.py #start python client to get twitter data
python3 client_stream.py # start python client for spark client and kafka producer
python3 mongo_client.py # start python client for inserting to database- 9008 -> Stream from twitter API to spark app
- 9092 -> Kafka endpoints
- 27017 -> Mongo client