Structured Streaming

In this project I created a structured streaming with common technologies.
It was a good practice to use these technologies while working together.

• Producer:
  Simple Flask App by ordering food and beverages.
  Flask App get that data and load to Kafka topic for every order.
  

• Consumer:
  Spark Session to read stream from Kafka topic.
  PySpark reads streaming from Kafka, does some data processing and writes to Elasticsearch.
  

• Visualization:
  Kibana to visualize Elasticsearch data.
  Elasticsearch index and Kibana dashboard operations handled by init_db

Pipeline

Flask App

Kibana Dashboard

Starting Services

Docker versions

$ sudo docker-compose up
is fine to start all services in proper sequence.

Usage

Flask App: localhost:5000
Kibana Dashboard: localhost:5602