Stock_Market_Data_Pipeline

This system fetches real-time stock data from Finnhub using WebSocket, streams it via Kafka, stores it in PostgreSQL, and visualizes it through a Streamlit dashboard using Plotly.

Flow Diagram

Stack Used

• WebSocket – For real-time data feed from Finnhub.

• Kafka – Handles real-time data streaming (Producer & Consumer setup).

• PostgreSQL – Stores the stock price data.

• Pandas – Used for DML statements.

• Streamlit – Builds the UI for data visualization.

• Plotly – Interactive charts for real-time stock prices.

Installation and Setup

1. Clone the repository

git clone https://github.com/adhishakya/Stock_Market_Data_Pipeline.git
cd Stock_Market_Data_Pipeline

2. Setup Environment Variables

URL= wss://your_finnhub_socket
TOKEN = your_finnhub_token
DB_NAME = your_db
DB_USER = your_user
DB_PASS = your_password
DB_HOST = localhost
DB_PORT = 5432
STREAM_TOPIC = your_kafka_topic

3. Install Dependencies

pip install -r requirements.txt

4. Start required services

• Start Zookeeper and Kafka:

# Example (if using local install)
zookeeper-server-start.sh config/zookeeper.properties
kafka-server-start.sh config/server.properties

• Start PostgreSQL and ensure the required table/schema is created.

5. Run the App

• Listener and Producer (WebSocket to Kafka):

python main.py

• Consumer (Kafka to PostgreSQL)

python consumer.py

• Streamlit Visualization

streamlit run chart.py

Demo

• Data Storage into Database

• Stock Data

API Source

• Finnhub Stock APIs