Kafka_Producer_Consumer_to_SQL_Project

Creating a Kafka producer.py, consumer.py & PgAdmin SQL DB

Required installations if not existing on the same environment:

• pip install kafka-python
• conda install -c conda-forge kafka-python
• conda install -c anaconda sqlalchemy
• conda install -c anaconda psycopg2

What the project about:

• Created Upstash account
• Created New single replica Cluster
• producer.py Python base code

• On VS created a producer.py to publish random message to Kafka every 5 seconds

• consumer.py Python base code

• On VS created a consumer.py to fitch the producer.py random messages from Kafka every 2 seconds

• After activating the environment: $ conda activate kafka-env
• On GitBash: $ python producer.py

• After activating the environment: $ conda activate kafka-env
• On GitBash: $ python consumer.py

• using engine Pushing all produced data thru consumer.py to PgAdmin SQL Database

Benificial links:

https://upstash.com/?gclid=CjwKCAjwvpCkBhB4EiwAujULMu611BMTjPj1328j0xQ8lKUNu55HbCBBtwRG8ysGnVgpFEML33sTRxoCWcMQAvD_BwE https://pypi.org/project/psycopg2-binary/