Skip to content
#

snow-pipe

Here is 1 public repository matching this topic...

I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations

  • Updated May 2, 2023
  • Python

Improve this page

Add a description, image, and links to the snow-pipe topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the snow-pipe topic, visit your repo's landing page and select "manage topics."

Learn more