Skip to content
#

s3-bucket

Here are 301 public repositories matching this topic...

Skytrax-Data-Warehouse

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

  • Updated Apr 18, 2020
  • Python

I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations

  • Updated May 2, 2023
  • Python

Improve this page

Add a description, image, and links to the s3-bucket topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the s3-bucket topic, visit your repo's landing page and select "manage topics."

Learn more