Skip to content
#

data-engineering-project

Star

Here are 9 public repositories matching this topic...

Language: All
Filter by language
All 9 Python 8 Jupyter Notebook 1

sanketrs / implementation-of-modern-data-engineering-architecture-with-fabric_analytics

Star 3

Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.

data-science etl azure data-visualization data-warehouse data-engineering data-analytics etl-framework azure-data-factory big-data-analytics cloud-dataflow etl-pipeline big-data-projects data-engineering-pipeline bi-analytics data-pipeline-monitoring cloud-data-warehouse azure-fabric data-engineering-project
  • Updated Dec 29, 2024
  • Python

waqarg2001 / Youtube-Data-Pipeline-AWS

Star 0

Leveraging AWS Cloud Services, an ETL pipeline transforms YouTube video statistics data. Data is downloaded from Kaggle, uploaded to an S3 bucket, and cataloged using AWS Glue for querying with Athena. AWS Lambda and Glue converts to Parquet format and stores it in a cleansed S3 bucket. AWS QuickSight then visualizes the materialised data.

python aws spark aws-lambda etl aws-s3 pandas pyspark data-engineering aws-iam aws-cloudwatch data-pipeline etl-pipeline aws-glue data-engineering-workflows data-engineering-pipeline aws-lambda-layers aws-data-engineering-project data-engineering-project
  • Updated May 30, 2024
  • Python

janaom / gcp-de-project-data-pipeline-with-cloud-run-functions-airflow-biggueryml

Star 1

Build a data pipeline on Google Cloud using an event-driven architecture, leveraging GCS, Cloud Run functions, and BigQuery. Explore both VM and Composer options for Airflow management, and utilize Logging & Monitoring for pipeline health. Discover how SQL-based BigQuery ML can be used for initial ML implementation in specific scenarios.

bigquery airflow composer google-cloud-platform cloud-functions bigqueryml data-engineering-project cloud-run-functions
  • Updated Aug 25, 2024
  • Python

Improve this page

Add a description, image, and links to the data-engineering-project topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-engineering-project topic, visit your repo's landing page and select "manage topics."

Learn more