In the Project Workspace, I'll find a data set containing real messages that were sent during disaster events. I will be creating a machine learning pipeline to categorize these events so that you can send the messages to an appropriate disaster relief agency. This project will include a web app where an emergency worker can input a new message …
Personal Apache Beam studies repository
ETL pipeline that extracts their data from S3, stages them in Redshift, and transforms data into a set of dimensional tables for their analytics team to continue finding insights in what songs their users are listening to. Then we will test the database and ETL pipeline by running queries given to us by the analytics team from Sparkify and compa…
This projects showcases a Dimensional Data Warehouse model, utilizing random data from Public APIs to populate the DW. The fact table is transactional based. Data endpoints are available through an Flask based API.
Personal data pipelines project to document knowledge of open source software and ETL process
Analysis of NYC's citibike data. Technologies: Python , Prefect, dbt, Terraform , Looker data studio
Streaming pipeline project
Developed a Data pipeline leveraging Google Cloud services to automate data extraction from Google Drive, upload to GCS, and manage BigQuery tables with Dataform. Includes Airflow DAGs for efficient orchestration of ETL processes.
Some useful stuff for Data Engineers
A simple etl pipeline orchestrated with dagster, to run both locally and/or on k8s. Uses minIO as both IOManager and destination when on k8s
Automatic Trading System for OANDA. Currency pairs can be analysed by customising the list in the regression algorithms.
Orchestrate OpenSearch operations with Apache Airflow and the OpenSearch Airflow provider
GUI based data transformation web-app
Programa para guardar cambios de forma automática mientras trabajas con Power BI
Data Engineering | ETL process
Data Warehouse project on corona data. Analytics is done on country level, region level, and sub_region_level and visualized using python matplotlib.
Learning Airflow by testing through a course
Collecting training data for fine tuning a model and building pipeline on the cloud to train the model.
ETL workloads for excel sheet data-sources
Add a description, image, and links to the dataengineering topic page so that developers can more easily learn about it.
To associate your repository with the dataengineering topic, visit your repo's landing page and select "manage topics."