End-To-End Data Engineering Project. Made to learn some common data engineering practices.
Build a data pipeline on Google Cloud using an event-driven architecture, leveraging GCS, Cloud Run functions, and BigQuery. Explore both VM and Composer options for Airflow management, and utilize Logging & Monitoring for pipeline health. Discover how SQL-based BigQuery ML can be used for initial ML implementation in specific scenarios.
A fully automated ETL pipeline that fetches and stores real-time traffic and weather data in BigQuery, with a live Looker dashboard for visualization.
PySpark-based ETL pipeline leveraging Dataproc, Cloud Storage, Cloud Run Functions and BigQuery, to automate Spotify "New Releases" data processing and visualization in Looker Studio.
Add a description, image, and links to the cloud-run-functions topic page so that developers can more easily learn about it.
To associate your repository with the cloud-run-functions topic, visit your repo's landing page and select "manage topics."