An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect

A Python package to enrich Google Cloud Data Catalog Fileset Entries with tags.

Creating a robust and scalable data pipeline on Google Cloud Platform (GCP) to monitor and analyze stock performance. Leveraging the power of GCP's data processing and storage services, a comprehensive solution has been built to efficiently collect, process, and visualize stock data.

Interacting with public cloud object storage

Batch ETL pipeline project on GCP to load and transform daily flight data using Spark to update tables in BigQuery. The pipeline is automated using Airflow.

Airflow script that ingest data from API and upload it to gcs bucket

Ingesting NYC Yellow Taxi Data to GCP using Airflow running on Docker

This is in order to add additional stocks data information using dbt.

Util functions to organize train/test data

This project aims to migrate data from MongoDB to Google Cloud Storage (GCS) and BigQuery automatically. It enables businesses to easily transfer and analyze the data in the cloud, improving data and cost management.

Simple python scripts to upload local files into GCP's bucket containerized with docker.

Lambda function for 'webapp' to save files in GCP bucket and logs in DynamoDB tables when SNS topic triggers

Analysis of NYC's citibike data. Technologies: Python , Prefect, dbt, Terraform , Looker data studio

Simplified blueprints for building data pipelines with Google Cloud Storage (GCS).