Better SQL in Jupyter. 📊

Spark based applications to perform big data analytics

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Streaming Processing

ORM for Apache Spark and DataFrames schema manager

Leveraged AWS, PySpark, and Power BI to analyze trends in PC video game genres. Optimized ETL processes and utilized datasets and the Steam API to reveal nuanced genre frequencies and distributions. Delivered insights driving decisions in game development, marketing, and platform enhancement.

PoC Minio Spark in Kubernetes

Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!

DataTalksClub Data Engineering Zoomcamp Project

A simple demonstration of an Airflow-Kafka-Spark (AKS) stack for online time series forecasting.

A Big Data project leveraging AWS services and Apache frameworks to identify and visualize fraudulent credit card transaction patterns, providing actionable insights to mitigate financial fraud.

Pequenos projetos utilizando ferramentas do ecossistema Apache Hadoop

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

Proceso de ETL: proceso de ingesta, transformación y carga de data al DataWarehouse. Todo esto es una guía personal sobre los pasos que realicé para llevar adelante el proyecto solicitado, igual cualquier sugerencia/error es bien recibida para seguir aprendiendo más y mejorar. Cualquier contirbución es recibida!!

Application that trains a classifier and predicts flight arrival delays based on past information. Uses the libraries pyspark.ml and pyspark.sql, performs feature engineering, cross-validation and tests various ML algorithms.

Self directed Python PoC etc/ PostgreSQL / Apache Spark / Pandas

ETL building and analysis

Batch/stream ETL pipeline of NOAA GLM dataset, using Python frameworks: Dagster, PySpark and Parquet storage.