Example of how to use Kafka and Spark to handle streaming submissions of urls.

Criação de datalake utilizando os dados da Gamers Club.

A transformation pipeline for Delta Lake using AWS SDK for Pandas

Data pipeline that processes Formula1 data with Azure Databricks, DeltaLake, and Azure Data Factory

Data Streaming with Debezium, Kafka, Spark Streaming, Delta Lake, and MinIO

Example of local pyspark setup including DeltaLake for unit-testing

Completed the SQL Basics for Data Science Specialization from the University of California, Davis, gaining proficiency in Data Analysis, SQL, Apache Spark, and Delta Lake.

deltalake tutorial w/ spark, hive, hadoop

This is a pyspark pipeline for consume message from kafka and insert into delta table

Exercício do módulo 1 do Bootcamp Engenheiro de Dados Cloud IGTI 2022

Formula1 Load, Ingestion, Transformation and analysis pipeline

Extraction and data wrangling from Twitter API with Apache Airflow, PySpark and DeltaLake

"Explore Formula 1 data analytics with this project. Leveraging the Ergast API, it utilizes Databricks Spark for ingestion, transformation, and analysis. ADLS acts as the storage layer, while Power BI visualizes the ADLS presentation layer. Uncover insights in the world of Formula 1 through powerful data analytics."

contains notebooks with solutions for data prepping and implementation of the Medallion Architecture and Delta Lake Storage

Qubole Delta Lake Spark Streaming ingestion end to end Demo

Type annotations for delta-spark

Jenkins Delta pipeline

Running Spark ETL Jobs with Airflow

🍺 A data engineering project showcasing an ELT pipeline using modern technologies such as Delta-rs, and Apache Airflow.

🛸 This project showcases an Extract, Load, Transform (ELT) pipeline built with Python, Apache Spark, Delta Lake, and Docker. The objective of the project is to scrape UFO sighting data from NUFORC and process it through the Medallion architecture to create a star schema in the Gold layer that is ready for analysis.