Qubole Delta Lake Spark Streaming ingestion end to end Demo

Example of how to use Kafka and Spark to handle streaming submissions of urls.

Criação de datalake utilizando os dados da Gamers Club.

A transformation pipeline for Delta Lake using AWS SDK for Pandas

Type annotations for delta-spark

Data pipeline that processes Formula1 data with Azure Databricks, DeltaLake, and Azure Data Factory

Data Streaming with Debezium, Kafka, Spark Streaming, Delta Lake, and MinIO

Example of local pyspark setup including DeltaLake for unit-testing

Ambiente de estudo com Apache Spark, MinIO e Delta Lake provisionado e gerenciado com Docker Compose.

This project benchmarks query performance between Databricks and Snowflake on Iceberg and Delta tables, allowing for detailed comparison and analysis. It supports customizable queries, logging of execution metrics, and fetches Snowflake query history to assist with performance optimization and cost analysis.

Implemented Azure Databricks for real-time data processing and governance using Unity Catalog, Spark Structured Streaming, Delta Lake features, Medallion Architecture, and end-to-end CI/CD pipelines. Focused on incremental loading, compute cluster management, maintaining data quality, and creating workflows.

deltalake tutorial w/ spark, hive, hadoop

Streaming data processing pipeline using Spark, PostgreSQL, Debezium, Kafka, Minio, Delta Lake, Trino and DBeaver

This is a pyspark pipeline for consume message from kafka and insert into delta table

🍺 A data engineering project showcasing an ELT pipeline using modern technologies such as Delta-rs, and Apache Airflow.

Data Pipeline from AWS SQS/S3 to Kubernetes w/ Spark using Airflow, EKS & Data Lakehouse

Completed the SQL Basics for Data Science Specialization from the University of California, Davis, gaining proficiency in Data Analysis, SQL, Apache Spark, and Delta Lake.

Streaming ETL job cases in AWS Glue to integrate Delta Lake and creating an in-place updatable data lake on Amazon S3