Implementing best practices for PySpark ETL jobs and applications.
-
Updated
Jan 1, 2023 - Python
#
etl-job
Here are 75 public repositories matching this topic...
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
Mass processing data with a complete ETL for .net developers
csv etl dotnet extract sftp entity-framework business-intelligence dotnet-core load transform dotnet-standard csv-parser csv-reader csv-writer etl-job
-
Updated
Jul 28, 2025 - C#
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
data-science sql sql-server etl data-warehouse sql-query datascience data-engineering data-analytics datawarehousing data-analysis sqlserver datawarehouse data-cleaning datalake data-warehousing etl-pipeline etl-job data-lakehouse medallion-architecture
-
Updated
Apr 23, 2025 - TSQL
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
-
Updated
May 29, 2024 - C#
-
Updated
Oct 28, 2024 - HTML
This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.
big-data analytics terraform kinesis-firehose cloudwatch-logs parquet terraform-provider etl-job terraform-aws big-data-processing
-
Updated
Aug 4, 2021 - HCL
A Python PySpark Projet with Poetry
python template data-science spark etl poetry jupyter-notebook project python3 pytest pyspark data-engineering black spark-sql isort etl-pipeline etl-job pyspark-boilerplate-template
-
Updated
Jul 14, 2025 - Jupyter Notebook
An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
python airflow twitter etl twitter-api scheduler s3 data-engineering tweepy boto3 data-pipeline apache-airflow etl-pipeline etl-job data-engineering-pipeline airflow-dags twitter-data-pipeline
-
Updated
Aug 26, 2023 - Python
A declarative, SQL-like DSL for data integration tasks.
-
Updated
Jul 4, 2018 - Go
Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE
python shell docker infrastructure instagram data-science machine-learning airflow environment spark etl data-engineering travis daas elt dag etl-pipeline etl-job etl-jobs astro-airflow
-
Updated
Dec 19, 2022 - Python
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.
database etl extractor google-sheets data-engineering data-lake business-intelligence dwh elt datapipeline data-pipeline extract-transform-load usql excel-to-sql etl-job etl-process sharepoint-integration sharepoint-sync sync-google-sheet
-
Updated
May 29, 2025 - Python
Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline
-
Updated
May 25, 2023 - Python
This is a PHP project which combines ETL with different strategies to extract data from multiple databases, files, and services, transform it and load it into multiple destinations.
csv etl transformation etl-framework etl-pipeline etl-job etl-jobs etl-automation etl-process etl-processes
-
Updated
Apr 3, 2025 - PHP
Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse.
-
Updated
Nov 26, 2018 - Python
Telecom ETL is a SSIS package that ingest it's data from CSVs to DB
-
Updated
Oct 28, 2022 - TSQL
Comms processing (ETL) with Apache Flink.
-
Updated
Oct 19, 2020 - Java
Improve this page
Add a description, image, and links to the etl-job topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the etl-job topic, visit your repo's landing page and select "manage topics."