The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

task management & automation tool

Example end to end data engineering project.

Smarter data pipelines for audio.

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

Code review for data in dbt

Streaming reactive and dataflow graphs in Python

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes

Making DAG construction easier

Fluent data pipelines for python and your shell

A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.

Tools for ASR Corpus Generation from Online Video

Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management

Build and deploy a serverless data pipeline on AWS with no effort.

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

Data pipelines from re-usable components

python ETL framework

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena