Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
-
Updated
Aug 30, 2024 - Jupyter Notebook
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
ETL (Extract, Transform and Load) library for .NET
Introduction to CPRD using synthetic datasets
CryptoDataPy is a python library that makes it easy to build high quality data pipelines for the analysis of cryptoassets
One ETL tool to rule them all
A framework for writing Unstract Tools/Apps
No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
Step into a world of data magic with this dynamic repository! 🧙♂️ From Shell Scripts to Apache Airflow and Kafka, explore a collection of ETL pipelines that blend clarity with power. Whether you're into seamless data integration or real-time streaming solutions, this repo is your ticket to mastering the art of data engineering. 🚀
The CNPJ Data ETL Pipeline is designed to automate the download, processing, and storage of public CNPJ data from the Brazilian Federal Revenue. The pipeline is built with Mage.ai and AWS S3 to ensure efficient data management and scalability.
Fetch data weekly using ETL approach from tommorow.io weather forecast data.
Airflow DAGs for the Stellar ETL project
Stellar ETL will enable real-time analytics on the Stellar network
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
Jayvee is a domain-specific language and runtime for automated processing of data pipelines
A minimalist framework for assembling data processing pipelines.
Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
Add a description, image, and links to the etl-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the etl-pipeline topic, visit your repo's landing page and select "manage topics."