Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Define and schedule workflow, support Flink Jar/SQL, ClickHouse/Hive/Mysql SQL, Shell, etc.

This repo consists of all important concepts for data engineers.

Compass is a task diagnosis platform for bigdata

open source based development related contents

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

HVAC Engine: Psychrometrics (Humid Air) analysis Java library. Humid air properties and thermodynamic processes, flows, heating, cooling, air mixing and more. Immutable, thread-safe, very accurate.

Analyze OS Logs using a Kafka, Airflow, and Spark pipeline

Unveiling Cinematic Brilliance: Illuminating the Future of Movie Production through Data-Driven Insights

Data pipeline for multi-model databases using Microsoft Azure Cosmos DB SQL and Gremlin APIs and orchestration using Apache Airflow.

First academic big data project to implement analysis using MapReduce and Hive platform

Examples for ETL pipeline services infrastructure

A tool to find podcast metadata over an external api, store them, get their rss feeds and run ETL using Airflow, Kafka, Spark, and Cassandra. The particular Cassandra distribution used is Elassandra, which allows seamless integration with Elasticsearch. Displayed using a Gatsby app, served using Flask

Use K8s CustomResourceDefinition to define and create Airflow Dags

Text similarity based on Word2Vec on Twitter and Reddit updates, using Spark, Spring, MongoDB and Apache Airflow

aiflow-dashboard

PaddleSoft Backbone