Docs · Report Bug · Roadmap · Get Help · Watch Demo · Free Swag
Backfill historical data, keep CDC in sync, handle schema changes, normalize messy data, and keep ClickHouse queries correct.
GlassFlow is an open-source stream processing engine designed for data ingestion and transformation from multiple sources into ClickHouse. GlassFlow comes with the following core functionalities:
- Stateless transformations: Powered by the expr expression engine, enabling flexible data transformations using helper functions and standard operators (e.g., removing null values or replacing missing timestamps).
- Stateful transformations: A built-in state store allows deduplication logic and temporal joins over configurable time windows.
- Filtering: Drop events you don’t want to ingest into ClickHouse before they reach your tables.
- Ingest only: Direct data transfer from many sources to ClickHouse without transformations.
- Metrics & OTEL: Built-in pipeline metrics with OpenTelemetry support.
- Dead-Letter-Queue: Keep pipelines running when faulty events occur. Inspect failed events and reprocess them later.
To get started with GlassFlow, you can:
- Install on Kubernetes: Follow our Kubernetes Installation Guide for any production deployment
- Learn More: Explore our Usage Guide to start creating data pipelines
GlassFlow is open source and can be self-hosted on Kubernetes. GlassFlow works with any managed Kubernetes service like AWS EKS, GKE, AKS, and more.
| Method | Use Case | Docs Link |
|---|---|---|
| ☸️ Kubernetes with Helm | Production and development deployment | Kubernetes Helm Guide |
For detailed documentation, visit docs.glassflow.dev. The documentation includes:
This project is licensed under the Apache License 2.0.