Skip to content

Release v0.15.1

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Jul 04:09
v0.15.1
86ecb39

v0.15.1 - 2026-07-11

Danube Iceberg - Native Lakehouse Integration

Streaming platforms and data lakes have historically been separate systems. Getting data from one to the other means building and maintaining ETL pipelines like Kafka Connect, Spark Streaming jobs, or custom Flink applications, that add operational complexity, introduce latency, and break when schemas change.

This release makes Danube a lakehouse-native messaging platform. The new danube-iceberg sidecar continuously converts sealed WAL segments into Apache Iceberg tables, making streaming data instantly queryable by Spark, Trino, DuckDB, Snowflake, and any engine that speaks Iceberg, with zero ETL pipelines to build or maintain.

Documentation: Danube Iceberg Overview · Configuration Reference

Danube Iceberg

  • Lakehouse sidecar (#237): New danube-iceberg binary that runs alongside the Danube cluster and continuously exports streaming data to Apache Iceberg tables. Discovers sealed .dnb1 segments from object storage, decodes WAL frames, writes Parquet data files with ZSTD compression, and commits them to an Iceberg catalog via atomic fast_append transactions. by @danrusei in f5e7abd

  • Schema-aware Parquet conversion (#237): Resolves schemas from the Danube Schema Registry (JSON Schema and Avro) and produces typed Parquet columns. A temperature reading becomes a Float64 column named temperature, not an opaque byte array. Query engines get real column names, native types, and per-column statistics (min/max bounds, null counts) for predicate pushdown and file pruning. Topics without a schema use an envelope fallback with offset, publish_time, producer_name, and payload columns. by @danrusei in f5e7abd

  • Multi-catalog support (#237): Supports REST catalogs (Nessie, Polaris, Tabular, Unity), AWS Glue, AWS S3 Tables, and SQL-backed catalogs (SQLite, PostgreSQL). A Parquet-only mode is available for environments without a catalog. by @danrusei in f5e7abd

  • Crash-safe checkpointing and Iceberg V2 table management (#237): Tracks progress per topic in object storage with at-least-once delivery semantics, and creates tables with Iceberg V2 format, metadata retention, and automatic cleanup after commits. by @danrusei in f5e7abd

Docker Compose Lakehouse Demo

  • End-to-end pipeline demo (#238): Complete Docker Compose Lakehouse setup demonstrating the full lakehouse pipeline: 9 simulated IoT devices → Edge Broker (MQTT) → 3-node Cluster → danube-iceberg → Apache Spark SQL (Jupyter Notebook). Includes MinIO for S3-compatible storage, Iceberg REST catalog, Prometheus monitoring, and pre-configured Jupyter notebooks for querying all three Iceberg tables (telemetry, sensors, alerts). by @danrusei in 8a06983

Storage Backend Migration

  • Replaced OpenDAL with object_store v0.14 (#236): Migrated the durable storage backend from OpenDAL to the object_store crate (the same library used by Apache Arrow, DataFusion, and Delta Lake). Adds streaming reads via GetResultPayload::Stream for memory-efficient segment processing, multipart uploads for large segment exports, and a cleaner async API surface. This also aligns danube-persistent-storage and danube-iceberg on the same storage abstraction, enabling both to share credentials and connection configuration. by @danrusei in f8e8f97

Release Notes - Binaries

Danube Broker:

Client CLI:

Admin (CLI + Server):

Danube Iceberg (Lakehouse Converter):

Checksums:

Docker Images:

  • Danube Broker: ghcr.io/danube-messaging/danube-broker:v0.15.1
  • Danube CLI: ghcr.io/danube-messaging/danube-cli:v0.15.1
  • Danube Admin: ghcr.io/danube-messaging/danube-admin:v0.15.1
  • Danube Iceberg: ghcr.io/danube-messaging/danube-iceberg:v0.15.1

Quick Start with Docker:

# Run Danube Broker
docker run -p 6650:6650 ghcr.io/danube-messaging/danube-broker:v0.15.1

# Use Danube CLI (client operations)
docker run --rm ghcr.io/danube-messaging/danube-cli:v0.15.1 danube-cli produce --help

# Use Danube Admin CLI (admin commands)
docker run --rm ghcr.io/danube-messaging/danube-admin:v0.15.1 brokers list

# Run Danube Admin Server (HTTP API for UI)
docker run -p 8080:8080 ghcr.io/danube-messaging/danube-admin:v0.15.1 serve \
  --broker-endpoint host.docker.internal:50051 \
  --prometheus-url http://host.docker.internal:9090

# Use with Docker Compose
curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/quickstart/docker-compose.yml
docker-compose up -d

Install from Archive:

# Linux
curl -LsS https://github.com/danube-messaging/danube/releases/download/v0.15.1/danube-broker-v0.15.1-x86_64-unknown-linux-gnu.tar.gz | tar xz
sudo mv danube-broker /usr/local/bin/

# macOS (Apple Silicon)
curl -LsS https://github.com/danube-messaging/danube/releases/download/v0.15.1/danube-broker-v0.15.1-aarch64-apple-darwin.tar.gz | tar xz
sudo mv danube-broker /usr/local/bin/

# Windows (PowerShell)
Invoke-WebRequest -Uri https://github.com/danube-messaging/danube/releases/download/v0.15.1/danube-broker-v0.15.1-x86_64-pc-windows-msvc.zip -OutFile danube-broker.zip
Expand-Archive danube-broker.zip -DestinationPath .
Move-Item danube-broker.exe C:\Windows\System32\

Verify Downloads:

curl -LO https://github.com/danube-messaging/danube/releases/download/v0.15.1/checksums.txt
sha256sum -c checksums.txt --ignore-missing

Contributors

danrusei
Assets 15
Loading