Skip to content

Releases

v0.1.0

v0.1.0 Latest
Latest

Choose a tag to compare

View all tags
@Jason1008Jia Jason1008Jia released this 31 Oct 07:52
v0.1.0
2d6fb66

Welcome to the very first public release of Dojao — a distributed data analysis compute engine designed for the AI era.

One SQL, across every data silo.

Dojao is not a database, nor is it just a query engine. It allows analysts to connect data scattered across heterogeneous multi-source systems with a single SQL statement, and enables data scientists to use Python statistics or small models directly as SQL functions without writing a single line of JVM language.

Our Core Philosophy: Memory runs out, disk takes over.
We prioritize determinism over peak performance. When memory runs out, Dojao seamlessly spills to disk, ensuring your big-data queries always finish without OOM failures.

Key Features in v0.1.0

True Federated Queries

  • Zero Data Movement: Query across MySQL, TiDB, Doris, PostgreSQL, SQL Server, Oracle, Hive, ClickHouse, and even local Excel files in a single SQL statement.
  • 9 Built-in Connectors: Fully aligned capabilities including metadata probing, parallel split scanning, column pruning, and predicate pushdown.
  • Pluggable SPI: Add long-tail data sources without changing a single line of engine code.

Disk-Assisted Compute & Spilling

  • Unified Spill Pipeline: Streaming spill and merge logic shared by core operators like GroupBy, Window, SortMergeJoin, and HashJoin.
  • Data Stays, Index Moves: Underlying data is stored in off-heap memory using Apache Arrow. Sorting and grouping during computation only rearrange row indices (int[]), avoiding physical data movement and memory copies.
  • Zero Serialization Overhead: The Arrow columnar format in memory is identical to the spill file format on disk. When spilling and reloading, the engine reuses the exact same memory structure, drastically reducing the heavy serialization and deserialization costs typical of traditional engines.
  • Reducing Spill: Unlike traditional engines that dump raw data, Dojao prunes data before writing to disk (e.g., Window rn=1 collapses to 1 row per group during spill), significantly reducing disk I/O and merge pressure.

Native Python UDF / UDAF

  • Register Python scripts using NumPy, Polars, or Scikit-learn models directly as SQL functions.
  • Zero-copy Arrow Flight: Language-neutral DFP protocol running in isolated sidecar processes.

AI-Native Agent Integration

  • Built-in streaming ReAct Agent (LangChain4j + OpenAI endpoints).
  • 13 built-in tools for reading SQL, metadata exploration, cluster diagnostics, and multi-format exports (CSV, Excel, HTML, PNG, PDF).
  • Role-based tool authorization with mandatory human confirmation for DDL execution.

Advanced Memory Model

  • Data off-heap (Arrow), Index on-heap: Both share a single budget (query.slot.memory).
  • Derives all memory limits automatically from a single config key (system.<role>.memory).
  • Shared root allocator (NodeMemory) ensures a query fails gracefully rather than crashing the entire node.

Built-in Web Console & Clients

  • Real-time query monitoring, memory/spill telemetry, lineage DAGs, SQL autocomplete, and dark/light themes.
  • Includes CLI and JDBC driver for seamless BI tool integration.

Positioning vs. Trino / Spark

Dojao doesn't chase benchmark peaks; it targets the most painful scenarios for traditional engines: small memory, scattered sources, forced spills, and zero tolerance for failure.

  • Trino's spill is marked as legacy and may be orders of magnitude slower.
  • Spark's spill is robust but dumps the full data volume.
  • Dojao's spill is reducing — 68M row window deduplication collapses to 1 row per partition during spill. The memory line is not a cliff for Dojao.

Installation & Quick Start

Download the dojao-0.1.0.tar.gz below, extract it, and you are ready to go:

tar -xzf dojao-0.1.0.tar.gz
cd dojao-0.1.0/dojao
./bin/dojao    # Defaults to system.role = both (Master + Worker in one process)
Assets 7
Loading