v0.2.0 - Minimum Viable Product Release (Apache DataFusion Engine)

[saurabhr2]

🚀 det v0.2.0 — Minimum Viable Product Release

Engine change from Polars to Apache DataFusion, addition of decoupled workspace modules, and release of the v0.2.0 MVP.

⚡ Core Engine & Architectural Directives

• Apache DataFusion & Arrow: Replaced Polars with embedded Apache DataFusion (vectorized Rust query engine) and Apache Arrow columnar memory for SIMD-accelerated, multi-threaded SQL execution across CPU cores.
• Modularity: Fully decoupled workspace architecture covering Catalogs (catalog/*.yaml), Pipeline DAGs (pipelines/*/*.yaml), Storage Connectors, Execution Runner, Static Documentation Generator, and Web IDE.
• Performance: Ingest, join, and transform multi-gigabyte datasets in-memory with sub-second step latency and zero cloud egress fees.
• Extensibility: Standardized BaseConnector plugin framework enabling custom data sources (PostgreSQL, ClickHouse, DuckDB, REST APIs) in pure Python.

📦 Polyglot Formats & Medallion Pipelines

• Unifies Parquet, Delta Lake (ACID), multi-sheet Excel (.xlsx), CSV, JSON, NDJSON, and XML in ANSI SQL joins.
• Declarative multi-step DAG orchestrator supporting intermediate caching, automated column lineage, and schema derivation.

🏪 Bundled NovaMart Global Warehouse

• Instantly unpack 14 connections, 8 file formats, and 10 production SQL pipelines using det sample-data.

🌐 Interactive Web IDE & Live Documentation Portal

• Visual DAG topology viewer, Monaco SQL runner, catalog schema tree, and telemetry monitor (det gui).
• Standalone HTML/Markdown documentation generation (det docs) and live showcase hosted on GitHub Pages.

🛡️ Governance & Security

• Licensed under MIT License with copyright attribution to Saurabh Rai and det-Labs.
• Added CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and automated Dependabot updates.

---

This discussion was created from the release v0.2.0 - Minimum Viable Product Release (Apache DataFusion Engine).