v0.2.0 - Minimum Viable Product Release (Apache DataFusion Engine)

🚀 det v0.2.0 — Minimum Viable Product Release

Engine change from Polars to Apache DataFusion, addition of decoupled workspace modules, and release of the v0.2.0 MVP.

⚡ Core Engine & Architectural Directives

• Apache DataFusion & Arrow: Replaced Polars with embedded Apache DataFusion (vectorized Rust query engine) and Apache Arrow columnar memory for SIMD-accelerated, multi-threaded SQL execution across CPU cores.
• Modularity: Fully decoupled workspace architecture covering Catalogs (catalog/*.yaml), Pipeline DAGs (pipelines/*/*.yaml), Storage Connectors, Execution Runner, Static Documentation Generator, and Web IDE.
• Performance: Ingest, join, and transform multi-gigabyte datasets in-memory with sub-second step latency and zero cloud egress fees.
• Extensibility: Standardized BaseConnector plugin framework enabling custom data sources (PostgreSQL, ClickHouse, DuckDB, REST APIs) in pure Python.

📦 Polyglot Formats & Medallion Pipelines

• Unifies Parquet, Delta Lake (ACID), multi-sheet Excel (.xlsx), CSV, JSON, NDJSON, and XML in ANSI SQL joins.
• Declarative multi-step DAG orchestrator supporting intermediate caching, automated column lineage, and schema derivation.

🏪 Bundled NovaMart Global Warehouse

• Instantly unpack 14 connections, 8 file formats, and 10 production SQL pipelines using det sample-data.

🌐 Interactive Web IDE & Live Documentation Portal

• Visual DAG topology viewer, Monaco SQL runner, catalog schema tree, and telemetry monitor (det gui).
• Standalone HTML/Markdown documentation generation (det docs) and live showcase hosted on GitHub Pages.

🛡️ Governance & Security

• Licensed under MIT License with copyright attribution to Saurabh Rai and det-Labs.
• Added CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and automated Dependabot updates.

v0.1.1a3

Release v0.1.1a3: table & column description display and schema nullability fix

v0.1.1a2

What's Changed in det-core v0.1.1a2

Core Database Connectors

• SQLite: Local SQLite files (.db, .sqlite, .sqlite3), object discovery, schema inference, and custom SQL execution.
• PostgreSQL: PostgreSQL databases via psycopg2 with standard type mapping, schema & table discovery, and Polars scanning.
• SQL Server: Microsoft SQL Server databases via pymssql with schema discovery and query modes.

API Connectors

• REST API: Automated ingestion from HTTP REST APIs with authentication, pagination, and JSONPath extraction.
• GraphQL: Query execution against GraphQL endpoints with variable support and nested record flattening.

File Connectors & Platform GUI

• Comprehensive support for CSV, Parquet, Excel, JSON, XML, and YAML.
• Unified Catalog & Connectors Explorer GUI with dark/light theme, live discovery, and pipeline management.

v0.1.0a1

Initial Draft including the first module dedicated for Catalog module of det. Supports both CLI and GUI (browser) interaction.