Skip to content

ZeaShell v0.4.0 - DuckDB SQL Integration & Enhanced Viewer

Choose a tag to compare

View all tags
@lmccay lmccay released this 21 Mar 17:56
· 13 commits to main since this release
v0.4.0
2be5b12

ZeaShell v0.4.0 - DuckDB SQL Integration & Enhanced Viewer

DataFrame Shell - CSV to petabytes, one pipe at a time

This release adds powerful SQL analytics with DuckDB integration and major viewer enhancements including unlimited scrolling and schema inspection. ZeaShell now offers both high-performance Arrow pipelines and reliable file-based processing with automatic fallback.

🎉 Major Features

DuckDB SQL Integration

Execute SQL queries directly in your data pipelines with full DuckDB analytics support. ZeaShell provides a dual-path architecture for both bleeding-edge performance and rock-solid reliability.

Key capabilities:

  • 🦆 Full DuckDB SQL - Aggregations, joins, window functions, CTEs, and more
  • ⚡ Arrow-native mode - Zero-copy processing for 2-10x faster performance
  • 📁 File-based mode - Reliable CSV temp files with debuggable intermediate results
  • 🔄 Auto-detection - Automatically selects best execution mode
  • 🎯 Stdin integration - Input data available as stdin table
  • 🔌 Pipeline compatible - Mix SQL with DataFrame operations seamlessly

Execution modes:

# Auto-detection (default) - picks best mode automatically
zea load sales.csv | zea sql "SELECT region, SUM(amount) FROM stdin GROUP BY region"

# Force Arrow-native mode (fastest, zero-copy)
zea load --output=arrow sales.csv | zea sql --arrow "SELECT * FROM stdin WHERE amount > 1000"

# Force file-based mode (reliable, debuggable)
zea load sales.csv | zea sql --file "SELECT COUNT(*) FROM stdin"

Example queries:

# Simple aggregation
zea load sales.csv | zea sql "SELECT region, SUM(amount) as total FROM stdin GROUP BY region"

# Window functions
zea load sales.csv | zea sql "SELECT *, ROW_NUMBER() OVER (PARTITION BY region ORDER BY amount DESC) as rank FROM stdin"

# Complex analytics with CTEs
zea load sales.csv | zea sql "WITH high_value AS (SELECT * FROM stdin WHERE amount > 1000) SELECT customer, COUNT(*) as transactions FROM high_value GROUP BY customer"

# Hybrid workflows - mix DataFrame ops and SQL
zea load sales.csv | zea filter "amount > 500" | zea sql "SELECT customer, SUM(amount) FROM stdin GROUP BY customer" | zea view

Performance tiers:

  1. Arrow-native (✅ Available) - Zero-copy, in-memory, ~2-10x faster
  2. File-based (default) - Temp files, reliable, debuggable
  3. Universal - CSV fallback for all formats

Arrow IPC Support

High-performance Arrow Interprocess Communication for zero-copy data transfer between pipeline stages.

Key capabilities:

  • 🚀 Zero-copy pipelines - Eliminate serialization overhead
  • 📊 Arrow IPC format - Apache Arrow's streaming format
  • 🔗 Full pipeline support - All data commands support --output=arrow
  • ⚡ 2-10x faster - Measured performance improvement over CSV
  • 🎯 Composable - Mix Arrow and CSV stages as needed

Enable Arrow pipelines:

# Single command
zea load --output=arrow sales.csv | zea sql --arrow "SELECT * FROM stdin"

# Full pipeline
zea load --output=arrow data.csv | zea filter --output=arrow "amount > 100" | zea select --output=arrow customer,amount | zea sql --arrow "SELECT * FROM stdin"

Supported commands with --output=arrow:

  • zea load --output=arrow - Load data as Arrow IPC
  • zea filter --output=arrow - Filter and output Arrow
  • zea select --output=arrow - Select columns as Arrow
  • zea sql --arrow - SQL with Arrow input

Enhanced Interactive Viewer

The TUI viewer has been completely upgraded with unlimited scrolling, dynamic position tracking, and schema inspection.

Major improvements:

1. Unlimited Scrolling

  • ❌ Removed: 1K row hard limit
  • ✅ Now: Scroll through datasets of any size
  • 🎯 Navigation: PgUp/PgDn, Home/End, arrow keys all work seamlessly
  • 📊 Performance: Memory-efficient rendering for millions of rows

Before: Limited to first 1,000 rows with truncation message
Now: Full access to all rows with smooth scrolling

2. Dynamic Status Bar

  • 📍 Position tracking: Shows current row and column (e.g., Row: 1,247/5,000 | Col: 3/4)
  • 🔄 Real-time updates: Status bar updates as you navigate
  • 📊 Filter/sort info: Shows active transformations
  • 🎯 Always oriented: Never lose your place in large datasets

Example status bar:

Row: 1,247/5,000 | Col: 3/4 | Filter: amount > 1000 | Sort: amount (desc)

3. Schema Display (NEW!)

  • Press d to view comprehensive schema and metadata
  • 📋 Column types: See inferred types (string, int64, float64, bool)
  • 🔍 Null statistics: Count and percentage of nulls per column
  • 📊 Dataset info: Total rows and columns
  • 🎯 Filter impact: Compare filtered vs original row counts
  • ✨ Sort tracking: See active sort column and direction

Example schema display:

┌─ Schema & Metadata ────────────────────────────────────┐
│ Schema:                                                 │
│ -------                                                 │
│   id                      int64                         │
│   customer                string                        │
│   amount                  float64                       │
│   region                  string      (2 nulls, 16.7%)  │
│   date                    string                        │
│                                                         │
│ Total Rows: 12                                          │
│ Total Columns: 5                                        │
│                                                         │
│ Active Filter: amount > 1000                            │
│ Original Rows: 50                                       │
│                                                         │
│ Press ESC or d to close                                 │
└─────────────────────────────────────────────────────────┘

Use cases:

  • Quick schema reference during exploration
  • Data quality checks (null counts)
  • Filter verification (see impact on row counts)
  • Type validation before writing expressions

📦 Installation

Homebrew (macOS/Linux)

brew tap open-tempest-labs/zeashell
brew install zeashell
# Or upgrade:
brew upgrade zeashell

Using Go Install

go install github.com/open-tempest-labs/zeashell/cmd/zea@v0.4.0

From Source

git clone https://github.com/open-tempest-labs/zeashell
cd zeashell
git checkout v0.4.0
go build -o zea ./cmd/zea

🚀 Quick Start

SQL Analytics

# Simple aggregation
zea load sales.csv | zea sql "SELECT region, SUM(amount) as total FROM stdin GROUP BY region ORDER BY total DESC"

# Window functions for ranking
zea load sales.csv | zea sql "SELECT customer, amount, RANK() OVER (ORDER BY amount DESC) as rank FROM stdin LIMIT 10"

# CTEs for complex analysis
zea load sales.csv | zea sql "WITH monthly AS (SELECT strftime('%Y-%m', date) as month, SUM(amount) as total FROM stdin GROUP BY month) SELECT * FROM monthly WHERE total > 10000"

Arrow Pipelines

# Enable Arrow for maximum performance
zea load --output=arrow large_dataset.csv | zea sql --arrow "SELECT region, COUNT(*) as count FROM stdin GROUP BY region"

# Full Arrow pipeline
zea load --output=arrow sales.csv | zea filter --output=arrow "amount > 100" | zea sql --arrow "SELECT customer, SUM(amount) FROM stdin GROUP BY customer"

Enhanced Viewer

# Open a large dataset
zea view large_dataset.csv

# Navigation:
#   PgUp/PgDn - Scroll through all rows (no limits!)
#   d - Show schema and metadata
#   s - Sort by column
#   f - Filter rows
#   g - Show graphs
#   e - Export

# Status bar shows your position: Row: 1,247/10,000 | Col: 3/8

📚 Documentation

New Documentation

Updated Documentation

🔄 Upgrading

Homebrew

brew update
brew upgrade zeashell

Go Install

go install github.com/open-tempest-labs/zeashell/cmd/zea@v0.4.0

📝 Complete Feature List

All features from previous releases plus new additions:

Core Features:

  • 🔄 Pipeable Commands - Full Unix pipe compatibility
  • 📊 Interactive TUI - Terminal viewer with unlimited scrolling
  • 🚀 Multi-Format - CSV, TSV, JSON, JSONL, XML, Parquet
  • 🗂️ Partitioned Data - Glob patterns, parallel multi-file loading
  • 🌐 HTTP/HTTPS - Load data directly from URLs
  • 🔗 Relational Joins - Inner, left, right, full outer joins
  • ↔️ Pivot/Unpivot - Transform between long and wide formats
  • 🦆 NEW: DuckDB SQL - Full SQL analytics in pipelines
  • ⚡ NEW: Arrow IPC - High-performance zero-copy pipelines
  • 🔌 Plugin System - Extend with custom commands
  • 🎯 Expressive - SQL-like filter expressions
  • 🏗️ Production Ready - Type inference, error handling, streaming I/O

Viewer Enhancements:

  • ✅ NEW: Unlimited scrolling - No more 1K row limit
  • 📍 NEW: Dynamic status bar - Real-time position tracking
  • 🔍 NEW: Schema display - Press 'd' for metadata and types
  • ✨ Full cell value viewing - Press Enter on truncated cells
  • Sort, filter, graph, and export
  • Interactive data exploration

Performance:

  • Arrow-native execution (2-10x faster than CSV)
  • Parallel file loading
  • Columnar storage
  • Memory-efficient streaming

🎯 Key Improvements

DuckDB SQL

  • Full SQL analytics engine integrated
  • Dual-path architecture (Arrow + File modes)
  • Automatic mode detection with manual override
  • Window functions, CTEs, complex joins
  • Seamless pipeline integration

Arrow Performance

  • Zero-copy data transfer
  • 2-10x performance improvement
  • All data commands support Arrow output
  • Compatible with CSV fallback

Viewer Usability

  • Removed artificial 1K row limit
  • Added dynamic position tracking
  • Schema inspection with 'd' key
  • Null statistics and data quality info
  • Filter impact visualization

🐛 Bug Fixes

  • Fixed viewer row limit that prevented exploring large datasets
  • Improved status bar to show current position instead of just totals
  • Better error handling in Arrow mode with clear fallback messages

🔮 What's Next

Looking ahead to v0.5.0:

  • MotherDuck integration for cloud analytics
  • Interactive REPL mode
  • Enhanced Arrow support across all commands
  • Performance optimizations for multi-GB datasets

🙏 Contributors

Thanks to all contributors who made this release possible!

🔗 Links

Process data at the speed of thought with ZeaShell v0.4.0! 🦆⚡🐚

Assets 2
Loading