New architecture and new visualization capability

[Chenglong-MS]

Data Formulator 0.7.0 Alpha — More Charts, New Experience, Enterprise-Ready

This PR merges the dev branch into main with four major areas of improvement.

1. Dramatically Expanded Visualization Support

The chart template system has been rebuilt with a new semantic engine (src/lib/agents-chart/), expanding from ~15 chart types to 30 Vega-Lite chart types:

• Scatter & Point: Scatter Plot, Regression, Boxplot, Strip Plot (new), Ranged Dot Plot
• Bar: Bar Chart, Grouped Bar Chart, Stacked Bar Chart, Histogram, Lollipop Chart (new), Pyramid Chart, Heatmap
• Line & Area: Line Chart, Dotted Line Chart, Bump Chart (new), Area Chart (new), Streamgraph (new)
• Part-to-Whole: Pie Chart (new), Rose Chart (new), Waterfall Chart (new)
• Statistical: Density Plot (new), Candlestick Chart (new), Radar Chart (new)
• Map: US Map (new), World Map (new)
• Custom: Custom Point, Custom Line, Custom Bar, Custom Rect, Custom Area

Semantic field analysis now automatically infers temporal, categorical, quantitative, and geographic types to recommend the right chart for the data.

2. New Experience: Hybrid Chat + Data Thread & Enhanced Agent Mode

• Redesigned Data Thread: Chat-based interaction is woven directly into the exploration thread — users converse with agents inline alongside data transformations and chart results, replacing the separate chat panel
• Richer thread cards showing transformation lineage, chart previews, and agent reasoning in a unified timeline
• New agent mode: Agents autonomously plan multi-step explorations, generate chart recommendations, and produce data insights — all surfaced inline in the thread
• Conversational data loading via integrated chat-based data ingestion

3. Redesigned Agent Architecture

The backend agent system has been significantly restructured — consolidating previously fragmented agents into a cleaner, more capable design:

• Unified DataAgent (agents/data_agent.py) replaces four separate agents (agent_py_concept_derive, agent_py_data_rec, agent_sql_data_rec, agent_sql_data_transform) with a single agent that handles both Python and SQL data transformations
• New agent_data_transform — dedicated data transformation agent
• New agent_data_rec — recommendation agent that suggests charts and exploration directions
• New agent_chart_insight — generates natural-language insights from chart results
• Shared semantic_types — type system used by both backend agents and frontend chart engine for consistent field inference

4. Workspace / Data Lake Architecture (Enterprise-Ready)

A new persistent, identity-based Workspace layer replaces the previous in-memory DB approach:

• Workspace manages per-user directories with a workspace.yaml metadata catalog tracking every table's lineage, schema, provenance, and source type
• Uploaded files (CSV, Excel, JSON, etc.) preserved as-is; data-loader sources stored as Parquet via PyArrow
• CacheManager and FileManager for efficient caching and file lifecycle
• Azure Blob and Cached Azure Blob workspace backends for cloud deployments
• WorkspaceFactory selects the correct workspace backend from configuration
• New modular route layer replaces monolithic app routes

Start at py-src/data_formulator/datalake/ — see __init__.py for module overview and usage examples.

5. Security Hardening

• Code signing (code_signing.py) for AI-generated Python code
• Sandboxed execution with local and docker backends (sandbox/)
• Authentication layer (auth.py)
• Flask rate limiting to protect API endpoints

Other Notable Changes

• UV-first build: Fully reproducible builds with uv.lock; uv sync + uv run data_formulator is now the recommended development workflow
• Unified data upload dialog and refresh data dialog
• Demo streaming routes for live data scenarios
• api-keys.env.template consolidated into .env.template

Community Contributions

Thanks to our contributors in this PR:

• @BAIGUANGMEI — Map projection support (projection types & centers) (Add Map Chart Support with Projection Controls #232)
• @IAMkecheng — ECharts renderer: scatter plot & bar chart (refine the echart scatterplot #236)
• @joshpoll — GoFish grouped bar sizing fix (Fix GoFish bar sizing to match other backends #235)

Version: 0.7.0a1 (alpha)
Files changed: ~282 files, +84k / -16k lines

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Mestway]

Good