undatum 1.7.0

What's new in 1.7.0

Added

• undatum tui — optional Textual explorer (pip install "undatum[tui]") over a bounded sample
• undatum web — optional local browser session (pip install "undatum[web]") on 127.0.0.1:8765
• --flatten-nested on inspect/transforms, plus --max-nested-depth / --keep-nested-parents
• Convert options: --profile, --level, --native-batch, --write-mode, --row-group-size, --filename-pattern
• --threads process-pool parallelism for Python-engine convert/validate/stats/frequency
• CLI defaults config (undatum config show), man page, plugin I/O, SDK result objects, Data API hardening

Changed

• iterabledata>=1.0.18 is now the floor dependency
• --filter is comparison/boolean only (==, AND/OR); no MistQL

Removed

• undatum query and the mistql dependency — use undatum sql or select --filter (undatum db query is unchanged)

Fixed

• pipeline run executes steps in-process with $step output references
• Minimum versions for setuptools, dnspython, numpy, and zipp (Snyk advisories)

Binaries

Linux/macOS/Windows PyInstaller artifacts were not published with this tag (packaging/undatum.spec is not in the repo yet). Install via PyPI:

pip install -U undatum

Full notes: CHANGELOG.md

undatum 1.6.0

What's new in 1.6.0

Added

• repack command — max-compression recompress for container codecs and Parquet/ORC/AVRO
• --low-memory on convert/sort/dedup — spill-to-disk paths for multi-GB files
• db dump — export table/SQL query to Parquet/CSV/JSONL
• package add-resource / package validate, Dataset.package(), pipeline package step, undatum[frictionless]
• pyarrow as a default dependency; CI install-gate smoke tests
• CSV delimiter auto-detection; Excel --start-page on more commands; AI doc schema enrichment

Changed

• Gzip (gz/gzip) routes to DuckDB; convert/select/analyze/package improvements
• Docs synced with iterabledata 1.0.14–1.0.18 (140+ formats, lakehouse/open-data notes)

Fixed

• repack progress bar; package create errors/--autodoc; empty analyze; DuckDB uniq results

Full notes: CHANGELOG.md

pip install -U undatum

v1.5.0 — Data API OpenAPI, usability, and hardening

Highlights

This release significantly improves the optional Data API (undatum api): richer OpenAPI documentation, better CLI usability, and targeted correctness fixes.

Added

• api openapi command — export OpenAPI 3.x schema from an API config without starting the server (--output, --format json|yaml)
• Data API discovery endpoint — GET / returns resource list and documentation links
• Data API startup banner — prints base URL, resource endpoints, and /docs links when the server starts
• httpx added to the api optional extra (required for HTTP integration tests)

Changed

• Data API list responses — endpoints now return {data, pagination} instead of a bare JSON array (breaking for API clients)
• Data API OpenAPI — per-resource query parameters, field schemas, and field__op filter documentation in Swagger UI
• Data API sorting — sort=field and sort=-field aliases supported alongside order_by / order_dir
• Data API pagination — optional include_total=true adds total matching row count to the response envelope
• CI — test job installs undatum[api] so Data API HTTP tests run in CI

Fixed

• Data API — discover stores absolute file paths; warns on resource name collisions; validates files at serve time; skips composite primary-key detail routes; serializes DuckDB types to JSON-safe values
• Data API — serve, run, and openapi raise a clear DependencyError when the api extra is not installed
• api-serve-data recipe — default flow uses api run; config path defaults to api-config.yml

Install

pip install undatum==1.5.0

# Data API extra
pip install "undatum[api]==1.5.0"

Full changelog: https://github.com/datacoon/undatum/blob/v1.5.0/CHANGELOG.md

v1.4.0

What's new in 1.4.0

This release builds on iterabledata as the I/O foundation to add several net-new capabilities.

Added

• ai commands — AI-assisted workflows backed by iterable.ai: ai doc (block-based dataset documentation with metadata enrichment and PII-safe sampling), ai filter (natural-language/DSL → filter translation, with --apply to execute and stream matching rows), ai plan (declarative conversion planning), and ai suggest (transform suggestions). Supports OpenAI, Anthropic, Gemini, Azure, OpenRouter, Ollama, LM Studio, and Perplexity.
• formats commands — formats list surfaces iterabledata's full format catalog; --capabilities shows the runtime capability matrix (read/write/streaming/etc.) per format, with machine-readable JSON output.
• mcp commands — mcp serve starts a Model Context Protocol stdio server exposing undatum's agent tools; mcp tools lists them. New undatum-mcp console script.
• Agent tools (undatum.tools) — 17 JSON-schema agent tools for LLM function calling: the 12 iterabledata foundation tools plus 5 undatum-specific tools (query_sql, frequency, deduplicate, mask_fields, sample_data). Includes OpenAI/Anthropic tool definitions, a unified call_tool, and a LangChain get_tools() adapter.
• SDK DataFrame & typed-row interop — Dataset.to_pandas(), to_polars(), to_dask(), as_dataclasses(), and as_pydantic().
• Bulk conversion — convert --recursive (with --to-ext) converts a directory or glob pattern.
• Extended database engines — db query and file-reading commands now reach MS SQL Server (mssql://, sqlserver://), ClickHouse (clickhouse://), MongoDB (mongodb://), and Elasticsearch/OpenSearch (elasticsearch://, opensearch://) via iterabledata's read-only drivers; driver options can be passed through the URI query string.
• Cloud storage URIs beyond S3 — GCS (gs:///gcs://), Azure (az:///abfs:///abfss://), and s3a:// are opened natively via iterabledata's fsspec support; s3:// writes are now supported.
• New optional extras: mcp, langchain, polars, dask, cloud (fsspec/s3fs/gcsfs/adlfs), mssql (pyodbc), clickhouse (clickhouse-driver).

Changed

• convert now routes through iterabledata's engine, supporting any format it can read/write (100+ formats, including cloud URIs). Read-only and schema-required output formats (protobuf, Cap'n Proto, Thrift) fail fast with actionable, capability-aware errors and writable-format suggestions.
• SUPPORTED_FILE_TYPES, COMPRESSED_FILE_TYPES, TEXT_DATA_TYPES, and BINARY_FILE_TYPES are now derived at import time from iterabledata's registries (with static fallbacks).
• doc command metadata extraction and semantic-type/PII detection now delegate to iterable.ai.metadata and iterable.ai.semantic while keeping CLI output backward compatible.

Full Changelog: v1.3.0...v1.4.0

v1.3.0

Highlights

This release delivers the improvement-plan work across product correctness, code quality, and new features.

New capabilities

• undatum sql — ad-hoc DuckDB SQL over CSV, JSONL, Parquet, and other file formats
• --version flag and built-in shell completion (--install-completion / --show-completion)
• profile alias for stats; pipeline templates list/init commands
• Python SDK — from undatum import Dataset with working stats(), count(), head(), and tail()
• S3 read support across file-reading commands (via undatum[s3])
• --progress on convert, validate, and join; --threads configures DuckDB-backed commands
• CI quality gates — ruff, black, coverage; PyPI release workflow via tag push

Architecture & quality

• Split monolithic core.py, ingester.py, and statistics.py into focused packages
• Unified error handling with UndatumError hierarchy (no more silent exit-0 failures)
• Shared command scaffolding in undatum/common/command_utils.py
• Packaging fully on pyproject.toml; recipes ship inside the wheel

Install

pip install --upgrade undatum
pip install "undatum[s3]"      # S3 support
pip install "undatum[api]"     # Data API
pip install "undatum[extract]" # PDF/table extraction

See CHANGELOG.md for the full list of changes.

v1.1.1

Added

• Added workflow and OpenSpec documentation for change proposals and agent workflows
• Added dataset documentation examples under examples/doc/

Changed

• Expanded README with documentation pointers and dataset doc references

v1.1.0 - Major Feature Release: 23 New Data Processing Commands

🎉 Major Feature Release

This release adds 23 new data processing commands across three phases, along with major improvements to schema generation, statistics performance, and database ingestion.

✨ New Commands

Phase 1 - Fundamental Data Processing (7 commands):

• count - Count rows with DuckDB optimization
• table - Pretty-print data as aligned table
• head - Extract first N rows
• tail - Extract last N rows
• enum - Add row numbers, UUIDs, or constants
• reverse - Reverse row order
• fixlengths - Normalize field counts

Phase 2 - Data Cleaning & Transformation (9 commands):

• sort - Sort by columns with numeric/descending options
• sample - Random sampling (fixed count or percentage)
• search - Regex-based search and filtering
• dedup - Remove duplicates with key-field options
• fill - Fill empty/null values with strategies
• rename - Rename fields by mapping or regex
• explode - Split columns by separator
• replace - String replacement with regex support
• cat - Concatenate files by rows or columns

Phase 3 - Advanced Data Processing (7 commands):

• join - Relational joins (inner, left, right, full outer)
• diff - Compare files and show differences
• exclude - Remove rows based on keys
• transpose - Swap rows and columns
• sniff - Detect file properties
• slice - Extract rows by range or index
• fmt - Reformat CSV with formatting options

🚀 Performance Improvements

• Stats Command: 10-100x faster with DuckDB engine for CSV, JSONL, JSON, and Parquet files
• DuckDB Integration: Automatic engine selection for optimal performance
• Batch Operations: Improved performance with write_bulk() for large datasets

📋 Schema Improvements

• Format Exports: Support for JSON Schema, Avro, Parquet, and Cerberus formats
• Full Output Support: Text, JSON, and YAML output formats now work correctly
• AI Documentation: Working AI-powered field descriptions with provider selection
• Record Counting: Statistics now include record counts in schema output

🗄️ Database Ingestion

• MySQL Support: Auto-create table, upsert, and batch operations
• SQLite Support: File and in-memory databases with PRAGMA optimizations
• Improved Performance: Better support for PostgreSQL, DuckDB, MongoDB, and Elasticsearch

🔄 Migration & Deprecations

• Iterabledata Migration: All commands now use external iterabledata library
• Resource Management: Improved cleanup with try/finally blocks
• Deprecated: Local IterableData and DataWriter classes (use open_iterable() instead)
• Deprecated: scheme command (use schema --format cerberus instead)

🐛 Bug Fixes

• Fixed resource leaks in statistics, textproc, and ingester commands
• Fixed schema command output format options being ignored
• Fixed schema command AI documentation not working
• Fixed missing record counting in schema output

---

Full Changelog: v1.0.18...v1.1.0

Release v1.0.17

Changed

• Improved CLI documentation: Enhanced all command-line interface functions with detailed help text using Typer's Annotated types
• Code refactoring: Refactored analyzer output writing into separate _write_analysis_output() function for better maintainability
• Better file handling: Improved file output handling in analyzer command with proper context managers

Fixed

• Fixed analyzer output not writing to files correctly when --output option was used
• Improved consistency between stdout and file output formatting

Release 1.0.16 - Multi-Provider AI Support

Release 1.0.16 - Multi-Provider AI Support

🎉 Major Features

Multi-Provider AI Support

undatum now supports multiple AI providers for automatic field and dataset documentation:

• OpenAI - GPT-4o-mini, GPT-4o, GPT-3.5-turbo, and more
• OpenRouter - Unified API for accessing models from OpenAI, Anthropic, Google, and others
• Ollama - Run local models without API keys
• LM Studio - Local models via OpenAI-compatible API
• Perplexity - Backward compatible with existing Perplexity integration

Structured AI Output

• Replaced fragile text parsing with JSON Schema-based structured output
• More reliable AI response parsing
• Better error handling and fallback mechanisms

Flexible Configuration

Configure AI providers through:

1. Environment variables (lowest precedence)
2. Config files (undatum.yaml or ~/.undatum/config.yaml)
3. CLI arguments (highest precedence)

✨ What's New

Added

• Multi-provider AI support: Added support for OpenAI, OpenRouter, Ollama, LM Studio, and Perplexity APIs
• Structured AI output: Replaced fragile text parsing with JSON Schema-based structured output for reliable AI responses
• Flexible AI configuration: Support for environment variables, config files (undatum.yaml or ~/.undatum/config.yaml), and CLI arguments with proper precedence
• AI provider factory: New get_ai_service() function for easy provider instantiation
• Enhanced error handling: Proper exception classes (AIServiceError, AIConfigurationError, AIAPIError) with clear error messages
• CLI arguments for AI: Added --ai-provider, --ai-model, and --ai-base-url options to analyze command
• Configuration management: New undatum/ai/config.py module for unified configuration handling
• Backward compatibility: Old get_fields_info() and get_description() functions maintained for compatibility
• Enhanced code quality improvements and Pylint score improvements
• Better error handling and resource management

Changed

• AI system refactoring: Completely refactored AI documentation system from Perplexity-only to multi-provider architecture
• Structured responses: All AI providers now use JSON Schema (response_format: json_object) instead of parsing CSV from markdown code blocks
• Provider architecture: Implemented abstract base class AIService with concrete provider implementations
• Improved code quality: fixed indentation, trailing whitespace, and formatting issues
• Refactored file operations to use with statements for better resource management
• Updated string formatting to use f-strings and lazy logging
• Fixed dangerous default arguments in function signatures
• Improved type hints and code documentation
• Updated analyze command to accept AI provider configuration
• Updated schemer command to use new AI service interface

Fixed

• Fixed critical bug: added missing _process_json_data function in analyzer module
• Fixed bad indentation issues in duckdb_decompose function
• Fixed redefined builtin id parameter (renamed to table_id)
• Fixed unused imports and arguments
• Fixed dictionary iteration patterns (removed unnecessary .keys() calls)
• Fixed isinstance() calls to use tuple syntax for better performance
• Improved file handling with proper context managers
• Fixed fragile AI response parsing: Replaced error-prone text extraction with proper JSON parsing
• Fixed AI service initialization: Added proper error handling and fallback when AI service fails to initialize

📦 Installation

pip install --upgrade undatum

🔗 Links

• Full Changelog
• Documentation

Release 1.0.14

Added JSON to JSON lines conversion
Fixed #19 missing xmltodict dependency