Repository navigation
NeMo Data Designer 0.10.0 adds flexible JSON storage and improves validation, processor integration, and multiprocessing reliability. This release requires Python 3.11 or later.
Highlights
- Flexible JSON storage: Preserve dictionaries, mixed JSON values, and exact numbers using
x-data-designer-storage: jsonon schema nodes orstorage_encoding="json"on whole columns. Data Designer restores logical values when reading the data; external Parquet readers see JSON text for encoded values. Storage validation catches unsupported native schemas before model calls. #1062 - Correct validation results: Keep results aligned after rows are dropped or reordered, and preserve each row’s result when validating multiple code columns. #1078, #1106
- Processor and column handling: Support
columns_addedandcolumns_removeddeclarations on pre-batch processors, and make drop flags independent of declaration order. #943, #1057 - Exact integers and embeddings: Preserve large integers in expression columns and correctly embed stringified JSON lists and scalar cells. #1099, #1059
- Multiprocessing reliability: Reset inherited OpenTelemetry metrics state and reseed shared sampler random generators in forked children. #1070, #1077
- Codex rollout data: Opt into raw reasoning content with
codex_reasoning_source="content_first", falling back to summaries. The default remains summary-only. #1063
Security
Require Pillow 13 and patched OpenTelemetry dependencies. Add maintained optional constraints for transitive dependency security floors and automated dependency audits. Refresh application lockfiles and existing environments when upgrading. #1091, #1074, #1093
Upgrade guidance
Python 3.10 is no longer supported. Upgrade to Python 3.11 or later first. #1075
python -m pip install --upgrade "data-designer==0.10.0"For standalone installations, upgrade data-designer-config or data-designer-engine to 0.10.0 instead.
New Contributors
Thanks to @v0ropaev, @namanraii, @suryapilla, @jiwasawa, and @edjson for their first contributions!
Full changelog: v0.9.7...v0.10.0