v0.8.0

Summary

This release transforms Dynamic DES from a real-time monitoring tool into a full-lifecycle simulation platform. v0.8.0 introduces high-performance historical data generation and native Data Lake egress, allowing users to fast-forward through years of simulation time or transform their live models into synchronized forecasting engines. By exporting schema-enforced Parquet or JSONL files natively to multi-cloud object storage, this release enables a seamless transition from live digital twin monitoring to massive synthetic dataset generation for Machine Learning and analytical backfilling.

🚀 Key Features & Improvements

• Historical Fast-Forwarding & Forecasting: Introduced support for Time Dilation using factor=0.0. Simulations now run as fast as your CPU allows, detaching logical simulation time from real-world wall-clock time for rapid data generation and future-state prediction.
• Native Data Lake Egress (PyArrow VFS): Added ParquetStorageEgress and JsonlStorageEgress connectors. Powered by PyArrow, these support automatic file rotation (UUID chunking), heavy compression, and lock-free writes directly to Local, AWS S3, Google Cloud Storage (GCS), Azure Blob, or SeaweedFS.
• Schema Drift Protection: Implemented strict schema inference and enforcement for columnar Parquet storage. The engine now caches the initial batch schema and automatically casts subsequent records to prevent pipeline-breaking data type changes.
• Dynamic Payload Routing (Multiplexing): Storage connectors now support a custom path_router callable. This allows users to dynamically split events, separate telemetry, or drop meaningless metrics on the fly before they are written to disk.
• Unified Event Buffering: Telemetry metrics now share the primary event buffer. This ensures that low-volume metrics are batched alongside lifecycle events, preventing the "tiny file problem" in storage buckets and reducing network overhead for Kafka.
• Resilient Teardown Mechanism: Upgraded the environment shutdown process with a 5-second queue-draining safety net. This guarantees that final events are flushed to disk or network before exit, preventing data loss without risking system deadlocks.
• Operational Observability: Introduced real-time heartbeat logging for background storage writers. Users can now monitor batch progress and queue depth during long-running historical batch runs.
• New Forecasting CLI Example: Added the ddes-history-example command to demonstrate a full week of factory simulation multiplexed and exported directly to chunked Parquet files in seconds.

Full Changelog: v0.7.0...v0.8.0