1.26.0
Added:
- New Heartbeat Sensors: we now have the ability to use heartbeat sensors in the engine. You can check the documentation on GitHub. The Heartbeat Sensor is a robust, configurable system designed to continuously monitor upstream systems for new data. It enhances the existing sensor infrastructure by addressing key limitations and providing significant improvements:
- Previous Sensor Architecture Limitations:
- Required individual sensor configurations for each data source.
- Limited scalability when monitoring multiple upstream systems.
- Manual job triggering and dependency management.
- No centralized control or monitoring of sensor status.
- Difficult to manage complex multi-source dependencies.
- Heartbeat Sensor Enhancements:
- Centralized Management: Single control table to manage all sensor sources and their dependencies.
- Automated Job Orchestration: Automatically triggers downstream Databricks jobs when new data is detected.
- Multi-Source Support: Handles diverse source types (SAP, Kafka, Delta Tables, Manual Uploads, Trigger Files) in one unified system.
- Dependency Management: Built-in hard/soft dependency validation before triggering jobs.
- Scalable Architecture: Efficiently processes multiple sensors in parallel.
- Status Tracking: Comprehensive lifecycle tracking from detection to job completion.
- Previous Sensor Architecture Limitations:
This provides a centralized, efficient, and automated mechanism to detect and trigger downstream workflows with minimal user intervention.
Fixed:
- Writing to result sink is failing when a different type appears in the observed_value column