Skip to content

v0.1.7

Latest
Latest

Choose a tag to compare

View all tags
@us3r us3r released this 25 Mar 14:58
v0.1.7
45d3b83

What's new

silentpulse-sdp: Streaming telemetry via foreachBatch

SDP decorators degrade to heartbeat-only for streaming DataFrames because collect() and groupBy() fail on streaming DFs. This release adds two new functions for full telemetry from streaming pipelines:

  • streaming_sink() - factory returning a foreachBatch handler with built-in telemetry
  • report_batch() - call inside a custom foreachBatch handler

Each micro-batch reports heartbeat + volume (always), completeness (if expected_columns set), and freshness (if max_delay_seconds set). Per-asset tracking via asset_column works the same as with batch decorators.

from silentpulse_sdp import streaming_sink

sink = streaming_sink(
    integration_point="zerobus-ingest",
    asset_column="host",
    min_rows=10,
    max_delay_seconds=600,
)

query = df.writeStream.foreachBatch(sink).start()

All exceptions are swallowed - telemetry failures never crash the pipeline.

Install

pip install "silentpulse-sdp @ git+https://github.com/silentpulse-io/silentpulse-databricks.git@v0.1.7#subdirectory=silentpulse-sdp"

Or download the wheel from this release and upload to a UC Volume.

Full Changelog: v0.1.6...v0.1.7

Assets 5
Loading