v0.1.7

What's new

silentpulse-sdp: Streaming telemetry via foreachBatch

SDP decorators degrade to heartbeat-only for streaming DataFrames because collect() and groupBy() fail on streaming DFs. This release adds two new functions for full telemetry from streaming pipelines:

• streaming_sink() - factory returning a foreachBatch handler with built-in telemetry
• report_batch() - call inside a custom foreachBatch handler

Each micro-batch reports heartbeat + volume (always), completeness (if expected_columns set), and freshness (if max_delay_seconds set). Per-asset tracking via asset_column works the same as with batch decorators.

from silentpulse_sdp import streaming_sink

sink = streaming_sink(
    integration_point="zerobus-ingest",
    asset_column="host",
    min_rows=10,
    max_delay_seconds=600,
)

query = df.writeStream.foreachBatch(sink).start()

All exceptions are swallowed - telemetry failures never crash the pipeline.

Install

pip install "silentpulse-sdp @ git+https://github.com/silentpulse-io/silentpulse-databricks.git@v0.1.7#subdirectory=silentpulse-sdp"

Or download the wheel from this release and upload to a UC Volume.

Full Changelog: v0.1.6...v0.1.7

v0.1.6

Full Changelog: v0.1.5...v0.1.6

v0.1.5

Full Changelog: v0.1.4...v0.1.5

v0.1.4

Full Changelog: v0.1.3...v0.1.4

v0.1.3

Full Changelog: v0.1.2...v0.1.3

v0.1.2

Full Changelog: v0.1.1...v0.1.2

v0.1.1

Full Changelog: v0.1.0...v0.1.1

v0.1.0

Full Changelog: https://github.com/silentpulse-io/silentpulse-databricks/commits/v0.1.0