Skip to content

Release v0.2.2

Choose a tag to compare

View all tags
@thanhtham010891 thanhtham010891 released this 03 Jun 10:13
· 5 commits to main since this release
v0.2.2
98a0ab2

Focus

Operational visibility for long-lived workers, safer partial-batch delivery,
and a more production-shaped Kafka worker example.

Highlights

Live worker observability for long-lived pipelines

WorkerPool now registers pipelines with the metrics collector before the first
run completes, so /health and /metrics can show long-lived consumers while
they are still running.

Live health payloads now expose:

  • registered pipeline metadata and schedule
  • running lifecycle state and active run id
  • live throughput and elapsed runtime
  • live records_consumed, records_written, records_dropped, records_errored
  • records_pending for in-flight records that have not reached a terminal outcome yet

This makes the health surface useful for continuous workers that may not finish
a run for hours or days.

Timed batch flushes for long-lived sources

DeliveryConfig now supports batch_flush_interval_ms for batched delivery.
This allows long-lived sources to flush partial sink batches after a bounded
wait, instead of holding records indefinitely until the batch fills or the run
ends.

The implementation uses a single-owner flush task for pending_writes, which
avoids duplicate delivery when timeout-based flushing and new arrivals happen
close together.

Live metrics no longer under-report buffered source counters

Live /health snapshots now include hot-path source counters that have not yet
been flushed into the main PipelineMetrics object. This fixes confusing
observability states where records_written could temporarily appear higher
than records_consumed during a live run.

Kafka worker example updated for long-lived operation

The etl-order-reliability example now models long-lived Kafka consumers more
closely:

  • consumer pipelines stay attached to their Kafka group instead of exiting
    after short chunked runs
  • the worker no longer auto-runs the sample producer
  • Kafka UI is included in the local Docker stack for topic, group, and lag inspection
  • the example uses timed batch flushing to avoid stuck partial batches at low traffic

Tests

Release validation includes:

  • worker observability and health response coverage for running pipelines
  • regression coverage for timed partial-batch flushes
  • example worker tests for the Kafka reliability example
  • full package CI from the source tree
Assets 2
Loading