Skip to content

v1.0.3

Latest

Choose a tag to compare

@ASHUTOSH-SWAIN-GIT ASHUTOSH-SWAIN-GIT released this 23 Sep 20:06
8d2f748

Upgrade notice: v1.0.0 to v1.0.2 are retracted

Those versions can silently lose records, with no error, log line or metric. If
you use the engine or SDK, upgrade to v1.0.3.

  • Window state overwritten after a restart (Pebble state backend). After a
    restart, new records overwrote the oldest records stored for the same key. A
    windowed job lost everything that was buffered in the window at the last
    checkpoint. In our test that was 62,174 orders (about 6 seconds of input at
    10,000 events/s) after killing the job.
  • Kafka watermark followed the fastest partition. When partitions are read at
    different speeds (parallel reads, catch-up after a restart) the watermark ran
    past records that slower partitions had not delivered, and the window operator
    dropped them as late. 0.6% to 31% of each window was lost at 10,000 events/s.

go.mod now retracts [v1.0.0, v1.0.2], so go get ...@latest and go list -u
skip them. The Go module proxy keeps every published version permanently, which is
why the versions are retracted rather than replaced. The old GitHub releases and
tags were removed.

What changed

Fixes

  • state: the Pebble list append counter is seeded from the highest stored entry
    instead of restarting at 0 after a restore.
  • source: Kafka watermarks are tracked per partition and the job's watermark is
    the minimum across partitions. Partitions that deliver nothing for 30 seconds
    stop holding it back (KafkaPartitionIdleTimeout; negative disables). A
    partition that has not delivered any record yet is not considered.
  • window: records rejected as late are counted in
    weibo_window_late_records_total{disposition="dropped"|"side_output"}. A
    non-zero dropped rate means the job is losing data to lateness.
  • control: /readyz and /healthz fetch container stats concurrently and are
    bounded to 5 seconds (was 7 to 8 seconds on a real host; still about 2 seconds).
    The controller starts when the YAML runner image is missing and prints a warning,
    so SDK-only hosts no longer need it.
  • Also since v1.0.0: an infrastructure failure no longer reports a job as
    successful; a failed savepoint promotion fails the run; corrupted checkpoints are
    logged instead of silently ignored; WatermarkSource keeps the wrapped source's
    checkpoint capabilities; OOM-killed and paused containers are detected; one job's
    store error no longer blocks reconciliation of the others; Postgres upsert with
    duplicate keys in a batch and the shutdown flush are fixed; checkpoint duration
    and size Prometheus metrics.

Docs

  • Every documented procedure was followed on a fresh Ubuntu machine and the
    references checked against the code. Corrections include the self-hosting guide
    (installing the binary, the -db directory, registry credentials for the systemd
    service account) and the watermark section of Core Concepts.
  • New benchmarks and validation results.
  • Emoji removed.

Verified

A Kafka to keyed window to S3 pipeline at 10,000 events/s (1,000 customers, 6
partitions) with the job killed mid-run: 4,414,010 orders, 7,000 of 7,000
(customer, window) totals exact, 0 records rejected as late. The previous release
lost data in the same test. Details, including what was not measured, are in
the benchmarks doc.

Limits worth knowing

  • Validated for one scenario: Kafka with parallel reads, the Pebble backend and the
    S3 sink, one job kill per run. Repeated restarts and a crash during a checkpoint
    were not tested.
  • Window(...).Reduce(...) emits a running total per record. For one row per
    (key, window) use WindowReduce.
  • NewKafkaSource panics on invalid configuration instead of returning an error.

Controller and CLI binaries are attached to
control/v1.0.3.