Repository navigation
Upgrade notice: v1.0.0 to v1.0.2 are retracted
Those versions can silently lose records, with no error, log line or metric. If
you use the engine or SDK, upgrade to v1.0.3.
- Window state overwritten after a restart (Pebble state backend). After a
restart, new records overwrote the oldest records stored for the same key. A
windowed job lost everything that was buffered in the window at the last
checkpoint. In our test that was 62,174 orders (about 6 seconds of input at
10,000 events/s) after killing the job. - Kafka watermark followed the fastest partition. When partitions are read at
different speeds (parallel reads, catch-up after a restart) the watermark ran
past records that slower partitions had not delivered, and the window operator
dropped them as late. 0.6% to 31% of each window was lost at 10,000 events/s.
go.mod now retracts [v1.0.0, v1.0.2], so go get ...@latest and go list -u
skip them. The Go module proxy keeps every published version permanently, which is
why the versions are retracted rather than replaced. The old GitHub releases and
tags were removed.
What changed
Fixes
state: the Pebble list append counter is seeded from the highest stored entry
instead of restarting at 0 after a restore.source: Kafka watermarks are tracked per partition and the job's watermark is
the minimum across partitions. Partitions that deliver nothing for 30 seconds
stop holding it back (KafkaPartitionIdleTimeout; negative disables). A
partition that has not delivered any record yet is not considered.window: records rejected as late are counted in
weibo_window_late_records_total{disposition="dropped"|"side_output"}. A
non-zerodroppedrate means the job is losing data to lateness.control:/readyzand/healthzfetch container stats concurrently and are
bounded to 5 seconds (was 7 to 8 seconds on a real host; still about 2 seconds).
The controller starts when the YAML runner image is missing and prints a warning,
so SDK-only hosts no longer need it.- Also since v1.0.0: an infrastructure failure no longer reports a job as
successful; a failed savepoint promotion fails the run; corrupted checkpoints are
logged instead of silently ignored;WatermarkSourcekeeps the wrapped source's
checkpoint capabilities; OOM-killed and paused containers are detected; one job's
store error no longer blocks reconciliation of the others; Postgres upsert with
duplicate keys in a batch and the shutdown flush are fixed; checkpoint duration
and size Prometheus metrics.
Docs
- Every documented procedure was followed on a fresh Ubuntu machine and the
references checked against the code. Corrections include the self-hosting guide
(installing the binary, the-dbdirectory, registry credentials for the systemd
service account) and the watermark section of Core Concepts. - New benchmarks and validation results.
- Emoji removed.
Verified
A Kafka to keyed window to S3 pipeline at 10,000 events/s (1,000 customers, 6
partitions) with the job killed mid-run: 4,414,010 orders, 7,000 of 7,000
(customer, window) totals exact, 0 records rejected as late. The previous release
lost data in the same test. Details, including what was not measured, are in
the benchmarks doc.
Limits worth knowing
- Validated for one scenario: Kafka with parallel reads, the Pebble backend and the
S3 sink, one job kill per run. Repeated restarts and a crash during a checkpoint
were not tested. Window(...).Reduce(...)emits a running total per record. For one row per
(key, window) useWindowReduce.NewKafkaSourcepanics on invalid configuration instead of returning an error.
Controller and CLI binaries are attached to
control/v1.0.3.