Skip to content

v0.5.1

Choose a tag to compare

@oliveigah oliveigah released this 13 Feb 04:05

Bug Fix: Producer record timestamps used seconds instead of milliseconds

Problem: The producer batcher set record batch timestamps (base_timestamp and max_timestamp) using DateTime.to_unix(DateTime.utc_now()), which defaults to seconds precision. Kafka expects these timestamps in milliseconds. As a result, the broker interpreted record timestamps as dates in January 1970, causing log retention to immediately delete segments that appeared to be ~56 years past their retention period.

This manifested as intermittent OffsetOutOfRange (error code 1) errors on consumers when the broker's retention check happened to run while consumers were actively reading. The non-deterministic nature of the bug depended on the timing alignment between the retention check interval and active consumption.

Fix: Changed DateTime.to_unix(DateTime.utc_now()) to DateTime.to_unix(DateTime.utc_now(), :millisecond) in the producer batcher, producing timestamps in the millisecond precision that Kafka expects.

Impact: All produced records had incorrect timestamps. Consumers could experience skipped records when the broker's log retention check deleted segments mid-consumption. The issue was more likely to surface under longer-running workloads or with shorter retention check intervals.