Skip to content

Releases: zendesk/sarama

v1.26.2-zd5: Remove idempotent production fix feature flag

24 Apr 00:05
ca14191

Choose a tag to compare

Add a functional test to cover idempotent production

v1.26.2-zd4: Add a feature toggle for the idempotent publish fix

16 Apr 08:08
642288b

Choose a tag to compare

Add a feature toggle for the broker fix.

v1.26.2-zd3: Fix "broker received out of order sequence" when brokers die

14 Apr 06:12
9df3038

Choose a tag to compare

When the following three conditions are satisfied, the producer code can
skip message sequence numbers and cause the broker to complain that the
sequences are out of order:

* config.Producer.Idempotent is set
* The producer loses, and then regains, its connection to a broker
* The client code continues to attempt to produce messages whilst the
broker is unavailable.

For every message the client attempted to send while the broker is
unavailable, the transaction manager sequence number will be
incremented, however these messages will eventually fail and return an
error to the caller. When the broker re-appears, and another message is
published, it's sequence number is higher than the last one the broker
remembered - the values that were attempted while it was down were never
seen. Thus, from the broker's perspective, it's seeing out-of-order
sequence numbers.

The fix to this has a few parts:

* Don't obtain a sequence number from the transaction manager until
we're sure we want to try publishing the message
* Affix the producer ID and epoch to the message once the sequence is
generated
* Increment the transaction manager epoch (and reset all sequence
numbers to zero) when we permenantly fail to publish a message. That
represents a sequence that the broker will never see, so the only safe
thing to do is to roll over the epoch number.
* Ensure we don't publish message sets that contain messages from
multiple transaction manager epochs.

v1.26.2-zd2: Fix session ID thrashing (take 2)

18 Mar 21:58
0c4ec76

Choose a tag to compare

Fix brokers continually allocating new Session IDs

KIP-227 says that if a client provides a session ID of 0 and a session
epoch of 0 in a FetchRequest, the broker should allocate a new
sessionID. Then, the consumer can use this ID to avoid repeating the
list of partitions to consume on every call.

Sarama is currently sending 0 for both values, but totally ignoring the
session ID generated by the broker. This means that the broker is
allocating a new session ID on every request, filling its session ID
cache and leading to FETCH_SESSION_ID_NOT_FOUND logs being written by
the broker because other _actual_ session IDs are evicted.

Instead, we should set the epoch to -1, which instructs the broker
not to allocate a new session ID.

v1.26.2-zd1: Fix session ID thrashing

18 Mar 05:10
d2343ed

Choose a tag to compare

Fix brokers continually allocating new Session IDs

KIP-227 says that if a client provides a session ID of 0 and a session
epoch of 0 in a FetchRequest, the broker should allocate a new
sessionID. Then, the consumer can use this ID to avoid repeating the
list of partitions to consume on every call.

Sarama is currently sending 0 for both values, but totally ignoring the
session ID generated by the broker. This means that the broker is
allocating a new session ID on every request, filling its session ID
cache and leading to FETCH_SESSION_ID_NOT_FOUND logs being written by
the broker because other _actual_ session IDs are evicted.

Instead, we should set the epoch to -1, which instructs the broker
not to allocate a new session ID.