v2.6.0

Queen MQ 2.6.0: the image is ghcr.io/queen-mq/queen:2.6.0 (linux/amd64 and linux/arm64), and :latest points at it. The clients are 2.6.0 too: JavaScript, Python and Rust on npm, PyPI and crates.io, Go and C++ tagged 2.6.0, and the PHP client 2.6.0 on Packagist.

The Kafka facade is a translation layer onto Queen, rewritten from scratch. The first facade
kept its own registry, fences, producer windows, group coordinator, transaction stages and config
mirror beside the broker. 2.6's maps each Kafka request onto what Queen already does, inside the
broker process: a produce is a push, a fetch reads at an offset without a lease, a committed offset
is the Queen consumer group's position. In a cluster the raft leader leads every partition and
coordinates every group and transaction, and the other nodes point clients at it. A group's
membership lives in the leader's memory: after a leader change the members rejoin and resume from
their committed offsets, and a group with no members leaves that memory ten minutes after its last
member did. An idempotent producer's records carry the transaction id k<producer id>.<epoch>.<sequence>,
so a batch sent again inside the topic's dedup window writes nothing: topics the facade creates get a
window of 15 minutes (QUEEN_KAFKA_DEDUP_WINDOW_SECONDS). A transaction is held on the leader and
committed as one Queen transaction, its pushes and its offsets together. CreateTopics applies
retention.ms and cleanup.policy to the topic's queue, and refuses a value it cannot apply with
INVALID_CONFIG. The leader writes each group's members to the key-value store every 15 seconds and
at once when they change, which is how the broker's group listing and the console see Kafka groups.
Measured against 2.5.0's facade on three 24-vCPU nodes (2026-10-20): both carry 500,000 msg/s of
1 KB records (2.6 end-to-end p50/p99 10.1/15.0 ms); at 1,000,000 msg/s 2.6 pushes 30% less, because
it takes produce on the leader only, where 2.5.0's followers relayed theirs. Not offered in 2.6:
AlterConfigs, IncrementalAlterConfigs, OffsetDelete, DescribeLogDirs, the ACL calls and a TLS
listener (terminate TLS in front of the Kafka port). The standalone facade binary is gone, and so are
the settings only the first facade read: QUEEN_KAFKA_PIPELINE, QUEEN_KAFKA_RELAY,
QUEEN_KAFKA_COALESCE, QUEEN_KAFKA_OFFSET_STORE, QUEEN_KAFKA_NODE_ID, QUEEN_KAFKA_CLUSTER,
QUEEN_KAFKA_CLUSTER_HEARTBEAT_MS, QUEEN_KAFKA_CLUSTER_TTL_MS, QUEEN_KAFKA_MAX_PARTITIONS,
QUEEN_KAFKA_MAX_PRODUCER_STATES, QUEEN_KAFKA_GROUP_MIN_SESSION_TIMEOUT_MS,
QUEEN_KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS, the QUEEN_KAFKA_TXN_MAX_* limits,
QUEEN_KAFKA_TLS_KEY and QUEEN_KAFKA_FORWARD_SNI_HOST. New: QUEEN_KAFKA_AUTO_CREATE,
QUEEN_KAFKA_DEDUP_WINDOW_SECONDS, QUEEN_KAFKA_MAX_REQUEST_BYTES and QUEEN_KAFKA_PEERS.
Topics and their records are queues and do not move; during the upgrade Kafka clients reconnect,
groups rejoin and transactions open on the old facade abort.

A member that comes back recovers from the leader's log or from a delta, not from the whole
state. The leader keeps its log for a member that is away (QUEEN_RAFT_MEMBER_HOLD_S, 6 hours by
default, ended sooner by QUEEN_RAFT_MEMBER_HOLD_DISK_PCT, the disk's low mark), so a member that
restarts, or is moved by a node upgrade, catches up from the log. When the log no longer reaches it,
the leader first asks where the member's log stands (POST /raft/v1/snapshot/basis) and, with the
same queue-log layout, sends the store and only the records above that point, which the member
appends byte for byte. A snapshot carries the sealed files' indexes, so the receiver no longer reads
every file to rebuild them, and a boot maps the indexes without reading them and checks them in the
background. The leader builds nothing for a member that is down, refuses at once a member that is
still starting instead of waiting on it, and backs off its reconnects (50 ms doubling to 1 s) instead
of opening about 4,000 a second. On three GCP nodes holding 521 GB each (2026-10-20), a member away
for five minutes came back with a 1.14 GB delta, appended in 1.3 s, and was healthy 28 s after it
started, where 2.5.0 sent it the whole state (566 GB, 15 minutes). On stage under a Kafka load
(2026-10-21), a killed leader's cluster was whole again in 47 s (141 s on 2.5.1), and none of six
restart and kill tests needed a snapshot.

A restart refills the dedup rows from a sidecar per file, not from the files. On those nodes
nine tenths of a restart went into reading the queue logs for the hash lists behind the dedup rows:
310 GB read in 300 s for lists under a gigabyte. Every sealed file now gets a sidecar, r<id>.qhsh
(each message record's offset, partition, base, count, stamp and hashes, with checksums), built off
the log writer's path and written in steps it waits for; a boot reads a sidecar that is whole and
matches its file, and the file otherwise. Paired cold restarts of 254 GB nodes (2026-10-21): the
refill took 20 s reading 0.7 to 1.4 GB with sidecars, against 171 to 201 s reading 185 to 217 GB
without, and the node was healthy after 30 to 33 s against 186 to 240 s. The sidecars cost about
16 bytes a message on disk (147 to 163 in the 3,000,000 msg/s benchmark), and latencies are 2.5.0's.
QUEEN_QLOG_HASH_SIDECARS=0 turns them off. The first boot of 2.6.0 still reads every sealed file
once, and writes their sidecars.

Raft: three fixes found while testing the recovery above. A node binds its raft port at the start
of its boot and listens only once its raft core exists, and a port another process already holds
fails the open instead of leaving the node up and deaf. A member's 404 to the snapshot question is
read from the answer's status: a member that refuses it is a node before 2.6 and gets the whole state.
A replication stream that asks for entries the leader purged since it chose them now sends a heartbeat
instead of entries that do not follow their prev_log_id; only a member back with an empty data
directory could start that race, and in a release build the member refused the request.

Empty partitions can go within the hour: PARTITION_CLEANUP_SECONDS. PARTITION_CLEANUP_DAYS
counts whole days, at least one, so a workload that opens a partition per entity kept a day of empty
ones in memory and in every retention round. PARTITION_CLEANUP_SECONDS, when set, replaces it (at
least 60), and whatever the setting a partition is never deleted inside its queue's dedup window, so
a repeated push is still recognized. The cleanup's check for a stream query's state seeks each query
instead of reading all of them, which a short age makes matter. On a production cluster with one
partition per trace, 3600 took the partitions from 520,857 to 151,120 within 40 s of the first 2.6
leader.

A queue log no longer fills its disk with raft entries nothing needs. The queue logs are the
write-ahead log: each entry is written to the logs of the queues it touches. A sealed file was
rewritten once half of its message bytes were dead, and its entries did not count. In production a
stream query wrote its whole state, up to 15 MB, on every commit (2026-10-22): one shared log
(QUEEN_QLOG_SHARDS) grew about 40 GB an hour, most of each file was those entries, and the few
messages beside them, of a queue kept 2 days, held every file for days. The entries now count as
dead: a file reclaim may touch is below the point recovery and the cluster need, and a rewrite
keeps only messages. And while the last reclaim pass left work, a pass gets
QUEEN_QLOG_RECLAIM_BOOST (4) times its budget (256 files, 200 ms, one rewrite), rewrites
included; passes go back to the small budget once one finds nothing left. Judging and copying stay
off the log's write lock, so appends do not wait for them.

A queue's status can be read without its partitions. GET /api/v1/status/queues/:queue with
includePartitions=false answers the queue's totals, its partition count and consumed total
included, without per-partition rows or the dead-letter scan; the default answer is unchanged. The
console's queue page asks for it on large queues and draws aggregate seeds, so a queue with 100,000
partitions no longer sends 44 MB to every refresh (835 bytes with the summary). From
#141 by albertovincenzi.

Every SDK reads a partition by offset and lists the partitions written since a time. Two queue
methods wrap POST /api/v1/fetch and POST /api/v1/partitions/changed, which only the Kafka facade,
the S3 sink and hand-written HTTP calls used until now. fetch(partitions) (Go Fetch) reads every
record of the named partitions, from the oldest one still kept or from given offsets, up to where each
partition ended when the call first read it: 1000 partitions per request, asked again until each one is
read. Each partition answers its records with their offsets, transaction ids, payloads and push times,
its high watermark, its log start, and the offset to start from next time. changedSince(since)
(changed_since in Python, Rust and C++, ChangedSince in Go) lists the partitions written at or after
a time, every page of them, in creation order, with the answer's safeTime to pass as the next
since. Neither takes a lease, moves a cursor or needs an ack. A loop that reads a partition from
offset 0 by hand reads nothing once retention has moved the partition's start, because the broker
answers an offset below logStartOffset with OFFSET_OUT_OF_RANGE and no records; fetch goes on
from logStartOffset. The methods ship in the 2.6.0 clients (all six, PHP included). Any 2.x broker
serves both routes; one before 2.3.0 ignores the byte ceiling of a request.

Server: the consumer-group listing reports a Kafka group by its members.
GET /api/v1/consumer-groups gives a group the Kafka facade's rows name kind: "kafka", and every
row three new fields, null on a native group: liveMembers, lastSeenAt and inactiveSeconds (0
while the queue is read for the group, else the seconds since it was last seen read). With presence, a
Kafka group's state is Stable or Lagging (past 300 s of lag) while members read the queue and
Empty when none does; a group whose members died is Empty three intervals after its last
presence write, judged on the broker's clock. A Kafka group without presence (a facade before
2.6) is reported as before. The group detail and the lagging list carry the same kind,
and GET /api/v1/resources/queues marks a queue the facade created with kafkaTopic: true. The
broker reads the facade's rows when it answers: no replicated row or command changes, and a native
group's row only gains the three null fields.

Server: retryDelay is enforced. After a failed ack that leaves retries, the message comes
back once the queue's retryDelay (milliseconds) has passed, and its partition waits with it for
that consumer group, so nothing behind the message overtakes it; a pop that names the partition
waits too. Every version up to 2.5.0 stored the option, echoed it, and redelivered at once. A queue
created by a push, or by /configure without the option, stores 1000: after the upgrade a failed
message comes back about a second later there, and retryDelay: 0 keeps the old behaviour. A
retry ack, an expired lease and a dead-lettered message do not wait, and a failed ack inside a
transaction waits from the commit. The wait lives in the leader's memory: after a leader change the
message comes back at once. The console's queue editor now offers the option beside the retry
limit, and the queue page no longer marks it as not enforced.

Server: an ephemeral pop no longer sleeps through a push that lands as it parks. A waiting
/api/v1/ephemeral/pop looked at its partitions and only then registered for a wake, so a push
landing between the two woke nothing and the pop answered at its own timeout, with the message
there all along. On 2.5.0, with 16 requesters and a new inbox per request, about one reply in 950
came back at the pop's 2 s timeout; with one inbox per requester, one in 28,000. The pop now
registers before it looks. Durable pops wait in the consumption engine and were not affected.

Console: a queue read by Kafka consumers shows its reads, and a chart missing a node says so. The
records the in-process Kafka facade serves to Fetch requests were counted nowhere, so a topic read
only by Kafka consumer groups showed pushes and no consumption on the Overview, and Workload showed
nothing for Kafka topics at all. Each node now counts the records its facade's fetches return, per
queue and minute, beside the pops it already counted: /api/v1/analytics/queue-ops answers them as
fetchMessages and fetchPerSecond, /api/v1/analytics/workload as window.fetchMessages and
series.fetch, and the console charts a "Fetch (Kafka)" line on the Overview and a "Fetched (Kafka)"
figure in Workload. Each record a fetch returns also counts as a pop, with its lag, and a commit that
moves a group past N records counts N acks and adds N to the group's consumed total, so two Kafka
groups reading one topic count its records twice, as two native groups' pops do. Dashboard rows are kept by the node that
served the traffic and gathered from the others on every read; a node that did not answer within the
gather's 1.5 s, as during a rolling restart, used to leave its rows out without a word, and on stage
(2026-10-19) the Overview read about 1 push a second instead of 1,500 while the leader restarted.
Gathered answers now name the nodes that did not answer in nodesMissing, the node logs it at most
once a minute per peer, and the console shows a "Partial" chip naming them.

Console: a Kafka group is a reader while it has members, and inactive after a grace. A Kafka
group with live members counts as its topic's reader and is judged by its lag like any group. One
with no member is still judged by its lag for 15 minutes (a restart, a rebalance, a crash not yet
called), then shown as inactive on Consumer groups, with when it was last seen, and no longer
reported as an issue; Settings moves the grace (Inactive after) for the tenant or one queue.
A queue is flagged "No reader" only when no consumer group is on it and it is not a topic the Kafka
facade created: on a queue with groups, each group answers for itself. "No reader is expected"
still silences a queue that holds messages nobody is meant to read.

Console: Workload opens on a broker with no queues. A tenant without queues is "tenant": {} in
the workload answer, and the page stopped on "Interface error: Cannot read properties of undefined
(reading 'groups')" on every fresh broker. It now shows zeros until the first queue exists.
Reported in #140.

Console: the time filters offer 12 hours, and sign-in recovers from a Google error. From
#139 by albertovincenzi. A Kafka record is shown decoded in the message drawer, and Kafka groups with no
members leave the Overview's lag panel and Workload's lag figures.

The cluster version stays 7. A cluster rolls from 2.5.0 to 2.6.0 one node at a time, and back
on the same data, with one condition for the way back: a node holding a snapshot it has not swapped
in yet (a snapshot.pending or snapshot.swapping that names a basis) boots 2.6.0 once more first,
because 2.5.0 would take that delta for the whole state. 2.5.0 ignores the sidecars and
raft/member_holds.json that 2.6.0 writes. Tested on three GCP VMs under load, with deduplication
on, and the full Jepsen campaign ran on 2.6.0's code: 257 tests, all valid
(benchmark-queen/2026-10-21-jepsen-26).