Skip to content

v2.7.1

Latest

Choose a tag to compare

@alice-viola alice-viola released this 03 Nov 10:03

Queen MQ 2.7.1: the image is ghcr.io/queen-mq/queen:2.7.1 (linux/amd64 and linux/arm64), and :latest points at it. The C++ client is 2.7.1 (tag clients/client-cpp/v2.7.1; with CMake, GIT_TAG v2.7.1), and queenctl 2.7.1 has its own release, queenctl 2.7.1. The JavaScript, Python, Go, Rust and PHP clients stay at 2.7.0.

Upgrading from 2.7.0: a rolling restart, nothing else. Nothing on disk or on the wire changes
(cluster version 7), so 2.7.0 and 2.7.1 nodes run side by side during the roll. The leader exports
five more Prometheus families, the consumer group lag below; on a cluster whose groups hold more
than 200,000 partition positions in all, they stay off (queen_consumer_group_lag_complete 0)
until QUEEN_METRICS_GROUP_LAG_MAX_CURSORS is raised. The C++ client 2.7.1 changes its API (its
README maps each 2.7.0 call to its new form); the JavaScript, Python, Go, Rust and PHP clients stay
at 2.7.0, and queenctl moves to 2.7.1 for dlq redrive (next section).

Console: Refresh now, the Raft term and the store on Cluster, Expand all, and pages that fit a
phone.
The band has a Refresh now button beside the time window (or press R): every read on the
page runs again at once, past the 25 to 60 s a read is kept, where before only the 30 s tick
refreshed a page. Cluster says which node leads in which term, and a Log and store table lists each
node's last applied entry, its Raft log (size and files), how much of its store's map is used (from
85% a node refuses writes) and its readers; the charts add the log's size and the map's use. Queues
and Consumer groups have Expand all and Collapse all for the namespaces. Workload's switch is
Namespace or Task, and the whole tenant is the first two lines of its list, All namespaces (or All
tasks) and All queues: the Queue option they replace showed the same list as Namespace, and the same
rows once a namespace was picked. In the Tone look, the Overview's cards and a hovered or picked
row are rounded like every other area. On a phone, the account, time window, cluster and theme
panels open inside the screen, the band keeps the same two rows on every page, the rail fits without
scrolling (Settings moves to the account menu), and the Cluster, Streams and queue tables show the
columns that fit. Why: there was no way to ask for fresh numbers now, the term and the store were a
click per node away (the 2.6 System page showed them), and on a phone the account menu, so Sign
out, opened off the screen.

Consumer group lag in Prometheus, with a Grafana board and alert rules (#146).
At every metrics bucket (METRICS_FLUSH_MS, 60 s) the raft leader exports, per queue and consumer
group, queen_consumer_group_lag_messages (what the group has not consumed),
queen_consumer_group_lag_seconds (how old the oldest of those is) and
queen_consumer_group_partitions_behind. A group counts on every partition of its queue from its
subscription start, also on a partition it has never read, so a new entity's messages while every
consumer is down are lag, and messages from before the group subscribed are not (the consumer
groups list counts only the partitions a group has read from). One pass over every group's
position per bucket, capped by QUEEN_METRICS_GROUP_LAG_MAX_CURSORS (200,000; 0 turns it off):
past it the leader exports queen_consumer_group_lag_complete 0 and no per-group series rather
than a part. deploy/grafana/queen-consumer-lag.json is a board on these series and
deploy/prometheus/queen-consumer-lag.rules.yml three alerts, at the console's lines (one minute
needs you, five minutes failing). Why: teams coming from Kafka watch consumer lag every day, and
until now nothing could alert on a group's backlog.

The console replays a partition's dead letters in order (#147). Replay on Dead
letters sends each partition's rows oldest first, one at a time, with partitions side by side; it
sent four rows at a time, so one partition's replays could land out of order. queenctl dlq redrive (queenctl 2.7.1, next section) replays the same way, many rows at once. Why: a partition
is one entity's order, and a replay that shuffles it breaks what the queue promised.

The console lists the PostgreSQL connectors (#155). A Connectors page, after
Streams in the rail: every source and sink with its state (Streaming, Snapshot, Running, Standby
and the node that runs it, Failing), what it moves where, what it has moved, a source's slot lag
and its last commit; a row opens its error and what it says, its slot and pointer or its group and
mode, and its document as stored, never the password. Read-only, from GET /api/v1/connectors;
behind the proxy it needs the admin role. Why: a connector was visible only through the API.

RustFS for the S3 sink (#144). The sink's tests against a real S3 server pass on
RustFS 1.0.4, and deploy/compose/rustfs/compose.yaml starts RustFS, creates a bucket and copies
a queue into it as Parquet; the S3 page says how, and what RustFS's S3 Tables (Iceberg) are not.
No code changed. Why: teams moving off MinIO asked whether Queen's lake works on RustFS.

C++ client 2.7.1: the API of the Go and Rust clients, a connection pool, and HTTPS. A
breaking change: the C++ client now has the shape of the other SDKs. queen::Queen (QueenClient
stays as an alias) hands out typed values: pop() returns std::vector<Message>, push() takes
one payload (push(payload, transactionId) with the dedup key, push_many, push_items) and
answers one PushResult per message, and ack(message) reads the consumer group and the lease
from the message and answers an AckResult, with nack, ack_with(AckStatus::Retry | Dlq) and
ack_all. consume() hands over one message per call and consume_batch() the whole batch;
limit() counts across the workers, concurrency(n) runs n threads (it was capped at the number
of cores), a Cancel stops the loop, and it returns a ConsumeSummary. A transaction pushes to a
partition and answers a TransactionResult. New: admin() (seek a group, retry a dead letter, the
queue and group routes), every QueueConfig option (only the ones set are sent), KV get_many and
get_prefix, timer peek and list, the affinity load balancing of the other SDKs (now the
default), and locks that renew in the background (is_lost()). Fixed: an https:// URL was sent
as plain HTTP, bearer token included; every request opened a new TCP connection (now a keep-alive
pool: half the time per call over a 1 ms network, and no accept and close per request on the
broker); a timeout under one second failed every request; pop() and the dead-letter reader
answered an empty result for any failure; and every client installed SIGINT and SIGTERM handlers
that exited without flushing or ignored SIGTERM (now opt-in, ClientConfig::handle_signals).
nlohmann/json is vendored in clients/client-cpp/third_party/, clients/server/ is gone, and the
CMake target queen::client finds or fetches cpp-httplib. The client's README lists every change.
Why: the C++ client was the one SDK whose programs called raw routes for common steps, and it could
not reach an HTTPS endpoint at all.

queenctl 2.7.1

queenctl dlq redrive: replay many dead letters with one command (#147). It takes
the filters of dlq drain (--queue, --cg, --partition, --from, --to), reads every
match first, then replays each row by its id, as the console's Replay does: each partition's
messages go back oldest first, one at a time, so they keep their order among themselves, and
partitions run side by side (--parallel, 4). --dry-run prints the plan, --yes is required,
and it refuses more than --max rows (10,000). A second run replays only what the first left
(each replay is dlq:<row id>), a message several groups dead-lettered needs --cg (one replay
reaches every group), and it stops at the first refusal that would repeat (401, 403, 429).
queenctl 2.7.1 is built against Go client 2.6.1, like 2.6.1, and its archives for Linux and macOS
(x86_64 and arm64) and Windows (x86_64) are on the GitHub release
queenctl 2.7.1. Why:
after an incident a team replays hundreds of messages, and dlq retry takes one; the reason given
for leaving bulk replay out (the route was not idempotent) no longer held.