RustQueue 0.8.4

RustQueue 0.8.4

RustQueue 0.8.4 is a performance release for the durable Channel acknowledgement
path. It reduces the work held under the Topic state lock and removes several
per-message allocation points without changing the v7 storage or wire
contracts.

Durable Channel commit path

• FIN and REQ for one Topic are collected for up to 1 ms and committed in
  one bounded Channel WAL group of at most 1,024 requests.
• The Topic state lock is released while the Channel WAL files are synced. An
  independent Channel commit gate still serializes append, checkpoint,
  rotation, deletion and other durable Channel mutations around that boundary.
• A successful FIN or REQ response still follows every affected Channel WAL
  fsync. Delivery may reserve already durable messages while a later group is
  syncing, but never observes an uncommitted tail.

Hot-path allocation reductions

• Channel WAL commands use fixed-size stack encoding instead of allocating a
  temporary command buffer for each frame.
• TCP SUB, FIN and REQ paths carry shared Arc<str> Topic/Channel
  identities through the broker, and touched Channel sets retain those shared
  identities.
• Channel operation futures no longer add an extra Box allocation. Each TCP
  session has a bounded pool of 256 active Channel operations.
• The benchmark consumer waits for the server EOF and verifies final Channel
  depth, in-flight and deferred counts before accepting a drained result.

Compatibility and durability

• The on-disk format remains v7; no migration is required from 0.8.3.
• NSQ V2 commands, TLS/mTLS, AUTH, compression, lookup and the opt-in Kodo
  profile retain their existing compatibility contracts.
• Delivery remains at least once. Durable FIN/REQ acknowledgement semantics
  and the single-copy share-nothing PVC model are unchanged.

Qualification protocol

The qualification script defaults to comparing the exact v0.8.3 tag with the
candidate on OrbStack. The formal optional run uses fresh volumes, 2 vCPU and
2 GiB for both Broker and load generator, 10 alternating pairs for each of the
three cases, a 30-second warmup and a 120-second measurement. Short development
A/B runs are useful for iteration but are not formal release evidence.

Release assets

• rustqueue-0.8.4-linux-x86_64.tar.gz: Linux x86_64 binaries, Console UI and
  example configuration
• rustqueue-0.8.4-linux-aarch64.tar.gz: Linux ARM64 binaries, Console UI and
  example configuration
• rustqueue-0.8.4-source.tar.gz: source archive for the tagged commit
• rustqueue-0.8.4.tgz: Helm Chart
• SHA256SUMS-0.8.4: SHA-256 checksums for all downloadable assets

RustQueue 0.8.3

RustQueue 0.8.3

RustQueue 0.8.3 removes full in-flight deadline scans from the hot delivery
path while preserving the v7 storage format and durable acknowledgement
contract.

Deadline-indexed delivery

• A durable Channel now keeps its active delivery deadlines in an ordered
  index alongside its ID lookup. Expiry checks visit only due deliveries rather
  than scanning the entire in-flight set on every fetch.
• TCP sessions use the same pattern for their local delivery leases. The next
  timer comes directly from the earliest active lease; TOUCH, FIN, REQ,
  delivery handoff, completion, and disconnect keep the index consistent.
• FIN and REQ operations in the asynchronous Channel pipeline are removed
  from the session deadline index while they are pending, then restored only
  if the operation fails. A successful operation removes both the ID lookup
  and deadline entry.

This is most relevant with high RDY values, long message timeouts, slow
consumers, and many active consumer connections. It lowers avoidable CPU work
and Topic-lock pressure without changing duplicate, timeout, or at-least-once
semantics.

Topic fsync and delivery concurrency

• A publish group still serializes appends for one Topic and acknowledges only
  after the segment group has been fsynced.
• The active segment file is now synced after releasing the Topic state lock.
  A separate commit gate keeps append, rotation, retention, and management
  changes serialized until that durable boundary completes.
• Delivery reservation reads only through a durable position advanced after a
  successful sync. Consumers can therefore reserve earlier durable messages
  during the next group's fsync, but never observe the uncommitted tail.
• Prometheus now exposes publish and delivery Topic-lock wait/hold histograms
  so mixed PUB/consume contention can be measured directly.

Explicit publish durability modes

• queue.publish_ack_mode = "durable" remains the default and preserves
  acknowledgement-after-fsync semantics.
• queue.publish_ack_mode = "write_ack" returns PUB, MPUB, and DPUB success
  after append/write, but consumers see only through last_durable_position.
• queue.publish_ack_mode = "nsq_relaxed" returns after append/write and makes
  that tail immediately consumable, matching NSQ's relaxed diskqueue boundary
  more closely.
• Both relaxed modes sync at the first of 2,500 messages, 8 MiB, or 10 ms by
  default. A sync failure isolates the broker from further writes.
• Unsynced messages, bytes, last durable position, and sync lag are exposed in
  broker statistics and aggregate/bounded per-Topic Prometheus metrics.
• Recovery preserves non-overlapping position gaps when an acknowledged
  relaxed tail disappears, preventing an older durable Channel ACK from
  skipping a later message. Recovered gaps are derived from segment ranges,
  Topic metadata, and durable Channel high-water marks; they do not consume
  Channel depth or its bounded ACK window.

NSQ comparison

NSQ v1.3.0 tracks in-flight timeouts with a priority queue, which exposed the
previous RustQueue scan as a direct scheduler mismatch. RustQueue now has the
same asymptotic deadline lookup shape, while keeping its own token-checked
delivery model.

Important throughput results still require a durability label:

• RustQueue's default successful PUB, MPUB, and DPUB response follows a
  local segment group fsync. write_ack and nsq_relaxed are separately
  labelled weaker acknowledgement profiles. Successful FIN and REQ always
  cross a Channel WAL group fsync.
• NSQ's diskqueue returns a write response before its next scheduled sync.
  Therefore NSQ --sync-every=1 is not equal to RustQueue's
  acknowledgement-after-fsync contract, and NSQ's default memory queue can
  further widen a raw-throughput result.
• scripts/benchmark-compare.sh continues to run NSQ with
  --mem-queue-size=0 and emits strict and relaxed sync profiles separately.
  Consumer runs require complete unique delivery, no unexpected duplicates,
  and a drained Channel before a result is accepted.

docs/architecture/nsq-performance.md
records the concrete source-level differences and the next measurements worth
making before any durability trade-off.

Compatibility

• On-disk format remains v7; no migration is required from 0.8.2.
• NSQ V2 commands, TLS/mTLS, AUTH, compression, lookup, and the opt-in Kodo
  profile retain their existing compatibility contract.
• The Broker remains share-nothing: one Broker owns one durable RWO PVC and
  the release does not add replica-based message durability.

Assets

• rustqueue-0.8.3-linux-x86_64.tar.gz: Linux x86_64 binaries, Console UI and
  example configuration
• rustqueue-0.8.3-linux-aarch64.tar.gz: Linux ARM64 binaries, Console UI and
  example configuration
• rustqueue-0.8.3-source.tar.gz: source archive for the tagged commit
• rustqueue-0.8.3.tgz: Helm Chart
• SHA256SUMS-0.8.3: SHA-256 checksums for all assets

RustQueue 0.8.2

RustQueue 0.8.2

RustQueue 0.8.2 aligns no-Channel retention with NSQ, hardens the direct Broker
data path, and corrects premature Channel group commits. It completed short
OrbStack correctness and regression preflights, but the optional 60-run
performance qualification was not run and this release makes no formal
throughput claim.

No-Channel durability

• A Topic now persists the earliest position published while it has no durable
  Channel. Normal GC cannot advance past this boundary, even after the 90-second
  bootstrap window or a Broker restart.
• The first durable Channel consumes from that boundary. Deleting the last
  durable Channel starts a new boundary at the current Topic tail, so later
  acknowledged publishes remain available to its replacement.
• Existing v7 Topic manifests without the new optional field are reconciled
  from the earliest retained position during open. The disk format remains v7
  and no migration command is required.
• Later Channels created while another durable Channel already exists retain
  the existing 90-second bootstrap behavior.

Broker data path

• A Channel commit worker now keeps collecting FIN and REQ requests until
  the end of its existing 1 ms coalescing window. A transiently empty receiver
  no longer forces an early durable commit.
• The group remains bounded at 64 requests and the queue remains bounded at
  1,024 requests. Every successful operation still crosses the affected
  Channel WAL fsync before the client can observe completion.
• Publish acknowledgement semantics, at-least-once delivery and on-disk format
  v7 are unchanged.

Benchmark correctness

• Consumer shutdown preserves an NSQ frame that was only partly read when
  CLS begins. The benchmark can finish that frame without restarting the
  parser in the message body.
• A warmup with consumers is fully delivered and durably acknowledged before
  the measured Topic starts. Warmup backlog can no longer inflate the measured
  drain time or Broker RSS.
• A nonzero benchmark exit, missing delivery, unexpected duplicate, incomplete
  drain, or nonzero final Channel depth, in-flight count or deferred count is a
  hard failure before statistics are evaluated.

Benchmark protocol and release decision

The optional full qualification uses OrbStack only, with both Broker and load
generator fixed at 2 vCPU and 2 GiB. It compares the exact v0.8.1 tag with
one exact candidate commit using a fresh volume for every run.

Each case runs as 10 alternating pairs (A -> B, then B -> A), with a
30-second warmup and 120-second measurement. Consumer cases have a 1,800-second
upper bound for the mandatory complete drain; a timeout remains a hard failure:

• raw write ceiling: one Topic, no consumer, 16 producers, 1 KiB messages and
  MPUB batch 64;
• sustainable throughput: one Topic and one Channel, 16 producers, 16
  consumers, 1 KiB messages and MPUB batch 64, followed by a complete drain;
• low-load guard: one Topic and one Channel, 16 producers, 16 consumers, 1 KiB
  PUB, and a fixed total arrival rate of 100 messages/s.

The evaluator uses a deterministic paired bootstrap of the geometric mean
candidate/baseline ratio. A throughput regression is reported only when the
one-sided 95% upper bound is below 0.95. Low-load PUB ACK p99 or peak Broker
RSS regresses only when the one-sided 95% lower bound is above 1.10.
Saturation RSS remains in every run record but is descriptive because a faster
candidate completes more work in the same measurement window.

Raw JSON, stderr and RSS samples remain under the ignored
benchmarks/results/ directory. By release decision, RustQueue 0.8.2 does not
include the optional compact 60-run artifact. Release readiness instead
requires the non-Kubernetes production gate, native x86_64 and aarch64 builds,
package checksums, and the existing compatibility tests.

Compatibility

• The on-disk format remains v7; no data migration is required from 0.8.1.
• NSQ V2 commands, TLS/mTLS, AUTH, compression and the opt-in Kodo profile keep
  their 0.8.1 compatibility contract.
• Proxy, Operator, Console and Kodo routing are outside the performance change.

Assets

• rustqueue-0.8.2-linux-x86_64.tar.gz: Linux x86_64 binaries, Console UI and
  example configuration
• rustqueue-0.8.2-linux-aarch64.tar.gz: Linux ARM64 binaries, Console UI and
  example configuration
• rustqueue-0.8.2-source.tar.gz: source archive for the tagged commit
• rustqueue-0.8.2.tgz: Helm Chart
• SHA256SUMS-0.8.2: SHA-256 checksums for all assets

RustQueue 0.8.1

RustQueue 0.8.1

RustQueue 0.8.1 is a reliability patch for the real publish-to-delivery path.
It fixes benchmark accounting that could hide missing deliveries and hardens
the queue, storage, authentication, proxy and operational control planes found
during the follow-up P0/P1 review.

Highlights

• rustqueue-bench now uses isolated durable Topics and Channels, waits for
  consumers to become ready before publishing, counts unique and duplicate
  deliveries, reports publish and receive throughput separately, and exits
  unsuccessfully when verified delivery is incomplete.
• Per-delivery generation tokens prevent a stale FIN, REQ or TOUCH from
  mutating a later redelivery of the same message. Delivery visibility also
  covers output buffering and pending durable channel operations.
• Dead-letter moves are serialized as durable transactions. Cancellation,
  restart and concurrent requests cannot publish multiple DLQ copies or
  acknowledge the source before the target is durable.
• Payload and recovery-index workers keep path guards and memory reservations
  until blocking I/O completes. Storage corruption is marked unhealthy before
  a response can escape.
• AUTH response parsing and compiled regex state now share an explicit
  node-wide memory budget. Broker management bodies, Kodo Stats aggregation,
  proxy control bodies and backend error bodies are bounded as well.
• Invalid semaphore capacities and timer ranges are rejected during startup,
  avoiding runtime panics or unbounded waits from malformed configuration.

Compatibility

• The on-disk format remains v7; no data migration is required from 0.8.0.
• NSQ V2 commands, TLS/mTLS, AUTH, compression and the opt-in Kodo profile keep
  their 0.8.0 compatibility contract.
• Delivery remains at least once. A message can be redelivered after an
  ambiguous disconnect, but a stale acknowledgement can no longer settle that
  newer delivery.

Release gate

The v0.8.1 tag is published only after GitHub Actions completes the full
non-Kubernetes production gate, builds native Linux x86_64 and ARM64 bundles,
packages the tagged source and Helm Chart, and verifies every SHA-256 checksum.

Assets

• rustqueue-0.8.1-linux-x86_64.tar.gz: Linux x86_64 binaries, Console UI and
  example configuration
• rustqueue-0.8.1-linux-aarch64.tar.gz: Linux ARM64 binaries, Console UI and
  example configuration
• rustqueue-0.8.1-source.tar.gz: source archive for the tagged commit
• rustqueue-0.8.1.tgz: Helm Chart
• SHA256SUMS-0.8.1: SHA-256 checksums for all assets

RustQueue 0.8.0

RustQueue 0.8.0

RustQueue 0.8.0 adds a production-shaped Kodo compatibility profile entirely on the RustQueue side. Upstream Kodo does not need to be modified.

Highlights

• Discovery /nodes advertises three stable producer Gateway identities while /lookup continues to return the real Broker owners used by consumers.
• The Gateway terminates the NSQ producer protocol, routes only to publish-ready Brokers, and retries another Broker only after an explicit pre-commit rejection. Ambiguous post-write failures are returned without an automatic replay.
• The NSQ-compatible /stats response now includes the standard topic_name, channel_name, depth, client, and cumulative counter fields expected by Kodo.
• Topic and Channel message, requeue, and timeout counters are durable and monotonic across restart, empty, and eviction.
• Kodo's unauthenticated channel cleanup is disabled and fails closed. Administrative operations use RustQueue's authenticated native management flow.
• The stable protocol/storage maximum is now 100 MiB. The Kodo profile accepts and validates Kodo's exact 104,857,500-byte application maximum.
• Token rotation, drain and disk-pressure routing, rolling cutover, readiness fences, metrics, alerts, and Operator status handling are hardened.

Deployment and upgrade notes

• Kodo compatibility remains opt-in with queue.kodoCompatibility.enabled=true.
• The profile requires three schedulable Broker/Gateway nodes and an immutable image digest. Consult the Kubernetes operations guide for resource, network-policy, and cutover requirements.
• Existing clusters must first roll the 0.8.0 Broker image and wait for Ready=True before enabling the Kodo profile or publishing messages above the previous limit.
• After Gateway advertisement, restart every Kodo publisher and update producerRestartNonce. Broker maintenance remains blocked until the restart fence is confirmed.
• Automatic Kodo channel cleanup must remain disabled.
• Storage feature level 2 is a monotonic rollback floor after large records are written.

Validation

• complete Rust 1.88 release gate: format, locked all-feature check/test, Clippy, and release build
• Console typecheck/build and Helm lint/template
• all seven fuzz targets
• Go and Python NSQ compatibility in plain and secure modes
• unmodified real Kodo source parser/admin-flow replay
• exact 104,857,500-byte PUB/DPUB with one Gateway failover
• three-Broker acceptance: 3,239 expected and unique messages, zero missing, duplicates, or publish errors

Assets

• rustqueue-0.8.0-linux-aarch64.tar.gz: Linux ARM64 binaries and example configuration
• rustqueue-0.8.0-source.tar.gz: source archive
• rustqueue-0.8.0.tgz: Helm chart
• SHA256SUMS-0.8.0: SHA-256 checksums

RustQueue 0.7.2

RustQueue 0.7.2 is a reliability hardening release for the v7 share-nothing architecture.

Highlights

• Hardened TCP/HTTP admission, request limits, timeouts, and cancellation safety for large messages.
• Made ephemeral channel registration safe across disconnects, cancellation, and concurrent sessions.
• Added durable, bounded DLQ outbox recovery and fail-closed broker integrity checks.
• Tightened segment and batch recovery validation, including malformed filenames and non-contiguous batch identities.
• Prevented protective eviction from purging around retained segments.
• Added AUTH backend failover without allowing explicit 401/403 denials to fall through.
• Improved proxy discovery refresh, Operator rollout safety, and Console management retry semantics.
• Made the containerized format and Clippy gates self-contained on clean CI runners.
• Made the secure Go/Python compatibility matrix portable across Linux bind-mount permissions.
• Isolated compatibility fixtures from production PVC disk watermarks on constrained CI runners and added actionable container health diagnostics.

Validation

• Full Rust workspace tests with crash-injection features: 181 tests passed.
• cargo check, strict Clippy (-D warnings), formatting, Console UI checks/build, Helm lint/template: passed.
• Seven fuzz targets completed without crashes.
• Official Go and Python NSQ client core and secure compatibility matrices: passed.
• 20 MiB publish/recovery coverage: passed.

Large-scale 500-broker and long-duration soak tests are intentionally outside this patch release gate.

Assets

• rustqueue-0.7.2-linux-aarch64.tar.gz: Linux ARM64 binaries.
• rustqueue-0.7.2-source.tar.gz: source archive.
• rustqueue-0.7.2.tgz: Helm chart.
• SHA256SUMS-0.7.2: SHA-256 checksums.

RustQueue v0.7.1

Reliability hardening, bounded channel resources, registry recovery fencing, HTTP body timeouts, and protocol limit validation. ARM64 release artifacts.

RustQueue v0.7.0

RustQueue 0.7.0 production-candidate release for trusted internal Kubernetes deployments. Includes the ARM64 Linux binary bundle, source archive, Helm chart, and SHA-256 checksums. The share-nothing model uses one durable RWO PVC per Broker; messages are not replicated between Brokers. The binary bundle is linux-aarch64; build linux-x86_64 separately for amd64 nodes.