[DISCUSS] Release Apache Iggy v0.10.0

[hubcio]

Hello, Apache Iggy Community,

This is a call to discuss the release of Apache Iggy version 0.10.0.

We first planned this release as 0.9.1, and master builds 0.9.1 edge versions today. We now propose 0.10.0 instead. Three large features landed since 0.9.0: multi-leader partitions, deferred polling and the Kafka gateway. The release also breaks the wire protocol, the data directory format and several public SDK APIs, and despite being Rust project we'd rather not do that ;)

As usual, we will also release the accompanying SDKs and tooling (CLI, Connectors, MCP server and Web UI) from the same repository. The Kafka gateway joins them for the first time, with the apache/iggy-kafka-gateway Docker image (#4593).

Below is the full changelog since 0.9.0 (243 commits from 44 contributors, plus dependabot).

Before we start the vote, we want to merge a few outstanding pull requests:

• Kafka gateway: test(cluster): test the Kafka gateway and the SDK on split primaries #4642, fix(gateways): stop Kafka Produce sends at the request deadline #4643, fix(gateways): survive Kafka accept errors and a late Iggy at startup #4645, feat(gateways): report Kafka group members' client id and host #4651 and feat(gateways): list every Kafka gateway and dial other Iggy nodes #4662.
• Deferred polling in the other SDKs, and a request that fetches several partitions in one call. The pull request is not open yet.
• Server fixes: fix(cluster)!: let slow replicas catch up under load #4658 and fix(configs): reject clusters exceeding VSR replica limit #4661.

Highlights

Multi-leader partitions

In 0.9.0, each partition took its primary (the node that orders its writes) from the metadata view at creation. So one node usually took the writes of all partitions. In 0.10.0, new partitions spread their primaries over the nodes. The metadata primary also moves existing primaries one at a time to keep the spread even, for example after a failed node returns (#4619).

Clients look up the primary of each partition and send partition requests straight to it. HTTP forwards a request to the primary in one hop (#4506). The Rust, Go, C#, Java and Node.js SDKs route natively, and Python, C++ and PHP route through the Rust SDK (#4561). The groundwork made retry protection and shared sessions durable (#4387) and fenced changes of partition ownership and history (#4456).

Deferred (long) polling

A consumer no longer has to poll a quiet partition in a loop. With deferred polling, the server holds a poll until messages arrive or the wait ends. Wait times, reply sizes and capacity are bounded (#4629). This is the long polling that discussion #2854 asked for.

The Rust high-level consumer and the connector sinks use deferred polls by default. The Java SDK supports them over TCP and HTTP, and the HTTP API serves them at GET /streams/{stream_id}/topics/{topic_id}/messages/deferred. The other SDKs follow in the outstanding pull request above.

Kafka gateway

The Kafka gateway (iggy-gateway-kafka) is a separate service. It speaks the Kafka wire protocol and stores the data in Iggy. 0.9.0 shipped only its listener, which answered every request with a placeholder (#3519). In 0.10.0, Kafka producers, consumers and admin tools work through it:

• Produce, Fetch and ListOffsets, also by timestamp (feat(gateways): send Kafka records to Iggy from the Produce handler #4282, feat(gateways): answer Kafka Fetch with records from Iggy #4336, feat(gateways): Implement ListOffsets Kafka Keys backed by Iggy bridge #4259, feat(gateways): answer Kafka ListOffsets by timestamp and MAX_TIMESTAMP #4545).
• Create and delete topics, add partitions and change retention.ms (feat(gateways): Implement CreateTopics and Metadata Kafka Keys backed by the Iggy bridge #4258, feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only #4400, feat(gateways): serve Kafka CreatePartitions through the Iggy bridge #4565, feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs #4398, feat(gateways): serve Kafka IncrementalAlterConfigs (key 44) #4572).
• Consumer groups with rebalances, static membership and offsets stored in Iggy (feat(gateways): coordinate Kafka consumer groups #4263, feat(gateways): honour Kafka static group membership #4575, feat(gateways): answer OffsetCommit and OffsetFetch #4415, feat(server): store offsets of groups managed outside Iggy #4392).
• SASL/PLAIN, TLS and mTLS listeners, and Iggy permissions shown as Kafka ACLs (feat(gateways): SASL/PLAIN authentication for the Kafka gateway #4222, feat(gateways): add TLS listeners and mTLS client identities to Kafka gateway #4535, feat(gateways): describe Iggy permissions as Kafka ACLs #4245).
• A Docker Compose quick start (feat(gateways): Kafka gateway Phase 1 quick-start and Docker Compose E2E #4372).

Delivery is at-least-once. Idempotent producers start, but the gateway does not deduplicate their retries. Transactions are not supported. Issue #3560 tracks the protocol coverage.

Per-stream encryption keys

In 0.9.0, one server key encrypted the stored payloads and user headers of every stream. In 0.10.0, each encrypted stream has its own data key (#4615). When a client creates a stream, it can supply the key. Otherwise, a server with encryption enabled generates one.

The server stores each stream key encrypted under a key derived from its [encryption] key, and it never returns a stream key. Reads report only the key source and fingerprints. Rotation of the server key runs in two phases and never re-encrypts messages. The server decides encryption once per stream, at create. If you enable encryption later, it applies only to new streams.

Breaking changes

• Wire protocol: a 0.10.0 server speaks protocol 0.12.0 and refuses 0.9.x clients at login. Upgrade the server and the SDKs together (feat(server)!: unify resource options and add per-stream keys #4615).
• Data directory: the on-disk format changed, and there is no migration of 0.9.x data directories. As with 0.9.0, until 1.0.0 an upgrade starts from a fresh data directory.
• Clusters: every node must run the same build and the same cluster.coordinator.replica_connections_per_peer. Rolling upgrades are not supported (feat(server)!: route partition requests to their primaries #4506, feat(message_bus)!: spread replica traffic over k links per peer #4656). HTTP relays between nodes carry a proof that is valid for 10 s, so node clocks must agree within a few seconds (feat(cluster)!: spread and rebalance partition primaries #4619).
• Server configuration: a fresh server no longer generates a root password. Set IGGY_ROOT_USERNAME and IGGY_ROOT_PASSWORD, or use --with-default-root-credentials for development (fix(server)!: drop generated root passwords, journal root, gate snapshots #4585). consumer_group.session_timeout is gone, and the server refuses to start while it is set. The session timeout is now 1.2 x heartbeat.interval (fix(server)!: derive session timeout from heartbeat.interval #4562). An invalid logging.level stops the boot, and log files rotate by size only (fix(server)!: make logging honor retention, level, path and file_enabled #4354). In http.jwt.trusted_issuers, username replaces user_id (fix(server)!: bind grants and HTTP tokens to the objects they name #4550).
• Removed commands: stream and topic purge (refactor!: remove the purge API and its server machinery #4477) and flush unsaved buffer (refactor!: remove the dead flush unsaved buffer command #4317). UpdateResource replaces UpdateStream, UpdateTopic and UpdateUser (feat(server)!: unify resource options and add per-stream keys #4615).
• Consumers: the Rust high-level consumer defaults to AutoCommit::Disabled, and the stream builder starts at PollingStrategy::next() instead of last(). An application that never stores offsets reads everything again after a restart. The Python consumer_group and the PHP consumerGroup take the same auto-commit default. Connector sinks ignore poll_interval and wait with max_wait and min_count (feat(server)!: add bounded deferred (long) polling #4629).
• SDK APIs: public APIs of the Rust, Go, Java, C#, Node.js, C++ and Swift SDKs changed for partition routing and recovery (feat(server)!: route partition requests to their primaries #4506, feat(sdk)!: activate partition routing and recovery #4561). The Rust stream configuration takes stream and topic names only, without IDs (fix(sdk): removes topic_id and stream_id from public Stream API #4249). The Python SDK requires Python 3.11 or later (chore(python): require Python 3.11 across SDK and CI #4421).

SDKs and Connectors

A Swift SDK is new (#4190, #4503, #4553, #4577, #4641). It has a TCP client with optional TLS, resource management, a batching producer and a consumer that commits its position. It does not yet expose the recovery APIs of the other SDKs. This release ships it as source in foreign/swift.

The C++ high-level client gained consumer, offset, user and client functions (#4180, #4256). The Python SDK gained disconnect and shutdown, segment deletion and identity details at login (#4288, #4357, #4276). The C#, Java and Go SDKs parse iggy:// connection strings (#4363, #4504, #4497). Every SDK can follow metadata changes without listing all resources again (#4625), and it can send a raw request to a chosen replica (#4639). The CLI reads IGGY_TOKEN (#4478) and shows partitions and configuration in topic get (#4558).

Two connectors are new: a Google BigQuery sink (#4394) and an MQTT source (#4303). The Delta Lake sink improved its S3 support (#4219).

Performance and operations

Partition file I/O runs as bounded jobs, so a slow disk no longer stalls other partitions on the same shard (#4311). Replica links batch small frames per socket read (#4224). They can also use several TCP connections per pair of nodes, set by cluster.coordinator.replica_connections_per_peer (default 1) (#4656).

One catalog of configuration keys handles creates and updates of streams, topics and users on every transport (#4615). At startup, the server raises its open-file limit. If storage runs out of file descriptors, the server exits with a clear status (#4318). /metrics reports the size and message count of each topic (#4480). A Docker Compose file starts a three-node cluster (#4499), and chaos tests kill nodes under load to find data loss (#4407, #4557).

Thanks,
Hubert

[slbotbm]

excited for the release!

[kparisa]

+1 binding (Apache Iggy PMC)
So many great features in this release.

[numinnex]

+1 binding (Apache Iggy PMC)

[haubur]

+1 (non-binding)
yet again impressive

[spetz]

+1 binding (Apache Iggy PMC)

Can't wait to have this officially released, tons of improvement optimizations, as well as the overall stability, reliability and even more features across the server + SDKs.

[lukaszzborek]

can't wait
+1 (non-binding)

[ryankert01]

+1 (non binding)

(Excited