Native MQTT protocol surface for Iggy

[ygndotgg]

Native MQTT protocol surface for Iggy

Status

Design validation proposal. This document describes a constrained first phase and its long-term architectural boundaries. It does not want to propose full MQTT broker compatibility.

Executive summary

Iggy is an append-only distributed log and message streaming platform. MQTT should therefore be introduced as an isolated protocol and delivery runtime above the existing streaming engine.

The proposed design is:

MQTT client
    │
    ▼
MQTT TCP/TLS listener
    │
    ▼
MQTT protocol and session runtime
    │
    ▼
MQTT-to-Iggy adapter
    │
    ▼
existing Iggy partition, consumer, offset, VSR, and storage paths

The initial phase does not modify:

• Iggy's native binary wire format
• Message
• the native TCP, QUIC, or WebSocket request path
• the append-only partition log
• the VSR protocol
• the existing storage format
• native client behavior

The initial phase supports TCP/TLS, exact topics, clean sessions, QoS 0, and constrained QoS 1. QoS 2, retained messages, wildcard subscriptions, persistent sessions, offline queues, shared subscriptions, and MQTT over WebSocket are deferred.

The central design position is:

MQTT should be native to the Iggy server process, but not native to the Iggy streaming core. MQTT owns protocol and broker-facing session semantics. Iggy owns durable messages, partitions, offsets, routing, VSR, and storage.

1. Strategic context

The primary architectural concern is preventing MQTT-specific semantics from leaking into Iggy's generic streaming core.

MQTT introduces state that does not naturally belong in Iggy's append-only log path:

• sessions
• subscriptions
• packet identifiers
• delivery acknowledgements
• in-flight QoS state
• retained values
• wildcard matching
• offline queues

The separation is important for two reasons:

• Isolation: MQTT-specific state and behavior remain outside the generic Iggy message path.
• Performance: Native Iggy traffic continues through the existing path without MQTT parsing, session lookups, or conditional branches.

The goal is not to make every Iggy operation understand MQTT. The goal is to expose MQTT semantics at a protocol boundary while reusing Iggy's existing streaming primitives underneath.

An external MQTT gateway remains a valid alternative. It provides stronger process isolation but adds another deployment unit and network hop. This proposal evaluates the in-process option because it offers direct connectivity and avoids an additional hop while preserving the same semantic boundary as a gateway.

2. Existing Iggy architecture

2.1 Native binary protocols

TCP, TCP-TLS, WebSocket, WSS, and QUIC carry Iggy's native binary protocol.

TCP / TLS / WebSocket / QUIC
        │
        ▼
transport listener
        │
        ▼
transport-specific framing
        │
        ▼
Message<GenericHeader>
        │
        ▼
RequestHeader
        │
        ▼
RequestHandler
        │
        ▼
server dispatch
        │
        ├── authentication and sessions
        ├── metadata consensus
        ├── partition operations
        ├── consumer groups
        ├── consumer offsets
        └── storage

Relevant code:

• core/message_bus/src/transports/mod.rs
• core/message_bus/src/framing.rs
• core/message_bus/src/client_listener/mod.rs
• core/message_bus/src/installer/mod.rs
• core/server/src/boot/listeners.rs
• core/server/src/dispatch/mod.rs

The existing TransportConn abstraction is designed for transports that decode Iggy GenericHeader messages. MQTT packets do not have that shape, and MQTT requires long-lived subscription and delivery state. MQTT should therefore be a parallel protocol surface rather than another TransportConn implementation in the first phase.

2.2 HTTP precedent

HTTP already demonstrates that Iggy can host a separate protocol surface:

HTTP request
    │
    ▼
Axum route and body decoder
    │
    ▼
HTTP authentication/session
    │
    ▼
internal Iggy request construction
    │
    ▼
partition and metadata operations

HTTP does not use the native RequestHandler, but it reuses Iggy's internal operations and storage.

Relevant code:

• core/server/src/http.rs
• core/server/src/http/submit.rs
• core/server/src/http/handlers.rs

HTTP is a precedent for protocol-path separation. It does not prove complete resource isolation: HTTP, MQTT, and native traffic still share process CPU, memory, runtime capacity, sockets, and storage resources.

3. Proposed boundary

                  Iggy Server
                       │
          ┌────────────┴────────────┐
          │                         │
   Native Iggy path            MQTT path
          │                         │
 TCP/TLS/WS/QUIC              MQTT listener
          │                         │
 GenericHeader                MQTT packets
          │                         │
 RequestHandler               MQTT runtime
          │                         │
          └────────────┬────────────┘
                       │
                 Shared Iggy operations
                       │
          ┌────────────┼────────────┐
       Storage      Partitions    Offsets

MQTT receives its own protocol-specific entry point and translates MQTT operations into existing Iggy operations. Native traffic does not enter the MQTT parser.

4. Proposed MQTT architecture

+-------------------------------------------------------------------+
| Layer 1: MQTT protocol and connection layer                       |
|                                                                   |
| TCP/TLS, packet parsing, CONNECT, CONNACK, PUBLISH, SUBSCRIBE,   |
| PUBACK, PINGREQ, DISCONNECT, packet limits, keepalive             |
+-----------------------------------+-------------------------------+
                                    │
                                    ▼
+-------------------------------------------------------------------+
| Layer 2: MQTT session and delivery runtime                        |
|                                                                   |
| Clean-session state, exact subscriptions, packet IDs, QoS 1      |
| in-flight messages, outbound queues, delivery, retry, backpressure|
+-----------------------------------+-------------------------------+
                                    │
                                    ▼
+-------------------------------------------------------------------+
| Layer 3: MQTT-to-Iggy adapter                                     |
|                                                                   |
| Topic mapping, payload/header conversion, partition-key selection,|
| publishing, polling, and offset commit                           |
+-----------------------------------+-------------------------------+
                                    │
                                    ▼
+-------------------------------------------------------------------+
| Layer 4: Existing Iggy streaming core                             |
|                                                                   |
| Partition routing, VSR, append-only storage, consumers, offsets, |
| authorization primitives, and native binary client paths         |
+-------------------------------------------------------------------+

Layer 4 is not modified for MQTT in the initial phase. The server does gain additive runtime code for Layers 1–3.

5. Listener and connection flow

The MQTT listener uses a separate configured address or port.

MQTT client
    │
    ▼
MQTT TCP/TLS listener
    │
    ▼
MQTT connection task
    │
    ├── decode MQTT packets
    ├── authenticate CONNECT
    ├── maintain clean-session state
    ├── process MQTT commands
    └── start delivery work

The native paths remain separate. The separate listener prevents native clients from entering the MQTT parser.

The initial implementation can reuse:

• existing TCP listener binding patterns
• existing TLS credential loading
• existing compio runtime conventions
• existing shutdown mechanisms
• existing server bootstrap patterns

This reuses listener infrastructure without making MQTT an implementation of the native Iggy frame transport.

Milestone 1 assumes a shard-0-resident MQTT listener and session runtime to mirror the existing HTTP/server bootstrap model. This is an explicit initial scalability tradeoff, not a general Iggy runtime guarantee. MQTT sessions are not migrated between shards or nodes in this phase.

6. Configuration and bootstrap changes

A new self-contained MQTT configuration section is required. A location consistent with the current repository is:

core/configs/src/server_config/mqtt.rs

The configuration would include, subject to final review:

• enabled flag
• bind address and port
• TLS settings
• handshake and keepalive timeouts
• maximum packet size
• maximum connections
• maximum subscriptions per connection
• maximum in-flight QoS 1 messages
• outbound queue limits
• topic mapping policy
• partition-key policy

Configuration integration would likely touch:

• core/configs/src/server_config/mod.rs
• core/configs/src/server_config/server.rs
• core/configs/src/server_config/defaults.rs
• core/configs/src/server_config/displays.rs
• core/configs/src/server_config/validators.rs
• embedded default configuration
• environment-variable mappings

Boot integration would mirror the HTTP lifecycle:

prepare configuration
    → bind listener
    → start MQTT runtime
    → track background task

The MQTT listener would not initially be registered as a native Iggy transport. It would use its own port and remain outside the existing native transport-discovery contract.

Endpoint discovery decision

Milestone 1 treats MQTT endpoint discovery as an independent deployment/configuration concern.

MQTT clients obtain the endpoint through:

• explicit client configuration
• deployment configuration
• external service discovery

MQTT is not added to TransportPorts, cluster metadata, or the derived native client-listener precedence in Milestone 1.

If future requirements need cluster-advertised MQTT endpoints, the following would need to change:

• TransportPorts
• BoundAddresses
• topology resolution
• cluster validation
• current-config generation
• roster endpoint rendering

7. Authentication and identity

The existing server contains reusable authentication and session functions, including:

• verify_login_credentials
• verify_pat_credentials
• submit_register_on_owner
• submit_logout_on_owner
• mint_shard_zero_client_id

Relevant code:

• core/server/src/dispatch/session_ops.rs
• core/server/src/session_manager.rs
• core/shard/src/coordinator.rs

The identities must remain distinct:

MQTT Client ID
    logical MQTT identity

Iggy transport client ID
    physical connection identity

Iggy authenticated user ID
    authorization identity

Iggy VSR client ID
    consensus/session identity

Iggy consumer identity
    delivery and offset identity

For the constrained phase, only clean sessions are supported. MQTT Client ID therefore does not restore persistent MQTT state after disconnect or server restart.

Initial authentication mappings may be:

MQTT username/password → Iggy user credentials
MQTT token             → Iggy PAT

Authorization must be applied independently to:

• publishing
• subscribing
• polling
• offset commits

Some current authentication and authorization helpers are internal to core/server. A protocol-neutral internal facade may need to be extracted rather than duplicating security logic in the MQTT module.

8. Publishing flow

MQTT client
    │
    │ PUBLISH(topic, payload)
    ▼
MQTT protocol layer
    │
    │ validate and authenticate
    ▼
MQTT-to-Iggy adapter
    │
    │ resolve topic and partition key
    ▼
Iggy SendMessages
    │
    ▼
partition owner
    │
    ▼
partition commit/completion path
    │
    └── completion confirmation
            │
            ▼
        MQTT PUBACK

A default mapping could be:

MQTT topic:
sensors/device-123/temp

Iggy:
stream: mqtt-default
topic:  sensors/device-123/temp

The initial implementation should prefer pre-provisioned Iggy streams/topics or explicit configured mappings. Automatic creation is deferred because it introduces metadata races, permission ambiguity, and unbounded resource creation.

Partition-key selection must be configurable. Possible policies include:

• full MQTT topic
• MQTT Client ID
• configured topic segment
• MQTT user property
• balanced partitioning

The implementation must not assume that every MQTT topic contains a device identifier.

QoS 1 publishing contract

MQTT PUBLISH
    → Iggy SendMessages
    → normal Iggy partition completion/commit point
    → MQTT PUBACK

PUBACK means that the mapped Iggy write reached the completion point defined by Iggy's normal partition semantics. It does not automatically mean that the message has been fsynced to disk, and it does not provide exactly-once delivery.

Retries may produce duplicates. The implementation must preserve MQTT packet identity for the duration of the connection and document duplicate behavior.

9. Consumption and delivery flow

Iggy remains pull-based and MQTT remains push-based. The MQTT runtime acts as a push facade over Iggy polling without changing the partition API.

                    EXISTING IGGY
                         │
                         ▼
                ┌─────────────────┐
                │ Iggy Partition  │
                └────────┬────────┘
                         │
                        POLL
                         │
                         ▼
              ┌──────────────────────┐
              │ MQTT Delivery Worker │
              │                      │
              │ poll                 │
              │ match subscriptions │
              │ track ACK state     │
              │ apply backpressure  │
              └──────────┬───────────┘
                         │
                        PUSH
                         │
                         ▼
                   MQTT Client
                         │
                      PUBACK
                         │
                         ▼
              StoreConsumerOffset
                         │
                         ▼
                  committed offset

The key invariant is:

POLL != ACK

Polling must not advance the durable committed consumer offset before the MQTT client acknowledges delivery.

For MQTT QoS 1 delivery:

PollMessages(auto_commit = false)
    → MQTT PUBLISH
    → wait for client PUBACK
    → StoreConsumerOffset

The current Iggy API represents the read as:

PartitionRead::Poll {
    consumer,
    args,
}

The auto_commit = false setting belongs to PollingArgs carried by args.

If the client disconnects before acknowledging, redelivery is allowed while the relevant delivery state remains available. Clean-session disconnects do not provide offline redelivery guarantees.

10. Subscription semantics

Milestone 1 supports exact topics only.

MQTT SUBSCRIBE sensors/device-123/temp
    │
    ▼
exact Iggy topic mapping
    │
    ▼
per-subscriber consumer state
    │
    ▼
delivery worker
    │
    ▼
MQTT PUBLISH to client

Ordinary MQTT subscriptions are fan-out. Iggy consumer groups are load-balancing. Therefore, ordinary MQTT subscribers must not share one Iggy consumer group.

MQTT Client A → independent consumer identity → offset A
MQTT Client B → independent consumer identity → offset B

If both clients subscribe to the same MQTT topic, both must receive every matching message. Client A may acknowledge a message while Client B has not, so their delivery positions must advance independently.

Consumer groups may become useful for future MQTT shared subscriptions, but they are not the implementation for ordinary MQTT fan-out.

The implementation must choose between:

• per-subscriber pollers, which are simpler but increase read and offset traffic
• shared topic readers with per-subscriber delivery cursors, which reduce polling but require more buffering and replay logic

This is an implementation decision that should be benchmarked rather than hidden in the initial design.

11. Session model

Milestone 1 supports clean sessions only.

On disconnect:

• active MQTT subscriptions are discarded
• in-flight MQTT state is discarded
• delivery workers are stopped
• temporary consumer state is released
• pending clean-session delivery is not retained for offline replay

The runtime still requires connection-local state for:

• active subscriptions
• MQTT packet identifiers
• QoS 1 in-flight messages
• outbound queues
• keepalive timers
• pending acknowledgements

Persistent MQTT sessions are excluded because they require durable storage for subscriptions, session expiry, pending messages, packet identifiers, redelivery state, and offline queues.

12. Milestone 1 scope

Supported

• MQTT over TCP
• MQTT over TLS
• MQTT 3.1.1 baseline
• CONNECT / CONNACK
• username/password or PAT authentication
• exact topic subscriptions
• QoS 0
• constrained QoS 1
• PUBLISH
• PUBACK
• PINGREQ / PINGRESP
• DISCONNECT
• clean sessions
• configurable stream/topic mapping
• configurable partition-key selection
• authorization checks
• packet limits
• connection and subscription limits
• in-flight limits
• outbound backpressure
• MQTT-specific metrics

Excluded

QoS 2

QoS 2 requires durable transaction state for:

PUBLISH → PUBREC → PUBREL → PUBCOMP

That state must survive retransmission, disconnect, node failure, and recovery.

Retained messages

Retained messages require latest-value semantics:

MQTT topic → current retained value

This is different from Iggy's append-only log and should be designed as a separate latest-value or compacted-state subsystem.

Persistent sessions and offline queues

These require durable per-client MQTT state and recovery across restart or failover.

Wildcard subscriptions

• and # require topic matching and indexing. Scanning Iggy topics or partitions for every subscription is not acceptable.

Shared subscriptions

These may eventually map to Iggy consumer groups, but they should be specified separately because their delivery semantics differ from ordinary MQTT subscriptions.

MQTT over WebSocket

Existing WebSocket support carries Iggy binary frames. It does not automatically provide MQTT subprotocol negotiation or MQTT packet handling. MQTT over WebSocket can be considered after the TCP/TLS path is validated.

13. Configuration, runtime, and core impact

The initial MQTT phase is not “no change.” It is a moderate server/runtime addition that deliberately avoids modifying the most sensitive storage and consensus components.

Area	Impact	Classification
MQTT packet parser	Low	Protocol-only
TCP/TLS listener	Medium	Server/runtime
MQTT connection tasks	Medium	Server/runtime
Authentication bridge	Medium	Server/runtime
Topic mapping	Medium	Server/runtime
Partition-key selection	Medium	Server/runtime
Publish adapter	Low/Medium	Server/runtime
QoS 0 delivery	Medium	Server/runtime
QoS 1 publishing	Medium	Runtime and acknowledgement semantics
QoS 1 delivery	Medium/High	Streaming delivery semantics
Consumer-offset integration	Medium	Streaming semantics
Native binary hot path	Low	Must remain unchanged
Message format	Low	No change required initially
Partition log	Low	No change required initially
VSR protocol	Low	Reuse existing partition consensus
Persistent sessions	High	Distributed state
Retained messages	High	New state semantics
Wildcards	High	New routing/index semantics
QoS 2	High	Durable transaction semantics
Testing	High	New protocol and failure matrix
Operations	Medium	New listener, limits, metrics, and configuration

14. Performance and safety requirements

The MQTT runtime must bound:

• maximum MQTT packet size
• maximum MQTT connections
• subscriptions per connection
• in-flight QoS 1 messages
• outbound queue size
• poll batch size
• delivery-worker count
• authentication attempts
• topic mapping count

Separate ports provide protocol-path isolation, not complete resource isolation. MQTT and native traffic still share process resources.

The implementation should benchmark at least:

MQTT disabled
MQTT enabled but idle
MQTT active under load

Metrics should cover:

• active connections
• active subscriptions
• publish count by QoS
• PUBACK latency
• delivery latency
• redeliveries
• offset commit failures
• dropped packets
• backpressure refusals
• authentication failures
• outbound queue saturation

15. Failure and recovery semantics

The first phase should explicitly define these cases:

Situation	Initial behavior
Client disconnects before QoS 1 PUBACK	Delivery may be retried while state remains available
Offset commit fails after PUBACK	Duplicate delivery is allowed
Partition changes owner during polling	Poll fails or retries through the existing shard/partition path
MQTT node fails	Clean-session state is lost; client reconnects
Delivery worker stops	Pending connection-local state is discarded or retried according to runtime policy
Client is slower than Iggy	Bounded queues apply backpressure; memory must not grow without limit

The first phase provides at-least-once behavior while relevant clean-session state remains available. It does not provide exactly-once delivery or offline recovery.

16. Pros

1. MQTT broker state remains outside the generic Iggy message path.
2. Native framing, RequestHandler, partition log, VSR, storage, and native client behavior remain unchanged.
3. No new wire-format or storage-format compatibility problem is introduced.
4. MQTT clients can connect directly to Iggy without an additional network hop or separately deployed broker.
5. Existing TCP/TLS binding, authentication, shutdown, and bootstrap patterns can be reused.
6. Publishing can reuse SendMessages and existing partition routing.
7. Consumption can reuse PartitionRead::Poll and StoreConsumerOffset.
8. The POLL != ACK invariant makes the QoS 1 delivery contract explicit.
9. Ordinary MQTT fan-out is kept separate from Iggy consumer-group load balancing.
10. The scope limits the first implementation to a moderate server/runtime change.
11. Future broker features can be introduced incrementally through separate subsystems.

17. Cons and risks

1. MQTT adds a permanent public protocol and operational surface to the Iggy server.
2. The delivery runtime is new stateful continuous-loop code with no direct equivalent in the current server.
3. QoS 1 subscription delivery is substantially more complex than QoS 1 publishing.
4. Per-subscriber consumer state can increase offset cardinality and replicated offset traffic.
5. A shard-0-resident Milestone 1 runtime may become a connection or delivery bottleneck.
6. Separate ports do not eliminate CPU, memory, latency, or storage cross-talk.
7. MQTT client retransmission and Iggy retry behavior can combine to produce duplicates.
8. Backpressure, in-flight windows, slow consumers, and worker cancellation require new logic.
9. Clean sessions do not support offline delivery or recovery after node failure.
10. QoS 2, retained messages, wildcards, and persistent sessions remain significant future work.
11. New configuration, metrics, documentation, interoperability tests, and failure tests must be maintained permanently.

18. Decisions requested

Before implementation, the following decisions should be confirmed:

1. Is an in-process native MQTT surface preferred over an external MQTT gateway for the first phase?
2. Is a shard-0-resident MQTT runtime acceptable for Milestone 1?
3. Should MQTT remain outside cluster endpoint discovery initially?
4. Must MQTT topics map to pre-provisioned Iggy topics?
5. What exact Iggy completion point should authorize MQTT PUBACK?
6. Is QoS 1 outbound delivery included in the first implementation or treated as a follow-up after QoS 1 publishing?
7. Should a protocol-neutral internal data-plane facade be extracted for authentication, publishing, polling, authorization, and offset commits?
8. What connection, subscription, in-flight, and queue limits are acceptable?
9. Should the first implementation support only MQTT 3.1.1 or a defined subset of MQTT 5?

Conclusion

MQTT is feasible as a separate protocol and delivery surface within the Iggy server.

The recommended first-phase design is:

MQTT TCP/TLS listener
    → MQTT protocol/session runtime
    → MQTT-to-Iggy adapter
    → existing Iggy publish, poll, offset, partition, VSR, and storage paths

The implementation should keep Iggy's native binary path and streaming core unchanged while adding the required MQTT runtime above them.

The main impact is moderate server/runtime work, not a partition or VSR redesign. The highest-risk area is QoS 1 subscription delivery:

polling
  → per-subscriber delivery state
  → MQTT packet identifiers
  → client PUBACK
  → offset commit
  → redelivery and backpressure

The final architectural position is:

MQTT should be native to the Iggy server process, but not native to the Iggy streaming core. The MQTT protocol and delivery runtime should remain isolated above the existing append-only engine.

[ygndotgg]

Cons of Native MQTT

• Protocol isolation doesn't mean resource isolation. Even with a separate MQTT port, MQTT and native traffic would still share the same process, CPU, memory, socket descriptors, and runtime capacity. As noted in Sections 2.2 and 14, the separate port mainly isolates the network path, not the underlying resources.

• Connection-heavy MQTT workloads could put pressure on the core runtime. Every MQTT connection brings TCP/TLS handshakes, CONNECT/PINGREQ processing, session lookups, keepalive handling, and delivery work. With a large number of IoT clients connecting or reconnecting at once, that work still lands inside the Iggy server process and could compete with the core streaming workload.

• It introduces MQTT-specific concerns into the core. Things like MQTT listeners, connection/session handling, keepalive timers, packet processing, and protocol-specific configuration would become part of the main server. Over time, this could make the core more coupled to MQTT-specific behavior, which seems contrary to the goal of keeping Iggy focused on its append-only log/streaming model.

• The failure domain remains shared. A sudden MQTT connection spike, abusive client, or unexpected protocol workload could consume resources that are also needed by the native Iggy paths. A standalone gateway would provide a cleaner boundary where that workload can be scaled, restarted, or isolated without directly impacting the core server.

• MQTT and Iggy scale differently. MQTT workloads can be dominated by connection count and network/session activity, while Iggy nodes are primarily concerned with storage and streaming workloads. Putting both into the same process means they have to scale together, even when only the MQTT side is under pressure.

[justinmclean]

Welcome @ygndotgg, and thanks for writing this up. One existing pattern worth looking at before choosing: gateways/kafka is a separate crate that runs as its own process, with its own listener and connection limit. The gateways README describes gateways as letting existing clients talk to Iggy without changing the core server wire surface. An MQTT gateway built the same way would sit alongside it. Also richardcocks's point on #4181 about running an existing broker with a connector is also worth considering.

[ygndotgg]

Thanks @justinmclean. I've been reviewing gateways/kafka, and I agree that the same architectural pattern could apply to an MQTT gateway as well. I'll write up a proposal covering the technical implementation shortly.

I also agree with @richardcocks' point in #4181. An MQTT gateway could provide a lightweight interface, while users requiring full MQTT broker semantics could use a mature MQTT broker together with an Iggy connector.

In my view, these are complementary options: the gateway would target simpler deployments and direct MQTT-to-Iggy connectivity, while the broker + connector model would provide the broader MQTT semantics and compatibility expected from a full MQTT broker.