Technical Implementation discussion of Mqtt to Iggy Gateway

[ygndotgg]

The discussion is branching out from the #4181 around the architectural and design of the gateway/mqtt. The design details and implementation details will be added here

[ygndotgg]

1. Motivation & Operational Context

MQTT is the industry-standard messaging protocol for IoT devices, microcontrollers, and edge gateways. Providing an MQTT entry point enables existing MQTT clients to publish to and consume from Iggy. To maintain architectural clarity, two distinct integration paths are defined:

1. Full MQTT Broker + Iggy Connector (MQTT Client → Mosquitto/EMQX → Iggy Connector → Iggy): The preferred, production-grade path for workloads requiring complete broker semantics (QoS 2, retained messages, persistent offline queues, wildcard topic trees, and broker-level failover).
2. Constrained Direct Gateway (iggy-gateway-mqtt): An optional, brokerless compatibility path for simpler deployments needing direct MQTT connectivity without the cost and operational overhead of maintaining a separate MQTT broker cluster.

The standalone gateway introduces an independently deployed process that must be configured, secured, monitored, scaled, and made highly available. This operational cost is justified specifically for brokerless edge deployments.

---

2. Architecture & Process Boundaries

The MQTT gateway operates as an independent, standalone process under gateways/mqtt/ . It is not added to the Iggy server's native transport path or core binary

┌─────────────────────────────────────────────────────────────────────────────┐
│                       STANDALONE PROXY PROCESS                              │
│                                                                             │
│  ┌──────────────┐     ┌──────────────────────┐     ┌─────────────────────┐  │
│  │ MQTT Clients │ ──► │  TCP/TLS Listener    │ ──► │ Bounded Wire Codec  │  │
│  │ (IoT / Edge) │     │  (Port 1883 / 8883)  │     │ (MQTT 3.1.1 / 5.0)  │  │
│  └──────────────┘     └──────────────────────┘     └──────────┬──────────┘  │
│                                                               │             │
│  ┌──────────────┐     ┌──────────────────────┐                │             │
│  │  Iggy Server │ ◄── │  IggyBridge          │ ◄──────────────┘             │
│  │  (Iggy Core) │     │  (Official Rust SDK) │                              │
│  └──────────────┘     └──────────────────────┘                              │
└─────────────────────────────────────────────────────────────────────────────┘

Process Boundary Guarantees

• Gateway Layer (gateways/mqtt/):
  • Listens for TCP and TLS connections on ports 1883 and 8883 .
  • Zero-copy wire codec for MQTT 3.1.1 and selected MQTT 5.0 frames.
  • Keeps track of each connected client's subscriptions and MQTT state, without persisting the session after disconnect.
  • Manages local topic subscription registries, shared-reader polling loops, and per-client bounded outbound queues.
  • Provides strong process isolation: a panic or memory leak in the proxy cannot crash the underlying Iggy server.
• Iggy Core (core/server/) [UNTOUCHED]:
  • Owns log partitioning, streams, topics, VSR consensus replication, append-only log storage, and consumer offsets.
  • No MQTT parsing in the native Iggy code path: Native binary clients experience zero impact on their execution paths.

Proposed Crate Structure

gateways/mqtt/
├── Cargo.toml
├── SCOPE.md
├── README.md
├── src/
│   ├── main.rs                  # Gateway CLI entry point
│   ├── config.rs                # Operational limits & Iggy endpoint configs
│   ├── server.rs                # TCP/TLS acceptor loop
│   ├── error.rs                 # Gateway error & MQTT reason code mappings
│   ├── protocol/
│   │   ├── codec.rs             # Bounded packet encoder/decoder
│   │   ├── packets.rs           # Packet structs (CONNECT, PUBLISH, SUBSCRIBE, etc.)
│   │   ├── requests.rs          # Packet decoders
│   │   └── responses.rs         # Packet encoders
│   ├── session.rs               # Client session context & auth state
│   ├── subscriptions.rs         # Local subscription registry
│   ├── reader.rs                # Topic Reader Manager (Iggy shared readers)
│   ├── delivery.rs              # Fan-out router, per-client queues, & ACK ledger
│   └── iggy_bridge.rs           # Iggy SDK wrapper & primitive mapping
├── tests/
│   ├── codec_tests.rs           # Frame parsing, varints, and header tests
│   ├── server_integration_tests.rs # Integration with running Iggy server
│   └── golden_wire_fixtures_tests.rs # Byte-exact validation against wire traces
└── tools/
    └── mqtt-tool/               # Packet generator for fuzzing & load testing

---

3. Protocol Scope & Compatibility Baseline

Baseline Specification

• MQTT 3.1.1 is the initial compatibility baseline.
• Selected MQTT 5.0 packet properties and reason codes (e.g., user properties, content type) may be supported where documented, but full MQTT 5.0 feature compatibility is out of scope. Full MQTT 5.0 features such as session expiry, topic aliases, message expiry, subscription identifiers, server properties, enhanced auth, flow control, and the AUTH packet are explicitly excluded.

Control Packet Scope (11 of 14 MQTT 3.1.1 Control Packets)

1. CONNECT: Client connection handshake.
2. CONNACK: Connection acknowledgment.
3. PUBLISH: Inbound and outbound message delivery (QoS 0 & QoS 1).
4. PUBACK: Inbound publish ACK or outbound delivery ACK.
5. SUBSCRIBE: Exact-topic subscription request.
6. SUBACK: Subscription confirmation.
7. UNSUBSCRIBE: Subscription removal request.
8. UNSUBACK: Unsubscribe confirmation.
9. PINGREQ: Connection keepalive ping.
10. PINGRESP: Keepalive ping response.
11. DISCONNECT: Graceful session teardown.

Explicit Exclusions & Error Behavior

The following features remain explicitly out of scope:

• QoS 2 (Exactly-Once Delivery): Rejected or returned with MQTT-specific reason codes / disconnects.
• Retained Messages: Excluded; avoids introducing latest-value state stores to Iggy's append-only log.
• Persistent Sessions (CleanSession = false) & Offline Queues: Excluded; requires durable offline message storage across restarts.
• Wildcard Subscriptions (+, #): Excluded; dynamic partition/topic scanning is deferred.
• MQTT Shared Subscriptions & WebSockets: Deferred.

Note on Error Codes: Unsupported features return MQTT-specific reason codes or disconnect behavior according to spec, rather than generic "unsupported version" errors [29].

4. Authentication, Authorization & Validation Pipeline

When an MQTT client connects, the gateway processes the request through a multi-stage pipeline.

1. Connection & Session Establishment

   The overall flow can be summarized as:

   TCP/TLS Connection ──► MQTT CONNECT ──► Packet Validation ──► Credential Auth ──► Session Context Created
                                                                                            │
                                                                                            ▼
   Iggy SDK Operation ◄── Iggy Stream/Topic ◄── Topic Mapping ◄── MQTT PUBLISH / SUBSCRIBE
                                        
                            ▲
                            │
                      Gateway ACL Check
   

   1. Connection Handshake:
      The gateway accepts the TCP/TLS connection and creates a session handler for the MQTT client.
   2. Packet Validation:
      The session validates MQTT packet size, remaining length, protocol version, flags, client ID, topic filters, QoS, packet identifiers, and UTF-8 fields before processing the request.
   3. Authentication:
      MQTT credentials are authenticated by the gateway, creating a client principal and session context. The gateway separately authenticates to Iggy using a dedicated, least-privileged service account or PAT. In the initial phase, MQTT clients do not require individual Iggy accounts or PATs; all authorized gateway operations use the gateway's shared Iggy service identity.
   4. Authorization:
      For every PUBLISH or SUBSCRIBE operation, the gateway maps the MQTT topic to the corresponding Iggy stream/topic and checks the client principal's publish or subscribe permissions. Only authorized operations are forwarded to Iggy through the Iggy SDK using the gateway's shared service identity.

---

5. Topic Mapping & Partitioning Strategy

Topic Mapping Abstraction

MQTT topic names are mapped to Iggy Streams and Topics through an explicit mapping layer within the gateway:

• Default Policy: Use pre-provisioned or explicitly configured MQTT-to-Iggy topic mappings defined in config.rs.
• Automatic Creation: Disabled by default to avoid metadata race conditions, permission ambiguity, and unbounded resource growth.
• Isolation: Topic resolution remains entirely within the gateway; Iggy Core receives standard SDK operations targeting existing Streams and Topics.

Partitioning & Ordering

The gateway selects the target Iggy partition before forwarding an MQTT PUBLISH operation:

• Partition Selection: Supported policies can include MQTT Client ID, a device identifier extracted from the topic, a configured topic segment, or Partitioning::balanced() for throughput-oriented workloads.
• Per-Device Ordering: When ordering is required, messages from the same device are mapped to the same partition using a stable partition key.
• Balanced Distribution: When strict per-device ordering is not required, Partitioning::balanced() can distribute messages across available partitions to maximize throughput.
• Ordering Guarantee: Ordering is preserved within a single Iggy partition. When messages are distributed across multiple partitions, there is no global ordering guarantee across those partitions.

Execution Flow

MQTT PUBLISH
     │
     ▼
Topic Mapping
     │
     ├── MQTT Topic
     │       ↓
     │   Iggy Stream / Topic
     │
     ▼
Partition Selection
     │
     ├── Client ID
     ├── Device ID
     ├── Topic Segment
     └── balanced()
     │
     ▼
Iggy SDK
     │
     ▼
send_messages()

All topic mapping and partition-selection logic is performed inside the gateway process. Iggy Core remains unaware of MQTT-specific topic semantics and receives a standard SDK request targeting the resolved Iggy Stream, Topic, and partition.

---

6. Inbound Publish & Outbound Delivery Flow (ACK Disambiguation)

A crucial architectural distinction is drawn between Inbound Publish Acknowledgements and Outbound Delivery Acknowledgements:

INBOUND PUBLISH FLOW:
MQTT Client ──► PUBLISH (QoS 1) ──► Gateway ──► IggyBridge.send_messages() ──► Iggy Log
                                       │
MQTT Client ◄────── PUBACK ────────────┘  (No consumer offset involved!)

OUTBOUND DELIVERY FLOW:
Iggy Partition ──► Topic Reader (poll_messages, auto_commit=false)
                        │
                  Local Fan-Out Router ──► Bounded Queue ──► MQTT Client (PUBLISH QoS 1)
                                                                  │
Iggy Partition ◄── store_consumer_offset() ◄── Advance Safe ACK ◄─ PUBACK ┘

1. Inbound Publish (Client → Gateway → Iggy):
   • QoS 0: Client sends PUBLISH → Gateway decodes and validates → Maps topic → Calls send_messages(). No ACK returned .
   • QoS 1: Client sends PUBLISH → Gateway writes message to Iggy → Upon reaching configured Iggy durability point → Gateway returns PUBACK to publishing client. No consumer offsets are touched .
2. Outbound Delivery (Iggy → Gateway → Client):
   • Shared Topic Reader polls Iggy (poll_messages with auto_commit = false).
   • Messages are routed through the local fan-out engine into per-client bounded queues.
   • Client receives PUBLISH and responds with PUBACK.
   • Gateway advances client delivery ledger and commits the safe Iggy offset.

---

7. Shared-Reader Fan-Out & Acknowledgement Rules

Shared Reader Model

• Shared Reader Model (Adopted): The gateway maintains one logical reader per Iggy topic/partition. The reader polls Iggy once and distributes each message locally to the outbound queues of all matching active MQTT subscribers.

Iggy Partition
      │
      ▼
Shared Topic Reader
      │
      ▼
Local Fan-Out Router
      │
      ├──► Client A Queue ──► Client A
      │
      ├──► Client B Queue ──► Client B
      │
      └──► Client C Queue ──► Client C

Contiguous Safe Offset Advancement (POLL != ACK)

Because a single Iggy reader serves multiple MQTT subscribers, polling a message from Iggy does not mean that the message has been safely delivered to all required subscribers.

For QoS 1 delivery, the gateway therefore maintains delivery and acknowledgement state for each message.

• Delivery Set: When a message is read, the gateway determines which active subscribers are eligible to receive it. That set becomes the message's delivery set.
• The Rule: The Iggy consumer offset may advance only through the highest contiguous sequence of messages for which every subscriber in each message's delivery set has acknowledged successful delivery.

For example:

Message       A        B        C       Safe Offset
----------------------------------------------------
M100          ACK      ACK      ACK        │
M101          ACK      ACK      pending     │
M102          ACK      pending  pending     │
                                             ▼
                                      Commit through M100

Even though Client A has acknowledged M101 and M102, the shared reader cannot safely advance past M100 because Clients B and C have not completed the required deliveries.

This preserves the shared reader's at-least-once delivery semantics.

• Disconnection Policy: If a client disconnects, its clean-session state is discarded according to the configured disconnect/session policy. The client is removed from the active delivery set, allowing messages to become eligible for offset advancement without waiting indefinitely for that disconnected client.
• Late-Joining Subscribers: A client that subscribes after the reader has already processed earlier messages joins the delivery set only from its subscription point onward. It does not block previously processed messages or require historical replay unless explicitly configured.
• Unresponsive / Slow Subscribers: A subscriber that continuously fails to consume its bounded outbound queue is subject to the gateway's backpressure policy. Once the configured threshold is reached, the gateway may disconnect the client and remove it from the active delivery set, preventing one slow subscriber from indefinitely blocking shared offset progress.
• Gateway Restart: With Clean Start = true / connection-local session handling, in-memory session and acknowledgement state is discarded when the gateway restarts. Any Iggy offsets that were not safely committed are re-read after restart, resulting in at-least-once delivery with possible duplicates. Reconnecting MQTT clients establish new sessions and re-subscribe.

8. Bounded Queues, Backpressure & Resource Protection

To prevent a single slow or disconnected device from consuming all proxy memory, every client session is assigned a bounded outbound queue:

• Queue Bounds: Bounded by maximum message count, maximum byte payload, or maximum in-flight QoS 1 packets.
• QoS 0 Backpressure: If a client queue fills, QoS 0 messages may be dropped or the client disconnected according to policy . since Qos 0 does not require delivery acknowledgement, the gateway does not need to retain the message indefinetely
• QoS 1 Backpressure: If a client's queue becomes full, the gateway stops advancing that client's delivery state until queued messages can be delivered and acknowledged. If the client remains unable to keep up, the gateway may eventually disconnect it according to policy. This prevents a slow client from indefinitely holding resources required for message delivery.
• Separate Session Tasks: Reading from the MQTT socket runs in an asynchronous task separate from the socket writer, ensuring a stalled socket write never blocks incoming PUBACK receipts or disconnect handling.

9. Horizontal Scaling & Multi-Instance Behavior

Running multiple gateway instances behind a TCP load balancer introduces coordination requirements around MQTT connection routing and Iggy partition ownership.

                         TCP Load Balancer
                         /              \
                        ▼                ▼
              Gateway Instance A    Gateway Instance B
                    │                    │
                owns P0, P1           owns P2, P3
                    │                    │
                    ▼                    ▼
                 Iggy P0/P1           Iggy P2/P3

• Single Reader Ownership (Phase 1 Baseline): Each Iggy topic/partition must have one logical gateway reader owner at a time. The initial implementation can achieve this through static partition/topic-domain assignment or by deploying a single gateway instance per topic domain.
• No Duplicate Reads: Multiple gateway instances should not independently consume the same Iggy topic/partition. This keeps the shared-reader model and per-subscriber acknowledgement/offset tracking local to a single gateway instance.
• Load Balancer Responsibility: The TCP load balancer distributes MQTT client connections but does not determine Iggy partition ownership. The gateway deployment must therefore maintain a deterministic relationship between MQTT topic domains/Iggy partitions and their gateway reader owners.
• Phase 1 Failure Handling: Transparent reader ownership transfer and MQTT session migration are out of scope. If a gateway instance fails, its in-memory session state, outbound queues, and acknowledgement state are lost. Reconnecting clients establish new sessions, while uncommitted Iggy messages may be re-read, providing at-least-once delivery with possible duplicates.
• Future Scaling: Dynamic partition ownership, failover, and rebalancing can be introduced later through an explicit coordination/lease mechanism. This would allow ownership of an Iggy partition and its associated shared reader to move safely between gateway instances.

---

10. Operational Capacity & Benchmark Targets

The gateway must enforce hard operational limits to protect system resources:

pub struct GatewayLimits {
    pub max_connections: usize,
    pub max_subscriptions_per_client: usize,
    pub max_packet_size_bytes: usize,
    pub max_inflight_messages_per_client: usize,
    pub max_outbound_queue_bytes_per_client: usize,
    pub max_topic_readers: usize,
    pub idle_timeout_seconds: u64,
}

Benchmark Target Suite

Maximum device capacity will be established through empirical load testing rather than unverified claims. Key benchmark targets include:

1. Idle TCP connection capacity & TLS handshake rates.
2. Inbound publish throughput (messages/sec & MB/sec) .
3. Fan-out expansion throughput at 1:10, 1:100, and 1:1,000 subscriber ratios.
4. Resilience under slow-client backpressure and reconnect storms.

---

11. Comprehensive Testing Infrastructure

1. Protocol Unit Tests (tests/codec_tests.rs):
   • Bounded varint length decoding, invalid flags, UTF-8 validation, and packet size limits.
2. Golden Wire Fixtures (tests/golden_wire_fixtures_tests.rs):
   • Byte-exact validation against standard MQTT 3.1.1 packet traces (CONNECT, CONNACK, PUBLISH, PUBACK, SUBSCRIBE, SUBACK, PINGREQ, DISCONNECT).
3. Integration Tests (tests/server_integration_tests.rs):
   • End-to-end integration with a live Iggy server instance: pub/sub delivery, exact-topic fan-out, QoS 1 ACK tracking, duplicate delivery on retry, and slow-client disconnects.
4. Packet Fuzzing Tool (tools/mqtt-tool):
   • Dedicated CLI utility generating valid and malformed MQTT wire frames to verify proxy stability.