emqx-go

Proof-of-Concept Implementation of the EMQX MQTT broker, rewritten in Go

简体中文 | Installation | Architecture | API Reference | Contributing

---

EMQX-Go: A Golang Implementation of EMQX

This repository is a proof-of-concept implementation of the EMQX MQTT broker, rewritten in Go. The project aims to replicate the core functionalities of the original Erlang-based EMQX, including MQTT connection handling, message publishing and subscribing, session management, and clustering.

🌟 Features

• Core MQTT v3.1.1 Broker: Supports essential MQTT functionalities, including client connections (CONNECT), topic subscriptions (SUBSCRIBE), and message publishing (PUBLISH).
• Actor-Based Concurrency: Leverages a lightweight, OTP-inspired actor model for robust and concurrent management of client sessions. Includes supervisors that follow the "let it crash" philosophy for fault tolerance.
• Dynamic Clustering: Nodes can form a distributed cluster for high availability and load distribution. Message routing between nodes is handled automatically.
• Kubernetes-Native Discovery: Automatically discovers peer nodes within a Kubernetes environment using a headless service, enabling seamless cluster formation.
• High-Performance Communication: Utilizes gRPC for efficient and strongly-typed inter-node communication for routing, state synchronization, and cluster management.
• Prometheus Metrics: Exposes key operational metrics (e.g., connection counts, actor restarts) in a Prometheus-compatible format for easy monitoring.

🚀 Getting Started

Prerequisites

• Go (version 1.20 or later)
• An MQTT client (e.g., MQTTX or mosquitto_clients)

Building and Running

1. Clone the repository:

   git clone https://github.com/turtacn/emqx-go.git
   cd emqx-go

2. Build the application:

   go build ./cmd/emqx-go

3. Run the broker:

   ./emqx-go

   The broker will start and listen for:

   • MQTT connections on port 1883.
   • gRPC connections for clustering on port 8081.
   • Prometheus metrics on port 8082 at the /metrics endpoint.

Connecting an MQTT Client

You can connect to the broker using any standard MQTT client.

• Host: localhost
• Port: 1883

Once connected, you can subscribe to topics and publish messages to test the broker's functionality.

🏗️ Project Structure

The repository is organized into the following main directories:

• cmd/emqx-go: The main application entrypoint, responsible for initializing and orchestrating all services.
• pkg/: Contains all the core packages of the broker.
  • actor: A lightweight, OTP-inspired actor model for concurrency.
  • broker: The central MQTT broker logic, responsible for handling connections and orchestrating message flow.
  • cluster: Components for clustering, including the gRPC server/client and the cluster state manager.
  • discovery: Service discovery, with a Kubernetes implementation for automatic peer finding.
  • metrics: Defines and exposes Prometheus metrics for monitoring.
  • proto: Contains the Protobuf definitions (.proto files) and generated Go code for gRPC-based cluster communication.
  • protocol/mqtt: Low-level parsing and encoding of MQTT packets.
  • session: An actor-based implementation for managing a single client session.
  • storage: A generic key-value store interface with an in-memory implementation for session management.
  • supervisor: An OTP-style supervisor for managing actor lifecycles and implementing fault tolerance.
  • topic: A thread-safe store for managing topic subscriptions and routing.
• docs/: Contains additional documentation on architecture and APIs.
• k8s/: Kubernetes manifests for deploying the application.

📚 Documentation

The source code is thoroughly documented using GoDoc conventions, providing detailed explanations for all public packages, types, and functions.

You can view the documentation online at pkg.go.dev.

Alternatively, you can generate and view the documentation locally by running:

godoc -http=:6060

Then, open your browser to http://localhost:6060/pkg/github.com/turtacn/emqx-go/.