feat: add admin HTTP server, MQTT schema discovery, CLI Protobuf decoding, and float display#3
Conversation
|
I'm deploying the changes to my server and investigating the functionality. |
The Python benchmark suite strictly relies on psutil for monitoring system memory footprints and CPU load during tests. Additionally, the internal common modules failed to package gracefully. This fixes the pyproject definitions to allow seamless uv execution across all environments. Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
This adds conditional compilation for a non-blocking Admin Server that safely shares the MQTT event loop. It introduces 'POST /api/v1/schemas' to dynamically register protobuf schemas without restarting the broker, and various telemetry 'GET' endpoints for runtime observability. It tracks total message counts directly inside TopicBroker. It has zero overhead footprint when disabled via build flags. Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
This introduces a robust bash integration test that boots the server, verifies bearer token rejection on unauthorized attempts, executes schema loading via POST, and checks the integrity of the /metrics payload. Hooked into run_all.sh to guarantee standard execution. Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Adding documentation to the main README detailing the optional nature of the Admin Server, the available endpoints for dynamic schemas, and the explicitly verified zero overhead memory profile when compiled for embedded deployment. Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Add MQTT Service Discovery support to the command: - Subscribe to `$SYS/discovery/response` and publish to `$SYS/discovery/request` before entering the main message loop to fetch topic-to-schema mappings. - Parse the protobuf payload using the decoded tag and field tag numbers (1/2/3 for topic/message_type/schema_source). - Populate a holding a and topic mapping so that can decode incoming Protobuf payloads automatically. - Load any embedded from the discovery response into the registry so external schemas do not need to be pre-loaded manually. - Change callback signature to use (mutable) to allow the callback to call methods with mutable-receiver requirements (e.g. , ). - Fall back to printing raw bytes if no schema mapping is found. All 5 integration tests still pass. Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Add two new variants to ProtoValue:
- float32: f32 (for Protobuf 'float' fields, wire type Fixed32)
- float64: f64 (for Protobuf 'double' fields, wire type Fixed64)
Previously these were stored as raw u32/u64 integers, which resulted
in unintuitive output like '1102315520' instead of '22.500000'.
Changes:
- types.zig: add float32/float64 union arms; debugPrint renders them
with 6 decimal places using '{d:.6}'
- decoder.zig: in decodeValue, check field.type == .Float/.Double when
handling Fixed32/Fixed64 wire types and @bitcast the raw bits into
the correct float type
All 5 integration tests pass.
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Allow passing --type alongside --proto-dir to the subscribe command to directly force a topic->message_type mapping for the subscribed topic. This bypasses the need for the server to advertise the mapping via Service Discovery, which is useful when the server's topic_mapping does not include the topic being subscribed to. The --type flag supplements (not replaces) discovery: discovery still runs first, then --type overrides/adds the mapping for the exact subscribed topic. Example: protomq-cli subscribe -t sensor/temp --proto-dir schemas --type SensorData Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
7a9fdad to
d245ba6
Compare
|
This PR was originally scoped to the HTTP Admin Server, but has since grown to include a full end-to-end feature chain: remote deployment, service management, CLI schema discovery, and Protobuf decoding improvements. All features are motivated by the goal of making ProtoMQ a self-describing, operationally friendly MQTT broker. What's Included1. 🛠️ HTTP Admin Server (src/server/admin.zig)A lightweight, optional HTTP server that runs alongside the MQTT broker and exposes runtime management endpoints.
Opt-in via 2. 🔍 MQTT Service DiscoveryThe broker now handles a reserved system topic pair for schema advertisement. Publishing to 3. 📟 CLI Schema Discovery & Auto-Decoding (src/mqtt_cli.zig)
# Auto-decode via Service Discovery
protomq-cli subscribe -t sensor/data --proto-dir schemas --host 192.168.178.31
# Direct decode — useful when the broker doesn't advertise the topic
protomq-cli subscribe -t sensor/temp --proto-dir schemas --type SensorData --host 192.168.178.31 |
Overview
Introduces a dedicated HTTP Admin Server to manage the ProtoMQ broker at runtime. This allows for live management without polluting the core MQTT hot-paths or relying on complex MQTT-based authentication for administrative tasks.
Key Features
zig build -Dadmin_server=trueflag.IOContextevent loop as the MQTT server.Authorization: Bearer <token>header against theADMIN_TOKENenvironment variable.Endpoints Implemented
POST /api/v1/schemas: Dynamically registers a new.protoschema file to memory and persists it to the./schemas/directory, mapping it to a specified MQTT topic.GET /api/v1/schemas: Retrieves a JSON map of all currently active topics and their linked Protobuf message types.GET /metrics: Telemetry endpoint returning a JSON payload of:Testing & Automation
psutildependency required for tracking performance footprints natively.