Skip to content

feat: add admin HTTP server, MQTT schema discovery, CLI Protobuf decoding, and float display#3

Merged
electricalgorithm merged 13 commits intomainfrom
feat/admin-server
Feb 23, 2026
Merged

feat: add admin HTTP server, MQTT schema discovery, CLI Protobuf decoding, and float display#3
electricalgorithm merged 13 commits intomainfrom
feat/admin-server

Conversation

@electricalgorithm
Copy link
Owner

@electricalgorithm electricalgorithm commented Feb 22, 2026

Overview

Introduces a dedicated HTTP Admin Server to manage the ProtoMQ broker at runtime. This allows for live management without polluting the core MQTT hot-paths or relying on complex MQTT-based authentication for administrative tasks.

Key Features

  • Conditionally Compiled: The Admin Server is strictly optional and disabled by default. It can be built into the server via the zig build -Dadmin_server=true flag.
  • Zero Overhead Footprint: When the build flag is disabled, the binary retains its exact bare-metal memory footprint (verified at ~2.62 MB for 100 concurrent clients), ensuring embedded targets aren't affected.
  • Shared Event Loop: To avoid complex mutex locking, the HTTP listener runs as a non-blocking TCP socket cooperatively on the exact same IOContext event loop as the MQTT server.
  • Bearer Token Auth: All endpoints are secured by checking the Authorization: Bearer <token> header against the ADMIN_TOKEN environment variable.

Endpoints Implemented

  • POST /api/v1/schemas: Dynamically registers a new .proto schema file to memory and persists it to the ./schemas/ directory, mapping it to a specified MQTT topic.
  • GET /api/v1/schemas: Retrieves a JSON map of all currently active topics and their linked Protobuf message types.
  • GET /metrics: Telemetry endpoint returning a JSON payload of:
    • Active TCP connections
    • Total routed messages across the broker
    • Total loaded schemas

Testing & Automation

  • Integration Tests: Added tests/cases/admin_server_test.sh to the main run_all.sh CI loop. This suite boots the server, tests 401 Unauthorized rejections, registers a schema, and pulls metrics.
  • Benchmark Fixes: Patched the Python benchmark suite (pyproject.toml) to correctly resolve the psutil dependency required for tracking performance footprints natively.
  • Documentation: Updated README.md to detail the new Admin Server parameters.

@electricalgorithm electricalgorithm self-assigned this Feb 22, 2026
@electricalgorithm electricalgorithm added the enhancement New feature or request label Feb 22, 2026
@electricalgorithm
Copy link
Owner Author

I'm deploying the changes to my server and investigating the functionality.

The Python benchmark suite strictly relies on psutil for monitoring system memory footprints and CPU load during tests. Additionally, the internal common modules failed to package gracefully. This fixes the pyproject definitions to allow seamless uv execution across all environments.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
This adds conditional compilation for a non-blocking Admin Server that safely shares the MQTT event loop. It introduces 'POST /api/v1/schemas' to dynamically register protobuf schemas without restarting the broker, and various telemetry 'GET' endpoints for runtime observability. It tracks total message counts directly inside TopicBroker. It has zero overhead footprint when disabled via build flags.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
This introduces a robust bash integration test that boots the server, verifies bearer token rejection on unauthorized attempts, executes schema loading via POST, and checks the integrity of the /metrics payload. Hooked into run_all.sh to guarantee standard execution.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Adding documentation to the main README detailing the optional nature of the Admin Server, the available endpoints for dynamic schemas, and the explicitly verified zero overhead memory profile when compiled for embedded deployment.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Add MQTT Service Discovery support to the  command:
- Subscribe to `$SYS/discovery/response` and publish to `$SYS/discovery/request`
  before entering the main message loop to fetch topic-to-schema mappings.
- Parse the  protobuf payload using the decoded
   tag and field tag numbers (1/2/3 for topic/message_type/schema_source).
- Populate a  holding a  and topic mapping so
  that  can decode incoming Protobuf payloads automatically.
- Load any embedded  from the discovery response into the registry
  so external schemas do not need to be pre-loaded manually.
- Change  callback signature to use  (mutable) to
  allow the callback to call methods with mutable-receiver requirements (e.g.
  , ).
- Fall back to printing raw bytes if no schema mapping is found.

All 5 integration tests still pass.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Add two new variants to ProtoValue:
  - float32: f32  (for Protobuf 'float' fields, wire type Fixed32)
  - float64: f64  (for Protobuf 'double' fields, wire type Fixed64)

Previously these were stored as raw u32/u64 integers, which resulted
in unintuitive output like '1102315520' instead of '22.500000'.

Changes:
- types.zig: add float32/float64 union arms; debugPrint renders them
  with 6 decimal places using '{d:.6}'
- decoder.zig: in decodeValue, check field.type == .Float/.Double when
  handling Fixed32/Fixed64 wire types and @bitcast the raw bits into
  the correct float type

All 5 integration tests pass.

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
Allow passing --type alongside --proto-dir to the subscribe command to
directly force a topic->message_type mapping for the subscribed topic.
This bypasses the need for the server to advertise the mapping via
Service Discovery, which is useful when the server's topic_mapping does
not include the topic being subscribed to.

The --type flag supplements (not replaces) discovery: discovery still
runs first, then --type overrides/adds the mapping for the exact
subscribed topic.

Example:
  protomq-cli subscribe -t sensor/temp --proto-dir schemas --type SensorData

Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
@electricalgorithm electricalgorithm changed the title feat: add admin http server for runtime proto to topic mapping feat: add admin HTTP server, MQTT schema discovery, CLI Protobuf decoding, and float display Feb 23, 2026
@electricalgorithm
Copy link
Owner Author

This PR was originally scoped to the HTTP Admin Server, but has since grown to include a full end-to-end feature chain: remote deployment, service management, CLI schema discovery, and Protobuf decoding improvements. All features are motivated by the goal of making ProtoMQ a self-describing, operationally friendly MQTT broker.


What's Included

1. 🛠️ HTTP Admin Server (src/server/admin.zig)

A lightweight, optional HTTP server that runs alongside the MQTT broker and exposes runtime management endpoints.

Method Path Description
GET /metrics Returns connection count, messages routed, schema count, and memory usage
GET /api/v1/schemas Lists all registered Protobuf schemas and their topic mappings
POST /api/v1/schemas Registers a new topic → message type mapping at runtime

Opt-in via -Dadmin=true build flag — adds zero overhead in the default build.

2. 🔍 MQTT Service Discovery

The broker now handles a reserved system topic pair for schema advertisement. Publishing to $SYS/discovery/request triggers a ServiceDiscoveryResponse Protobuf reply on $SYS/discovery/response containing all known topic→schema mappings and the embedded .proto source for each type. This allows any client to become schema-aware with no out-of-band configuration.

3. 📟 CLI Schema Discovery & Auto-Decoding (src/mqtt_cli.zig)

protomq-cli subscribe now automatically decodes Protobuf messages:

# Auto-decode via Service Discovery
protomq-cli subscribe -t sensor/data --proto-dir schemas --host 192.168.178.31

# Direct decode — useful when the broker doesn't advertise the topic
protomq-cli subscribe -t sensor/temp --proto-dir schemas --type SensorData --host 192.168.178.31

@electricalgorithm electricalgorithm merged commit cbaffca into main Feb 23, 2026
1 check passed
@electricalgorithm electricalgorithm deleted the feat/admin-server branch February 23, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant