Quark-Queue

A high-performance, AMQP 1.0-compliant message broker and client SDK written in Rust

“Born from a desire to truly understand the AMQP protocol by building it from scratch – and then accidentally creating a production‑grade broker.”

Quark-Queue is a complete, production-grade message-oriented middleware (MOM) implementation that provides both a standalone AMQP 1.0-compliant message broker and a high-performance Rust client SDK. Unlike traditional message queues that only offer a client library for external brokers, Quark-Queue delivers the entire stack: low-level protocol handling, type system, client abstractions, broker daemon, clustering, persistence, and operational tooling.

┌────────────────────────────────────────────────────────────┐
│  Quark-Queue Workspace                                     │
│                                                            │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ qq-types │  │ qq-cli   │  │ examples │  │ benches  │    │
│  │  (core)  │◄─┤(mgmt CLI)│  │          │  │          │    │
│  └────┬─────┘  └────┬─────┘  └──────────┘  └──────────┘    │
│       │             │                                      │
│       ▼             │                                      │
│  ┌───────────┐      │      ┌──────────┐                    │
│  │qq-protocol│      │      │quantum-  │                    │
│  │(AMQP 1.0) │      │      │restaurant│                    │
│  └────┬──────┘      │      └──────────┘                    │
│       │             │                                      │
│       ▼             ▼                                      │
│  ┌──────────┐  ┌─────────┐                                 │
│  │qq-client │◄─┤qq-broker│◄──── Daemon & Binaries          │
│  │(SDK)     │  │(server) │                                 │
│  └──────────┘  └────┬────┘                                 │
│                     │                                      │
│                     ▼                                      │
│              ┌────────────────┐                            │
│              │  Storage, Mgmt,│                            │
│              │  Clustering    │                            │
│              └────────────────┘                            │
└────────────────────────────────────────────────────────────┘

Features

Full AMQP 1.0 Protocol: Wire-level compatibility with standard AMQP clients
Multiple Exchange Types: Direct, Topic, Fanout, and Headers routing
Pluggable Storage: In-memory or RocksDB-backed persistence
Clustering: Built-in SWIM gossip protocol for multi-node deployments
Management API: RESTful interface with Prometheus metrics
Async Client SDK: High-performance Rust library with connection pooling
WebSocket Support: AMQP-over-WebSocket for browser clients
TLS Encryption: Built-in rustls support

Roadmap

Planned enhancements for upcoming releases:

CLI --headers support – Allow publishing messages with headers directly from qqctl queue publish, making the headers exchange fully usable without an external AMQP client.
Message TTL / Dead Letter Exchanges – Add time‑to‑live and dead‑letter routing for queues.
Queue Mirroring – Replicate queues across cluster nodes for high availability.
WebSocket Compression – Reduce bandwidth for browser clients.
Admin UI – A graphical dashboard for broker management.

Quick Start

1. Run the Broker

# Clone and build
git clone https://github.com/guap-codes/quark-queue
cd quark-queue
cargo build --release
cargo build --release -p qq-cli

# Start the broker daemon
./target/release/qqd --config docker/config/broker.toml

Default broker listens on:

AMQP: amqp://localhost:5672
Management API: http://localhost:8080

2. Use the CLI

The qqctl CLI tool allows you to manage queues, exchanges, and the broker itself. All commands support multiple output formats (table, json, yaml) and a --raw flag to dump the raw API response.

Queue Management

# List all queues
./target/release/qqctl queue list

# Create a durable queue
./target/release/qqctl queue create orders --durable

# Get queue details
./target/release/qqctl queue get orders

# Delete a queue (with confirmation)
./target/release/qqctl queue delete orders

# Delete without confirmation
./target/release/qqctl queue delete orders --force

# Purge all messages from a queue
./target/release/qqctl queue purge orders --force

Exchange Management

# List all exchanges
./target/release/qqctl exchange list

# Create a direct exchange
./target/release/qqctl exchange create orders-ex --exchange-type direct --durable

# Create a topic exchange with auto-delete
./target/release/qqctl exchange create events --exchange-type topic --auto-delete

# Get exchange details
./target/release/qqctl exchange get orders-ex

# Delete an exchange
./target/release/qqctl exchange delete orders-ex --force

# Bind a queue to an exchange (direct/topic)
./target/release/qqctl exchange bind orders-ex orders --routing-key "orders.*"

# Bind a queue to a headers exchange (requires `--arguments`)
./target/release/qqctl exchange bind headers-ex my-queue \
    --routing-key ignored \
    --arguments '{"format":"jpeg","size":1024}'

# Unbind a queue
./target/release/qqctl exchange unbind orders-ex orders --routing-key "orders.*"

Publishing Messages

For direct, topic, and fanout exchanges, you can publish using the CLI:

./target/release/qqctl queue publish my-exchange --message "Hello" --routing-key test-key

Note: The CLI currently does not support sending message headers, which are required for headers exchange routing. To publish to a headers exchange, use an AMQP 1.0 client (e.g., the Python qpid-proton example in the User Guide) or wait for the planned --headers CLI enhancement (see Roadmap).

Broker Information

# Show broker status (state, uptime, etc.)
./target/release/qqctl broker status

# Perform a health check
./target/release/qqctl broker health

# Show detailed statistics (queues, exchanges, connections)
./target/release/qqctl broker stats

# Show broker configuration
./target/release/qqctl broker config

Output Format Examples

# JSON output
./target/release/qqctl queue list -o json

# YAML output
./target/release/qqctl exchange list -o yaml

# Raw API response (dumps the exact HTTP body)
./target/release/qqctl broker stats --raw

Global Options

-u, --api-url: Override the management API endpoint (default http://localhost:8080)
-t, --timeout: Request timeout in seconds (default 30)
-o, --output: Output format (table, json, yaml, pretty)
--raw: Dump the raw HTTP response body
-D, --debug: Enable debug logging

3. Use the Client SDK

use qq_client::{Connection, Session};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Connect to broker
    let mut conn = Connection::open("my-app", "amqp://localhost:5672").await?;
    
    // Start a session
    let session = Session::begin(&mut conn).await?;
    
    // Create a sender
    let sender = session.create_sender("orders").await?;
    
    // Send a message
    sender.send("Hello, Quark-Queue!").await?;
    
    Ok(())
}

Architecture

Crate Organization

qq-types: AMQP 1.0 type system (primitives, messages, descriptors)
qq-protocol: Frame codec, SASL auth, and transport layer
qq-client: Async client SDK with connection pooling
qq-broker: Core broker daemon with routing and persistence
qq-cli: Management CLI tool (qqctl)

Key Design Decisions

Zero-Copy Where Possible: Bytes for protocol parsing, newtype wrappers for type safety
Async-First: Built on Tokio for high concurrency
Bottom-Up Testing: Each layer is independently testable
Pluggable Backends: Storage trait allows RocksDB, PostgreSQL, or custom implementations

File/Folder Structure

quark-queue/
├── Cargo.toml
├── README.md
├── LICENSE
├── rust-toolchain.toml
│
├── crates/
│   ├── qq-protocol/
│   │   ├── Cargo.toml
│   │   ├── src/
│   │   │   ├── lib.rs
│   │   │   ├── framing/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── frame.rs
│   │   │   │   ├── codec.rs
│   │   │   │   └── transport.rs
│   │   │   ├── sasl/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── anonymous.rs
│   │   │   │   └── plain.rs
│   │   │   └── error.rs
│   │
│   ├── qq-types/
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── message.rs
│   │       ├── primitives.rs
│   │       └── described_types.rs
│   │
│   ├── qq-client/
│   │   ├── Cargo.toml
│   │   ├── src/
│   │   │   ├── lib.rs
│   │   │   ├── connection.rs
│   │   │   ├── session.rs
│   │   │   ├── sender.rs
│   │   │   ├── receiver.rs
│   │   │   ├── pool.rs
│   │   │   └── metrics.rs
│   │
│   ├── qq-broker/                     # The heart of the full broker
│   │   ├── Cargo.toml
│   │   ├── src/
│   │   │   ├── lib.rs
│   │   │   ├── bin/
│   │   │   │   └── qqd.rs            # Daemon: "quark-queue daemon"
│   │   │   ├── broker.rs             # Core routing & state machine
│   │   │   ├── config.rs
│   │   │   ├── server/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── session.rs
│   │   │   │   ├── tcp_server.rs     # AMQP-over-TCP listener
│   │   │   │   └── websocket.rs      # AMQP-over-WebSocket
│   │   │   ├── queue/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── memory_queue.rs   # In-memory fast path
│   │   │   │   └── persistent_queue.rs # Disk-backed durability
│   │   │   ├── exchange/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── direct.rs         # 1:1 routing
│   │   │   │   ├── topic.rs          # Pattern matching
│   │   │   │   ├── fanout.rs         # Broadcast
│   │   │   │   └── headers.rs        # Header-based routing
│   │   │   ├── storage/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── backend.rs        # Storage trait
│   │   │   │   └── rocksdb_impl.rs   # Embedded persistence
│   │   │   ├── management/
│   │   │   │   ├── mod.rs
│   │   │   │   ├── api.rs            # REST management surface
│   │   │   │   └── metrics.rs        # Prometheus exposition
│   │   │   └── cluster/
│   │   │       ├── mod.rs
│   │   │       ├── gossip.rs         # SWIM-style membership
│   │   │       └── node.rs           # Node state & replication
│   │
│   └── qq-cli/
│       ├── Cargo.toml
│       ├── src/
│       │   ├── main.rs
│       │   ├── commands/
│       │   │   ├── mod.rs
│       │   │   ├── queue.rs          # `qqctl queue purge`
│       │   │   ├── exchange.rs       # `qqctl exchange create`
│       │   │   └── broker.rs         # `qqctl broker status`
│       │   └── config.rs
│
├── examples/
│   ├── quantum-restaurant/           # Named for "quark" theme
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── bin/
│   │       │   ├── customer.rs
│   │       │   ├── kitchen.rs
│   │       │   └── waiter.rs
│   │       └── lib.rs
│   └── particle-decay-demo/          # Physics-themed persistence demo
│       └── src/
│           └── main.rs
│
├── benches/
│   └── hadron_collider.rs            # High-energy throughput test
│
├── docker/
│   ├── docker-compose.yml
│   ├── Dockerfile.broker
│   └── config/
│       └── broker.toml

Building from Source

Prerequisites

Rust 1.75 or higher
CMake (for RocksDB build)

Build Options

# Fast debug build (without RocksDB)
cargo build --no-default-features

# Full release build with RocksDB
cargo build --release

# Run tests
cargo test --workspace

# Run benchmarks
cargo bench

Development Build (Fast)

# Skip RocksDB for quick iteration
cargo check --no-default-features
cargo test --no-default-features

Configuration

Create broker.toml:

[server]
bind_addr = "127.0.0.1"
amqp_port = 5672
max_frame_size = 1048576
tls_enabled = false
idle_timeout = "300s"
# Optional WebSocket address
# ws_bind_addr = "127.0.0.1:15672"

[storage]
backend = "Memory" # or "RocksDB"
path = "./data"
sync_writes = false
cache_size_mb = 128

[management]
enabled = true
bind_addr = "127.0.0.1"
port = 8080
prometheus_enabled = true
prometheus_path = "/metrics"

[cluster]
enabled = false
node_id = "node-1"
bind_addr = "127.0.0.1"
cluster_port = 5673
seeds = []
election_timeout = "1500ms"
heartbeat_interval = "500ms"

[logging]
level = "info"
format = "Pretty"
# file = "/var/log/quark-queue/broker.log"  # optional

[sasl]
enabled = false

CLI Reference

qqctl [OPTIONS] <SUBCOMMAND>

Commands:
  queue       Queue management
  exchange    Exchange management
  broker      Broker management
  completion  Generate shell completion
  config      Show configuration
  help        Print this message or help about subcommands

Options:
  -u, --api-url <API_URL>    Management API endpoint [env: QQ_API_URL=]
  -t, --timeout <TIMEOUT>    Request timeout in seconds [default: 30]
  -f, --output <FORMAT>      Output format [default: table] [possible values: table, json, yaml]
      --raw                  Dump raw API responses
  -d, --debug                Enable debug logging
  -h, --help                 Print help

Performance

On a modern machine (AMD Ryzen 9 5950X, 64GB RAM):

Message Routing: ~500k messages/sec (fanout)
Persistent Queue: ~100k messages/sec (RocksDB)
Connection Limit: 10k+ concurrent connections
Memory Usage: ~10MB per 1000 idle connections

Contributing

Run cargo fmt --all before committing
Add tests for new features
Update benchmarks if performance-critical
Ensure cargo clippy --all-targets --all-features -- -D warnings passes

License

Licensed under either of:

Apache License, Version 2.0 (LICENSE-APACHE)
MIT license (LICENSE-MIT)

at your option.

Acknowledgments

Built with:

Tokio - Async runtime
RocksDB - Persistent storage
Axum - HTTP server
Clap - CLI framework

Protocol reference: AMQP 1.0 Specification

Navigation Menu

Repository files navigation

Quark-Queue

Features

Roadmap

Quick Start

1. Run the Broker

2. Use the CLI

Queue Management

Exchange Management

Publishing Messages

Broker Information

Output Format Examples

Global Options

3. Use the Client SDK

Architecture

Crate Organization

Key Design Decisions

File/Folder Structure

Building from Source

Prerequisites

Build Options

Development Build (Fast)

Configuration

CLI Reference

Performance

Contributing

License

Acknowledgments

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.github/workflows		.github/workflows
benches		benches
crates		crates
docker		docker
examples		examples
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE-APACHE		LICENSE-APACHE
LICENSE-MIT		LICENSE-MIT
README.md		README.md
User-Guide.md		User-Guide.md
broker.toml		broker.toml
rust-toolchain.toml		rust-toolchain.toml

Latest commit

History

Folders and files

Repository files navigation

Quark-Queue

Features

Roadmap

Quick Start

1. Run the Broker

2. Use the CLI

Queue Management

Exchange Management

Publishing Messages

Broker Information

Output Format Examples

Global Options

3. Use the Client SDK

Architecture

Crate Organization

Key Design Decisions

File/Folder Structure

Building from Source

Prerequisites

Build Options

Development Build (Fast)

Configuration

CLI Reference

Performance

Contributing

License

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages