Broker Configuration

Broker Config Examples

This document shows sample BrokerConfig YAML files for common performance goals.

The broker attempts to load a YAML file from:

• FELIX_BROKER_CONFIG if set
• otherwise /usr/local/felix/config.yml

All fields are optional. Omitted values fall back to defaults. Every field also has an environment variable override — see Environment Variables for the full FELIX_* mapping, including legacy aliases.

Defaults favor bounded latency and visible overload over unbounded buffering: subscriber queues default to drop_new (shed and count, rather than block or grow), and ingress sheds fire-and-forget publishes under sustained overload unless pub_ingress_wait is enabled.

General Purpose

Balanced settings for mixed workloads and moderate fanout — these are the built-in defaults, shown explicitly.

quic_bind: "0.0.0.0:5000"
metrics_bind: "0.0.0.0:8080"
controlplane_url: null
controlplane_sync_interval_ms: 2000
ack_on_commit: false
max_frame_bytes: 16777216
publish_queue_wait_timeout_ms: 2000
ack_wait_timeout_ms: 2000
disable_timings: false
control_stream_drain_timeout_ms: 50
cache_conn_recv_window: 268435456
cache_stream_recv_window: 67108864
cache_send_window: 268435456
event_batch_max_events: 64
event_batch_max_bytes: 65536
event_batch_max_delay_us: 250
fanout_batch_size: 64
pub_workers_per_conn: 4
pub_queue_depth: 64
pub_inflight_bytes: 67108864
pub_conn_inflight_bytes: 16777216
pub_ingress_wait: false
core_shards: 0
subscriber_queue_capacity: 512
max_subscriptions_per_conn: 4096
subscriber_queue_policy: drop_new
subscriber_writer_lanes: 4
subscriber_lane_queue_depth: 64
subscriber_lane_queue_policy: drop_new
max_subscriber_writer_lanes: 8
subscriber_lane_shard: auto
subscriber_single_writer_per_conn: false
subscriber_flush_max_items: 16
subscriber_flush_max_delay_us: 50
subscriber_max_bytes_per_write: 65536
sub_streams_per_conn: 4
sub_stream_mode: per_subscriber

Latency Optimized

Targets low tail latency for small payloads and limited batching. Matches the latency-demo harness's latency profile (batch = 1): blocking queues so no event is silently dropped, immediate flush, single writer per connection for stable per-message ordering.

ack_on_commit: true # NOTE: serializes commit/ack; reduces pipeline parallelism
publish_queue_wait_timeout_ms: 1000
ack_wait_timeout_ms: 1000
control_stream_drain_timeout_ms: 25
event_batch_max_events: 8
event_batch_max_delay_us: 100
fanout_batch_size: 8
pub_workers_per_conn: 2
pub_queue_depth: 32
subscriber_queue_capacity: 64
subscriber_queue_policy: block
subscriber_writer_lanes: 2
subscriber_lane_queue_depth: 32
subscriber_lane_queue_policy: block
max_subscriber_writer_lanes: 8
subscriber_lane_shard: auto
subscriber_single_writer_per_conn: true
subscriber_flush_max_items: 1
subscriber_flush_max_delay_us: 0
subscriber_max_bytes_per_write: 65536

Throughput Optimized

Targets max lossless publish/delivery throughput with higher batching. Matches the latency-demo harness's throughput profile (batch > 1): blocking queues plus pub_ingress_wait so the publisher is paced to the pipeline's sustainable rate instead of shedding — every message delivered, delivery drops 0.

ack_on_commit: false
publish_queue_wait_timeout_ms: 60000
ack_wait_timeout_ms: 3000
control_stream_drain_timeout_ms: 100
event_batch_max_events: 256
event_batch_max_bytes: 1048576
event_batch_max_delay_us: 2000
fanout_batch_size: 256
pub_workers_per_conn: 8
pub_queue_depth: 256
pub_inflight_bytes: 268435456
pub_ingress_wait: true
subscriber_queue_capacity: 4096
subscriber_queue_policy: block
subscriber_writer_lanes: 4
subscriber_lane_queue_depth: 1024
subscriber_lane_queue_policy: block
max_subscriber_writer_lanes: 8
subscriber_lane_shard: auto
subscriber_single_writer_per_conn: false
subscriber_flush_max_items: 16
subscriber_flush_max_delay_us: 50
subscriber_max_bytes_per_write: 65536

!!! note publish_queue_wait_timeout_ms bounds acked publishes only. With pub_ingress_wait: true, un-acked publishes use unbounded, cancellable backpressure instead: there is no ack channel on which a timeout could be reported, so a bounded wait there could only end in a silent drop. Such a publish waits for capacity and is abandoned only if the connection is torn down.

!!! note Lossless pacing (block queues + pub_ingress_wait: true) is a deliberate opt-in for pipelines that cannot tolerate drops, or for benchmarking sustainable throughput. Production defaults favor shedding (drop_new) so overload stays visible and bounded instead of building unbounded backlog under a slow subscriber. See Benchmarks.

Multi-Core Scaling

core_shards runs each stream's publish worker and subscription lane feeders on a dedicated, core-pinned (Linux) single-threaded runtime, selected deterministically by stream handle id. This keeps a stream's fanout enqueue and dequeue on one core instead of bouncing across tokio's work-stealing pool. Off by default (0); benefits scale with stream count and are neutral-to-positive for single-stream workloads.

core_shards: 4 # e.g. physical cores - 2, leaving headroom for QUIC I/O

See Benchmarks for measured results.

Client-Side Parallelism (Important)

Broker throughput and fanout scalability depend on client publish parallelism. For high throughput or fanout workloads:

• Use multiple QUIC connections
• Use multiple publish streams per connection
• Use round-robin (RR) or hash-based sharding across streams

A single connection with a single publish stream will bottleneck regardless of broker tuning.

Worker Sizing (Important)

IMPORTANT: pub_workers_per_conn should not exceed the number of active publish streams. Excess workers increase contention and can worsen tail latency.

When core_shards > 0, the shard count replaces pub_workers_per_conn as the publish worker count (one worker per shard, so each stream has a single owning core). pub_workers_per_conn is ignored in that mode.

Notes

• All byte values are raw bytes; use powers of two for MiB values (e.g., 1048576 = 1 MiB).
• Tune pub_workers_per_conn and pub_queue_depth together; deep queues trade latency for throughput.
• Increasing pub_workers_per_conn only helps if publish load is spread across multiple streams or connections. Oversubscribing workers relative to streams can degrade performance.
• pub_inflight_bytes bounds actual queued-or-processing publish bytes, independent of pub_queue_depth's item count — a handful of large batches can't blow past the ingress memory budget even with a small queue depth.
• pub_conn_inflight_bytes is a per-connection share of pub_inflight_bytes: it bounds how much of the shared budget a single connection can occupy, so one connection publishing large batches can't starve every other connection's admission. Must be smaller than pub_inflight_bytes to have any effect; a value equal to or larger than it degenerates to "no per-connection cap."
• max_subscriptions_per_conn bounds how many concurrent subscriptions a single QUIC connection may hold, independent of subscriber_queue_capacity. It protects broker memory from a connection that opens unbounded subscriptions rather than bounding any one subscription's buffer size.
• subscriber_queue_capacity and subscriber_queue_policy control broker-core per-subscriber buffering and drop behavior (the fanout enqueue path); subscriber_lane_queue_depth and subscriber_lane_queue_policy control the writer-lane stage one hop later (the actual QUIC write). Both default to drop_new.
• subscriber_writer_lanes and subscriber_lane_shard control outbound event write parallelism.
• subscriber_lane_shard: auto is the default and is usually the best starting point.
• Lanes often help high fanout + large payload workloads, but gains can plateau; do not assume that more than 8 lanes will improve performance.
• Event delivery uses binary EventBatch (or shared EventBatch — see Wire Protocol) frames. Unacknowledged client publishes are binary-encoded by default; acked publishes currently use the JSON control encoding (Publisher::publish_json/publish_batch_json select JSON explicitly).
• Queue depths directly impact memory usage. Large queue depths combined with large batch sizes and high fanout can significantly increase resident memory usage.

---

Generated from docs/broker-config.md in the repository. That file is the source of truth; this page may be behind it.