WebSocket rooms: no rate limit on broadcast — one client can starve every worker's mailbox

[EdmondDantes]

Follow-up to #2 (cross-worker rooms). Not a regression — a limit was never in the design.

What

WebSocketRoom::broadcast() hands one command per worker to that worker's mailbox (thread_mailbox_t, capacity 4096). Nothing bounds how often a connection may broadcast, and nothing separates one room's traffic from another's: every room in the process shares the same per-worker inbox.

So a single client looping on a message that the handler relays with broadcast() fills every other worker's inbox. Once an inbox is full, thread_mailbox_post() fails and the message is dropped — including messages from other rooms and other tenants. Availability damage is process-wide, not confined to the abusive room.

Current behaviour

• Overflow is no longer silent (fixed in 7485dbd): drops are counted, readable via ws_hub_dropped().
• But there is still no counter-measure — the traffic is simply lost, and the sender gets no signal (broadcast() returns members served on the calling worker, which the flood does not affect).

Why it matters

The room API is the one WebSocket surface where an unprivileged peer can cause work on every worker in the process with a single frame. send()/trySend() only ever touch the peer's own connection, and the outbound FIFO is already capped per session (1013 close on overflow). Rooms have no equivalent.

Possible directions (not decided)

• Per-connection broadcast rate limit / token bucket, with a knob on HttpServerConfig (setWsBroadcastRateLimit()), mirroring how the other WS limits are exposed as setters.
• Per-room fair queueing, so one hot room cannot evict another's commands.
• Backpressure instead of drop: let broadcast() report that the fan-out could not be queued, so a handler can slow down (it must stay non-suspending, so this would be a return value, not a park).
• Surface ws_hub_dropped() in the server stats so an operator can see it happening before users report it.

Notes

• Discovered during the review that produced 7485dbd; the drop counter landed there, the limit did not.
• WS_HUB_MAILBOX_CAPACITY = 4096 (src/websocket/ws_hub.c).

[EdmondDantes]

Fixed on 2-ws-cross-worker-rooms (70b8ecc). The room API this issue was written against became topics ($ws->subscribe() / $ws->publish()), but the hazard carried over unchanged: publish() still hands one command per interested worker to that worker's mailbox, and nothing metered it.

HttpServerConfig::setWsPublishRateLimit($perSecond, $burst = 0) — a token bucket per connection, off by default. That default is deliberate: EMQX ships messages_rate as infinity for the same reason, only the application knows its own traffic.

Over the rate publish() throws WebSocketBackpressureException and the connection stays up. That is the part the issue actually named — "the sender gets no signal". An over-rate message otherwise disappears into a full mailbox where neither the sender nor an operator can see it, so refusing loudly beats dropping quietly.

Credit is carried in nanoseconds of allowance rather than whole tokens: the coarse monotonic clock ticks every 4–10 ms, and integer tokens would round away to nothing at any rate finer than that.

On the other three directions listed:

• Surface ws_hub_dropped() in the stats — done, getRuntimeStats() now reports ws_topic_posted / ws_topic_skipped / ws_topic_dropped. The counter existed but was never wired to anything.
• Backpressure instead of drop — that is what the throw is.
• Per-topic fair queueing — deliberately not built. The eviction it was meant to prevent takes a flood to cause, and no single connection can flood any more. Adding it now would be machinery for a case the leash already removed; worth revisiting only if a legitimate multi-tenant workload is measured evicting itself.

Beyond the limit, the blast radius shrank on its own: a publish now only wakes workers that actually hold a matching subscriber (interest filter, counting Bloom over topic prefixes). A publish to a topic nobody in the process listens to costs zero cross-worker wake-ups instead of one per worker — measured, 44 800 useless posts → 0 on 8 workers.

Covered by tests/phpt/websocket/050-topics-publish-rate-limit.phpt: the burst goes through, the flood is refused, the connection survives, and the bucket refills.