The long-lived-connection workload: notes on the Net layer and the collector

[rubys]

The long-lived-connection workload: notes on the Net layer and the collector

Why this workload is new

Everything spinel has been driven with so far — the blog, lobsters, campfire's
HTTP lane — is request-shaped: work arrives, allocates, answers, and the
live set returns to roughly where it started. scripts/bench measures req/sec
and RSS against exactly that shape.

A chat server is not that shape. Basecamp publishes Campfire's requirements as
concurrent users, each holding a WebSocket:

Concurrent users	RAM	CPU
250	2 GB	1
1,000	8 GB	4
5,000	32 GB	16
10,000	64 GB	32

Perfectly linear — ~6.5–8 MB and ~1/300th of a core per idle connection. That
is a CRuby-shaped budget (a connection object graph, a share of the DB pool, a
Redis subscriber, COW decay across forked workers). It is a low bar on memory
and it is the wrong axis anyway: what separates runtimes here is fan-out
latency in a hot room and tail latency under collection.

The interesting property is that the workload inverts the usual generational
assumption. The live set is a large, long-lived connection table that survives
every cycle; the allocation is per-frame and dies immediately. Steady state is
"almost everything I allocate is garbage, and almost everything that is live
has been live for hours."

Campfire is also pinned to one machine by adapter: sqlite3 and Active
Storage's service: Disk, and ONCE's framing is "one person installs it on a
server you control." So there is no distributed tier to fall back on: whatever
one process does is the whole product. That is what makes these two layers
load-bearing rather than interesting.

---

Part 1 — The Net layer

Five observations, roughly in order of how soon each one bites. The first is a
hard wall; the rest are design shape.

1. SP_NET_POLL_MAX is 256, and overflow is silent

/* lib/sp_net.c */
#define SP_NET_POLL_MAX 256
static struct pollfd sp_net_poll_set[SP_NET_POLL_MAX];

int sp_net_poll_add(int fd, int mode_bits) {
    if (sp_net_poll_n >= SP_NET_POLL_MAX) return -1;

Consumers that treat -1 as "not polled" — which is the natural reading —
leave those fds permanently unwatched. There is no error, no log; the
connections simply go deaf. Campfire's smallest published tier is 250
concurrent users, so this is reached before tier one.

Whatever else changes, -1 here would be more useful as a loud failure than a
silent one.

2. The readiness API's shape forces O(n) per tick, independent of backend

reset / add / run / ready is a rebuild-the-set-every-time contract. A
caller with N parked fds walks all N, makes N FFI calls to re-register what has
not changed, calls poll, then walks N again to read results — per tick.
Moving the backend to epoll/kqueue under this API buys a better syscall and
keeps the rebuild.

The shape that matches an event-driven server is persistent registration:
register(fd, mode) / modify(fd, mode) / unregister(fd) / wait(timeout) -> ready list. Registration cost is paid once per connection rather than once
per tick, and wait returns only what is ready, so the caller's work is
proportional to events rather than to connections.

This matters more than the backend swap. It is worth settling the API before
the backend, because a poll-shaped API can hide behind epoll and still be
quadratic.

3. Writes block below the scheduler

int sp_net_write_str(int fd, const char *s) {
    ...
    if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK) && sp_net_wait_io(fd, POLLOUT)) continue;

sp_net_wait_io is a blocking poll(fd, ..., 1000) inside C. For a
fiber-scheduled server this is below the scheduler's floor: a single slow
subscriber with a full socket buffer stalls the entire worker's event loop, up
to a second at a time, and no Ruby-level scheduler can see it happening.

This is the failure mode most likely to be misdiagnosed, because the symptom
is "the runtime is slow" and the cause is "the runtime is parked waiting on one
client."

A partial-write return (bytes written, or -EAGAIN) would let the caller
own buffering and backpressure, which is where that policy belongs — the caller
knows whether to buffer, drop, or disconnect; the socket layer does not.

4. The recv buffers are file-static, so the layer is not reentrant

static char sp_net_recv_buf[SP_NET_BUFSIZE];
static char sp_net_recv_all_buf[SP_NET_BUFSIZE];

One shared buffer per function. That is fine for a cooperative single-worker
model and correct for everything shipped so far. It does mean the net layer
cannot be called from two green threads on two OS workers at once — which is
precisely spinel's differentiator for this workload, since no GVL means one
process can hold every connection and fan out across cores without a broker in
the path.

Caller-supplied buffers would make the layer thread-safe without changing what
the single-worker path does.

5. No TLS

lib/sp_net.h:10 describes sp_net_tls in the future tense. That is fine for
inbound (a chat server behind a proxy is normal) but gates outbound HTTPS,
which in Campfire's case is bot webhooks and web push — two of its three
background jobs. Not urgent for a first binary; it is the next dependency after
one, and it interacts with the single-binary story (static-link vs. a vendored
small TLS stack is a decision, not a detail).

---

Part 2 — The collector

Stated as theory and questions rather than findings — this workload has not
been run yet, so what follows is what we expect to matter and would like to be
wrong about in specific ways.

What the docs already say

docs/internals/gc.md: mark and sweep, non-moving, precise, two heaps — object
(48-byte header) and string (24-byte header plus a marker byte, swept through
its own gate). docs/thread.md: collection stops every worker at a
safepoint; the object sweep is parallel across parked workers; the string
sweep is still serial and is what bounds that shape today;
SPINEL_GC_THRESHOLD_KB (default 256, per worker) is the tuning lever and
scales with worker count.

Why this workload probes an untested corner

The good news first: the allocation pattern is close to ideal for a
generational collector. Frames are born and die within a tick; a minor cycle
should reclaim nearly all of it and, per sp_gc.c, does not free the old list.

The concern is the other half. The old generation is the product: N
connections × (buffer + subscription rows + per-connection state), live for
hours. Every major cycle walks it, and the pause is stop-the-world across all
workers. For a request server a pause is amortized into a percentile nobody
looks at. For a chat server the pause is the product — it is simultaneously
visible to every connected user, because they are all waiting on the same
process.

The specific things we expect to bite, offered as theories:

1. Major-cycle pause scales with connection count, and the connection table
   is exactly the structure that never dies. Any per-connection object we can
   fold into fewer, flatter allocations pays off at 10^4 connections in a way
   it would not at 10^2.
2. A chat server is unusually string-heavy. Every frame is a rendered HTML
   fragment; every subscription holds an identifier string. If the string sweep
   is the serial bound under threads, this workload is close to the worst case
   for it — not because there are many live strings, but because there is a
   high allocation rate of short-lived ones alongside a large long-lived set.
3. SPINEL_GC_THRESHOLD_KB was tuned against allocation-bound benchmarks,
   where retained garbage is the cost being traded. Here the tradeoff is
   different: memory is genuinely plentiful (the published budget is 64 GB) and
   pause is the scarce resource, so the right setting is plausibly much higher
   than anything a request benchmark would suggest.

The competitive frame, stated honestly

This is the one axis where BEAM's structure gives it something spinel's does
not: per-process heaps collected independently mean the pause is proportional
to one connection's own tiny heap, and there is no global stop. Go's answer is
different — a concurrent collector whose pause does not scale with live heap at
all. Both are architectural, not tuning.

Spinel's counter-position is real and worth naming: it is the only one of the
five that runs the actual Rails application code, with no GVL and AOT
compilation — on the HTTP lane the emitted campfire already renders these pages
~4.8× faster than Rails does, on ~30% less RSS. The question this document is
asking is what the tail looks like when the live set stops being small.

Questions

• Is there an existing or planned notion of an arena/zone whose lifetime is
  scoped to something the program knows about — a request, or here a
  connection? Roundhouse has stage-1 AOT zones landed and per-request zones
  sketched; a connection-scoped zone would take most of the long-lived set out
  of the general heap, which would address theory (1) directly.
• What is known about major-cycle pause at 10^5–10^6 live objects? Even a
  rough shape (linear in live objects? in live bytes? dominated by the string
  heap?) would tell us whether to attack allocation shape or pause frequency
  first.
• Is the serial string sweep on the roadmap, and is a high
  SPINEL_GC_THRESHOLD_KB a reasonable interim answer for a
  memory-rich/pause-sensitive deployment, or does something else degrade at
  large thresholds?

We expect to have measurements rather than theories within a few weeks — the
binary is close, and the harness shape is already proven on two other lanes.
Happy to run whatever instrumentation would be most useful and report back
rather than guess.

[rubys]

The two concrete Net-layer asks from Part 1 are now filed upstream as narrow, self-contained issues, so they can be acted on without carrying this whole document:

• sp_net: readiness set is capped at 256 and overflows silently; API shape forces O(n) per tick matz/spinel#4103 — readiness set capped at 256 with silent overflow; and the reset/add/run/ready shape forcing O(n) per tick regardless of backend.
• sp_net: blocking write below the caller's scheduler, and file-static recv buffers under no-GVL threads matz/spinel#4104 — blocking write below the caller's scheduler; file-static recv buffers under no-GVL threads. TLS folded in as a roadmap question.

Part 2 (the collector) stays here for now — those are theories with questions attached, and we do not have measurements yet. Expect to replace them with numbers rather than argument once the binary is up.

The roundhouse-side plan those feed into is #71.

[rubys]

Part 1 is largely answered. matz fixed both issues the same day, in 689c01fd and 55638986. Marking what is closed so this document does not read as a live list of complaints:

#	observation	status
1	SP_NET_POLL_MAX capped at 256, silent overflow	fixed — the set grows; add() returns -1 only on allocator failure, and reports it on stderr
2	reset/add/run/ready forces O(n) per tick	fixed — poll_register/modify/unregister + wait/event_fd/event_mode; O(1) ops, work proportional to events
3	writes block below the caller's scheduler	fixed — sp_net_write_partial returns bytes accepted, 0 for a full buffer, -1 for a gone peer
4	file-static recv buffers, non-reentrant	fixed — per-worker, lazily allocated
5	no TLS	open

Two details in the fixes that are better than what this document asked for, and
that anyone building on them should know: the ready set is a compacted copy
rather than a view, so unregistering an fd while walking events — what a server
does on every EOF — does not disturb the walk; and both sets are per-worker,
so two accept loops no longer share one set. The old four calls still work on
their own set, so adoption can be incremental.

poll(2) remains the backend by design — the new contract is meant to survive an
epoll/kqueue swap underneath without the caller changing again. Which is the
right resolution of the point Part 1 was actually making: the shape mattered more
than the syscall.

Part 2 (the collector) stands unchanged, as does the TLS question. Those are
still theories with questions attached, and the plan is still to replace them
with measurements rather than argument once the binary is up.