puma-plus

A Ruby web server with a Go front end, built around one idea: request queue time is the number that matters, so measure it exactly and use it to size the backend automatically.

Go owns HTTP/1.1, HTTP/2 and HTTP/3. A Ruby gem runs the Rack app. Between them is a small binary protocol over unix sockets.

gem install puma-plus

It runs Rails and Sinatra, conforms to Rack 3, and is configured by a Ruby file that reads like a puma config. It is still experimental — a spike that grew up enough to install, not a product. Nothing here has run in production, the version is 0.1.0, and the interfaces may change.

                                   ┌─────────────────────────────┐
  h1 / h2 / h2c / h3   ──────────▶ │  Go: parse, queue, measure  │
                                   │      dispatch, control      │
                                   └──────────┬──────────────────┘
                                              │ binary frames over unix sockets
                                   ┌──────────▼──────────────────┐
                                   │  Ruby: shepherd → workers   │
                                   │  each thread = one conn     │
                                   └─────────────────────────────┘

Queue time — how long a request sat runnable but unserved — is the single best signal of whether a Ruby app is under-provisioned. Puma does not measure it. X-Request-Start appears in puma's repository only in documentation; the actual measurement is delegated to APM middleware, which computes it from a header a load balancer set, at millisecond granularity, across a clock boundary.

The only per-request timing puma emits is env['puma.request_body_wait'] (lib/puma/client.rb:428,746) — time spent waiting for the client to send the body. Its entire stats surface (lib/puma/thread_pool.rb:143) is backlog, backlog_max, running, pool_capacity, busy_threads, requests_count, reactor_max. No latency. No percentiles. Nothing per-request.

This is not an oversight, it is structural. Puma's queue is smeared across three places: the kernel accept backlog (invisible), each worker's @todo array, and the reactor. No single component knows the depth, and what does get reported is sampled worker→master every worker_check_interval (default 5 seconds) over a hand-rolled pipe protocol.

So puma's thread pool scales reactively and blindly: threads spawn only on enqueue when @waiting < @todo.size (thread_pool.rb:347), shrink on a 30-second trim timer, and know nothing about latency.

The Go-in-front idea comes from puma-dev, which proves the architecture works but punts on everything this project is about. It has no queue depth, no backpressure, and no per-app connection cap. Its readiness check dials the app's socket every 250ms, with a comment in the source calling it "a poor substitute for getting an actual readiness signal" (dev/app.go:336-361). Its reverse proxy sets FlushInterval: 1 * time.Second (dev/http.go:63), which batches server-sent events into one-second hiccups.

Ruby worker threads dial in to Go rather than Go connecting to them. An idle connection is a unit of capacity. Everything follows from that inversion:

Because the queue lives in exactly one Go data structure, we own both sides of Little's Law: L is an integer we maintain, W is measured per request. That makes W_pred = (L+1)·S̄/c — the expected wait of the request arriving now — computable. Measured percentiles are lagging indicators; a request must suffer the queue before it can report it.

All numbers from an ARM64 Lima VM, 12 CPUs, Ruby 3.2.3 without YJIT, over loopback. Servers pinned to CPUs 0–7, load generator to 8–11. Compared against puma 8.0.2 and falcon 0.54.3 running the same Rack app.

These are same-machine relative comparisons. They are not absolute throughput and do not transfer to the x86 hardware this would really run on.

On the load generator. wrk, hey and ab are closed-loop: each virtual user waits for its response before issuing the next request, so when the server slows down the generator slows with it and the queue never grows the way it does under real traffic. That is coordinated omission, and it makes those tools structurally incapable of measuring this. The suite ships its own open-loop generator that schedules arrivals from an independent clock and measures latency from each request's intended start time. See bench/README.md.

The workload. Every server runs the same Rack app, which picks its behaviour from the request path. /mix/<cpu_ms>/<sleep_ms> burns that many milliseconds of CPU and then sleeps that many more, so /mix/5/20 is 5ms of computation followed by 20ms of waiting — roughly the shape of a Rails request that renders a little and waits on a database. The CPU portion is calibrated against CLOCK_THREAD_CPUTIME_ID rather than a fixed iteration count, so "5ms of CPU" means the same thing on ARM and on x86.

25ms of service time means one thread can serve ~40 requests/second, so puma -w2 -t5 (10 threads) tops out near 400 rps.

The traffic. Offered load steps 60 → 700 → 60 rps, holding each for 30s, 60s and 30s. The middle phase is deliberately ~1.75× what an average-sized server can handle: an earlier version of this scenario stepped to 260 rps, stayed inside capacity, and every server scored identically because nothing ever queued.

Requests arrive at random times rather than on a metronome — a Poisson arrival process:

evenly spaced   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·
Poisson         ··  ·      ····  ·  ·       ···  ·   ··      ·

Both average 700 per second. But real users don't take turns, so they clump — and a clump is what makes a queue. Spacing requests evenly would smooth away the exact bursts we are trying to measure.

¹ total latency above the 150ms SLO, summed over every request. 1.1 Gms is not a typo: ~45,000 requests each waiting ~25 seconds over a 150ms target.

These numbers predate the settled-gate fix described in docs/RACTORS.md, which changed how quickly the controller ramps. They are reported as measured, on Ruby 3.2.3; the autoscaling row in particular would move on a rerun. The shape of the result — autoscaling matching peak provisioning on throughput and p50, losing on p99 — has held across every rerun since.

This table is a single run per server, unlike the steady-state and overhead tables below which are medians of three. The effects here are large enough (seconds versus milliseconds) that run-to-run variance cannot account for them, but a single rep is weaker evidence than three and should be read that way.

Three readings, including the one that does not flatter us:

Autoscaling matched peak provisioning's throughput and p50, starting from one worker. 382 rps either way, p50 30.2ms vs 27.6ms, at 28% less average memory (130MB vs 180MB). Against the configuration people actually deploy — sized for the average — it delivered 21% more throughput and cut excess latency 56×.

Peak provisioning still wins p99 decisively: 38ms vs 2.76s. Scaling 1→6 workers takes seconds, and every request arriving during the ramp queues.

The obvious explanation is fork dead time, and that turned out to be only part of it. Replacing fork with Ractor.new — a ~125× faster actuator — barely moved p99 at all, because the controller adds one unit per decision and each decision costs a confirmation window no actuator can shorten. Fork dead time is real and secondary: with the ramp measured properly, forking costs 2.78s p99 where Ractors cost 1.16s. Most of the remainder is the ramp policy itself.

That investigation is written up in docs/RACTORS.md, including the controller bug it uncovered — readiness was inferred from a 2s timer rather than from the connection that already signals it — and the mistake in reasoning it produced. If your load is predictable, provision for it.

puma-plus static ties puma at the same shape — 316 vs 315 rps, 24.88s vs 25.32s p50. The Go↔Ruby hop costs nothing measurable under load.

The table above is an average over the whole run, which hides the interesting part. Here is the same autoscaling run second by second (bench report -timeline):

    t     rps        p50        p99    procs  queue_p95
  39s      59     28.7ms     37.7ms        3        0.0
  40s     699      830ms      1.23s        4      604.0
  44s     660      2.73s      2.83s        5     2415.9
  48s     726      2.49s      2.53s        6     2415.9
  52s     700      1.58s      1.73s        7     1208.0
  56s     705    214.6ms    410.2ms        8     1006.6

Reading the columns:

• rps — requests completed that second, so you can see the load step land.
• p50 / p99 — what clients experienced, measured from intended arrival.
• procs — the whole process tree: the Go server, the Ruby shepherd, and one per worker. So 3 is one worker, and 8 is six.
• queue_p95 — milliseconds requests spent waiting for a free worker, read from puma-plus's own /stats. This is the number the controller steers on.

What happens:

t=39s. One worker, 59 rps, queue_p95 is flat zero. There is always a free thread, so nothing waits.

t=40s. Load jumps to ~700 rps against a server sized for 60. Capacity is instantly ~12× short. Queue time goes from 0 to 604ms in a single second, and p50 follows it to 830ms. The controller sees the breach and forks — procs goes 3 → 4.

t=40→48s. Workers are added one at a time, not all at once: a fork must finish booting before the next is allowed (the "settled gate"). Meanwhile the backlog is still growing, and queue_p95 peaks at 2.4 seconds.

Here is the part worth understanding. Latency keeps climbing while capacity is being added. That is not the controller failing — it is the backlog that built during the ramp having to be worked off. New workers first drain the queue that already exists before newly arriving requests feel any relief. The gap between "capacity added" and "latency recovered" is fork dead time, and it is the entire reason peak provisioning still wins p99 in the table above.

t=52→56s. Capacity now exceeds the arrival rate, so the backlog drains. queue_p95 falls 2415 → 1208 → 1006ms, and p50 follows it down from 2.5s to 215ms.

Total: full recovery from a 12× step in about 16 seconds, with zero shed requests and zero errors — every one of those 45,853 requests was eventually served.

For contrast, here is average-sized puma under exactly the same load:

    t     rps        p50        p99    procs  queue_p95
  30s      73       27ms     32.3ms        3      (n/a)
  42s     704      2.05s      2.78s        3      (n/a)   ← step lands
  54s     737     12.35s     12.79s        3      (n/a)
  66s     717     22.54s     22.99s        3      (n/a)
  78s     737      32.5s     32.96s        3      (n/a)
  90s     717     43.38s     43.82s        3      (n/a)   ← still climbing
 102s      69     49.11s     49.49s        3      (n/a)   ← load drops, backlog remains
 126s      59     28.64s     29.09s        3      (n/a)   ← run ends, still draining

Latency grows linearly and without bound — roughly 10 seconds of added p50 for every 12 seconds of overload. That straight line is the signature of a queue with no admission control: nothing sheds, nothing scales, so the backlog simply accumulates. When the load finally drops at t=90s, latency keeps rising for another 12 seconds before draining begins, and the run ends with p50 still at 28.6 seconds. Those requests all eventually returned 200, long after any real user had given up.

The queue_p95 column is (n/a) for a reason worth sitting with: puma cannot report it. The harness polls each server's own stats endpoint, and puma has no latency or queue-time metric to give. Everything in the left-hand columns is measured by the client from outside. The one number that would have told an operator what was happening — and told a controller what to do about it — does not exist inside puma.

The same /mix/5/20 request at a flat 120 rps against servers that are correctly sized for it. Median of 3 reps, 0.0% spread.

Autoscaling is slightly worse here, and that is the expected answer. All four delivered the offered rate exactly. The three static configurations are indistinguishable on latency. Autoscaling costs ~3ms of p50 and ~17ms of p99 because it settles at fewer workers and therefore runs closer to the edge — in exchange for 33% less memory than static puma.

That is the honest shape of the trade: when you have sized your server correctly, this buys you nothing and costs you a little tail latency. It earns its keep when the load moves or the operator guessed wrong.

puma-plus does something puma does not: it moves every request across a process boundary. /hello returns a fixed string, so almost all of what is measured is that overhead. 805 rps offered, median of 3 reps, spread 0.0%.

puma and puma-plus differ by 3 microseconds, which this harness cannot resolve. Its floor is ~1.5ms, measured against a server that does nothing at all. The honest statement is that the Go↔Ruby hop is not measurable here — not that it is free, and certainly not that puma-plus is faster. A machine with finer timing resolution might well find puma ahead.

The memory column is above the noise and is worth a note: puma-plus used 105MB across more processes (6 vs 5 — a Go process, a shepherd, and four workers) than puma's 128MB. The Ruby workers are thinner, because HTTP parsing, the reactor and the connection machinery all live in Go instead. That is a real consequence of the architecture rather than a tuning artifact, though on a trivial app it is also the least interesting thing about it.

Falcon uses fibers, not threads, and that makes "equivalent configuration" genuinely hard to define. Two things had to be checked before any falcon number above could be trusted.

Does the benchmark app block falcon's reactor? Ruby 3.x routes Kernel#sleep through Fiber::Scheduler#kernel_sleep when a scheduler is installed, and Async installs one — so sleep should yield rather than block. "Should" is not good enough: if it did not, falcon would be serializing every request and every number here would be unfair to it. Measured directly (bench/checks/fiber-scheduler.sh), 20 concurrent requests each sleeping 500ms against a single server process:

The single-thread control serializing is what makes this meaningful — the test could have detected the failure and did not. sleep yields properly, and the Rack app needs nothing special for falcon.

But the concurrency budgets are not comparable, and this favours falcon. Falcon got 20-way concurrency out of one process; puma needed 20 threads for the same. --count sets processes only — there is no fiber limit to match against a thread count. So in the tables above, falcon -n2 and puma -w2 -t5 have CPU parity (two processes each, so two cores' worth of GVL) but falcon has unbounded request concurrency where puma has exactly ten threads.

That asymmetry is visible in the step results: falcon reached a better p50 than average-sized puma (17.93s vs 25.32s) precisely because nothing was queued behind a thread limit. What it could not do was convert that into throughput — /mix has a 5ms CPU component, and burn_cpu is a tight loop with no yield points, so two falcon processes are capped at two cores just as two puma workers are. It also completed 4,175 fewer requests than the others, having hit the scenario's 20-second client timeout.

Matching falcon on processes is the fairest single knob available, but it is not equivalence, and the tables should be read with that in mind.

puma-plus only — puma has no HTTP/2 or HTTP/3 at all, and falcon has no HTTP/3.

These rows are a capability result, not a performance one. All five land within noise, which is exactly what should happen: everything h2 and h3 buy — fewer round trips, 0-RTT resumption, no head-of-line blocking — needs latency to be visible, and loopback has none. h3's slightly worse p99.9 is QUIC's userspace packet handling costing something for no compensating benefit at zero RTT. A fair protocol comparison needs synthetic RTT (tc netem in a netns), which this VM cannot do unprivileged. Do not quote these as evidence about h3.

What they do show: all three protocols produce byte-identical Rack envs apart from SERVER_PROTOCOL, verified by TestProtocolsProduceIdenticalEnv. The Ruby side contains no protocol-specific code whatsoever.

Queue time is only useful if it is correct. 40 concurrent requests against 4 threads with a 100ms service time, on a cold server:

queue p50  369ms      (expected ~450: 10 waves of 4 at 100ms)
queue p95  872ms
service    109.1ms    (app sleeps 100ms)
TTFB p50   503.3ms  vs  queue + service = 478.2ms

Three independently measured intervals that reconcile. Under light load, queue p99 is 1.9 microseconds.

Client slowness is kept out of the signal. A 2MB body dribbled over 2 seconds:

body read   997.3ms     ← charged here
service     15.7ms      ← after subtracting body read
queue       0.0ms       ← uncontaminated

Cost on the hot path: two time.Now() calls (vDSO, no syscall) and one atomic histogram increment, measured at 2.0ns uncontended and 182ns with all 12 cores recording in a tight loop. Zero allocations.

• Sub-millisecond overhead. The harness's own resolution floor is ~1.5ms on this machine, measured against a server that does nothing. An early /hello run showed puma, falcon and puma-plus all at "p50 1.2ms" — that was the harness measuring itself. Three servers reporting the same number there are indistinguishable, not equal.
• Beating a correctly-sized static server. On steady load a correctly sized puma matches puma-plus exactly. Autoscaling wins when the operator guessed wrong or the load moves — which is most production, but it is not a speed result.
• Helping CPU-bound work past saturation. Twelve cores is twelve cores. Past hardware saturation, capacity cannot fix queue time and a controller that keeps forking is actively harmful. What differs there is bounded brownout via shedding versus unbounded collapse.
• Anything about HTTP/3 performance. See above.
• Thread scaling. Implemented behind a flag and off by default. Under the GVL, adding a thread to CPU-bound work makes queue time worse, so the gain sign is unknown — which is also why the controller is tiered thresholds with hysteresis rather than a PID. GVL wait is now measured directly, through rb_internal_thread_add_event_hook in an optional C extension, so the controller can tell "the GVL is the bottleneck" from "more threads would help" rather than inferring it. Without the extension it reports unknown rather than zero: an absent measurement must not read as "no contention", or the controller confidently adds threads to a process that is already GVL-bound.
• Production readiness. It is v0.1.0 and has never run in production. It serves Rails and Sinatra correctly in testing, which is a much weaker claim than being safe to put traffic on.

8-byte header (u8 type | u24 reserved | u32 length) then payload. Full spec in docs/PROTOCOL.md; the two implementations (internal/wire/wire.go, gem/lib/puma_plus/wire.rb) are held byte-identical by a golden-corpus round-trip test.

Go builds the complete Rack env and sends it as a flat key-value blob, so Ruby performs zero string transformation — it dups a frozen prototype hash and merges byteslices of one payload string. All the CGI-ification puma does across five layers happens once, in Go.

T1 first byte   T2 headers parsed   T2b body buffered (runnable)
T3 dispatched   T4 app.call entered   T5 app returns   T6 flushed

Queue time is T3 − T2b, and nothing else — the interval where the request was fully runnable and we had no capacity. That property is what makes it the right control input; every other interval is polluted by things scaling cannot fix. Bodies are buffered up to 1 MiB before the clock starts, so slow uploaders land in body wait; above that they stream and the worker reports body_read_ns for subtraction.

Tiered thresholds with hysteresis, cooldowns and gain probing — not PID. The plant has discrete actuators whose costs differ by three orders of magnitude, seconds of variable dead time, and an unknown gain sign. Decide() is a pure function: no clocks, no IO. That makes the JSONL decision log replayable, so a production incident becomes a reproducible test case:

tick   3  spawn_worker   queue p95 503ms over target 50ms for 1 ticks (predicted 557ms, rho 0.04, 1 workers)
tick   4  none           breach for 1 ticks but worker 1 is still booting
tick   6  none           breach for 3 ticks but spawn cooldown has 0s left
tick   7  spawn_worker   queue p95 302ms over target 50ms for 4 ticks (predicted 227ms, rho 0.25, 2 workers)

Declined decisions explain themselves too. That log is how the spawn cooldown got retuned from 15s to 3s: it showed fifteen consecutive breaching ticks with the controller sitting on its hands. The discrete-event simulator (internal/control/sim) then quantified the trade — 15s→2s improved queue p95 from 1.597s to 405ms and cut shedding 6.5× — and showed why it was safe: at realistic fork latencies the settled gate ("never fork while a worker is still booting") is what actually prevents thrash, so the cooldown was redundant.

That gate originally approximated "still booting" as "younger than 2 seconds". It now reads the truth instead: a worker dials in only once its app is loadable, so the connection appearing in the registry is the readiness signal, and the shortfall between expected and observed capacity is exactly the capacity still on its way. The approximation was wrong in both directions — it blocked scaling for 2s after a worker was already serving, and declared ready a worker that might still have been booting a slow application.

gem install puma-plus

That installs two gems: puma-plus (this gem, plus a small C extension) and puma-plus-core (the Go server, precompiled for your platform). No Go toolchain is needed unless you are on a platform with no prebuilt binary, in which case the source gem builds it at install time. See docs/PACKAGING.md.

In a Rails or Sinatra app, replace puma in the Gemfile:

gem "puma-plus"

Then configure it the way you would configure puma, in config/puma-plus.rb:

port 9292
threads 5
workers 2
environment "production"

# puma-plus specific
autoscale min: 1, max: 8, target_queue_p95: "25ms"
activate_control_app "127.0.0.1:9293"

before_fork  { ActiveRecord::Base.connection_handler.clear_all_connections! }
on_worker_boot { |i| ActiveRecord::Base.establish_connection }

puma-plus                 # reads config/puma-plus.rb
puma-plus --port 3000     # flags beat the file
puma-plus --dry-run       # show what it resolved, and exit

DSL names match puma wherever the concept exists, so a puma config ports by deleting lines rather than being rewritten — and a directive with no equivalent is reported rather than silently ignored. Full reference in docs/CONFIG.md.

go build -o puma-plus-server ./cmd/puma-plus

./puma-plus-server -app examples/hello/config.ru -workers 2 -threads 5
curl localhost:9292/
curl localhost:9293/stats?pretty=1     # puma-compatible JSON + queue metrics
curl localhost:9293/metrics            # Prometheus

# Everything at once, with autoscaling
./puma-plus-server -app examples/hello/config.ru \
  -listen 127.0.0.1:9292 -h2c \
  -listen-tls 127.0.0.1:9443 -tls-cert cert.pem -tls-key key.pem \
  -listen-h3 127.0.0.1:9443 \
  -autoscale -min-workers 1 -max-workers 8 -target-queue-p95 25ms \
  -decision-log /tmp/decisions.jsonl

/stats is a strict superset of puma's shape, so existing dashboards and pumactl-style tooling keep working, with everything new under a queue key.

go build -o bench ./bench/cmd/bench
./bench smoke -server puma -path /hello -rate 300 -dur 10s
./bench run bench/scenarios/step-mix.yaml
./bench report -timeline pumaplus-auto1-6 bench/results/<run>

Scenarios, fairness rules and the harness's known limits are documented in bench/README.md.

Experimental, v0.1.0, published to RubyGems. It has never run in production.

Works, and verified against real applications:

• HTTP/1.1, h2c, h2 and HTTP/3, all reducing to the same request frame, so the Ruby side never learns which one it was. Real TLS certificates (--tls-cert/--tls-key, or ssl_bind in the config); a self-signed development certificate is generated when none is given. HTTP/3 is advertised with Alt-Svc, without which no browser would ever find it.
• Rack 3 conformance, validated with Rack::Lint against rack 3.2.6 — zero violations across enumerable and streaming bodies, chunked upload, multi-value headers, rack.errors and full hijack. See docs/RACK.md.
• Rails 8.1 and Sinatra 4.2, booted from the published gems: preloading, forked workers, lifecycle hooks, ActiveRecord writes, HTML views.
• Configuration by a Ruby file with puma's DSL names, plus before_fork, on_worker_boot and on_worker_shutdown. docs/CONFIG.md.
• Exact queue-time measurement and autoscaling driven by it.
• Go-managed WebSockets and WebTransport, where the app receives tagged messages rather than connections, with sticky or flexible routing.
• Ractors as a unit of capacity (--ractors), an experiment inside an experiment: same throughput as forked workers at 56% of the memory, and it is where the autoscaler's ramp behaviour was finally understood. docs/RACTORS.md.

Not implemented: early hints, partial hijack (use a streaming body), Rack 2 compatibility, phased restart, config reload, multi-app routing, adaptive shed limits, state_path, stdout_redirect, and plugins. Lifecycle hooks do not run inside Ractors — they are blocks, which cannot cross a Ractor boundary.

Known sharp edges: the app must be deep-freezable to use --ractors, which excludes Rails; the config file is read twice, so top-level side effects in it happen twice; and only aarch64-linux has been exercised end to end from the published gem, the other platform binaries being cross-compiled and statically verified.

Both live in this repo's lineage and both were read closely while building it: puma for the request lifecycle, thread pool and cluster protocol, and puma-dev for the Go-in-front architecture. Where puma-plus diverges, the source says why and cites the file and line it is diverging from.