Skip to content

Commit f136122

Browse files
committed
docs(bench): define the Bun idle RSS comparison
1 parent fb79f0e commit f136122

1 file changed

Lines changed: 30 additions & 12 deletions

File tree

bench/memory/README.md

Lines changed: 30 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,28 @@
11
# Idle memory benchmark
22

33
This suite measures resident memory after a server has handled sustained load
4-
and then sat completely idle. Its default method matches the Bun 1.4.1 release
5-
comparison: 60 seconds of load, followed by 180 seconds of idle.
4+
and then sat completely idle. Its default API profile follows the Bun 1.4.1
5+
comparison method: 60 seconds of load, 180 seconds of idle, 64 connections,
6+
and RSS sampled every 100 milliseconds.
67

78
```bash
8-
bun run bench:memory
9+
bun install --cwd bench/routing --frozen-lockfile
10+
BENCH_DEDICATED=1 bun run bench:memory --driver oha --runs 3
911
```
1012

11-
The runner uses the same targets, scenarios, byte-for-byte response parity
12-
checks, and load generators as `bench/routing`. It samples only the server PID,
13-
so memory used by the load generator is not counted. Linux reads `VmRSS` from
14-
`/proc`; macOS and other Unix systems read RSS through `ps`.
13+
The default profile compares byte-identical JSON responses at the fixed rates
14+
shown in the Bun graphic. Stacks, Elysia, Hono, and the raw Bun baseline receive
15+
40,000 requests per second. Express and Fastify receive 25,000. Stacks is held
16+
to the higher tier rather than being assigned an easier rate.
17+
18+
Next.js SSR and Vite dev are separate workload classes. They are not included
19+
in the API table because comparing an SSR render or development transform with
20+
a static JSON response would be misleading. They need equivalent Stacks SSR
21+
and Stacks dev fixtures before becoming executable profiles.
22+
23+
The runner uses the same byte-for-byte response parity checks as
24+
`bench/routing`. It samples the entire server process tree, so launchers cannot
25+
hide worker memory. The separately launched load generator is not counted.
1526

1627
## Reported value
1728

@@ -28,18 +39,25 @@ Every run writes:
2839
- `measurements.json`, the compact values and complete run metadata
2940
- `raw/<target>--run<N>.json`, every RSS sample and raw load-generator result
3041

31-
No baseline is checked in. A publishable baseline requires a controlled machine
32-
and a native load generator such as `oha` or `bombardier`. The built-in Bun
33-
driver keeps the suite runnable everywhere, but its report is marked
34-
direction-only because it competes with the server for runtime and CPU.
42+
The report declares the requested and delivered rate. A row below 98% rate
43+
attainment is marked invalid and cannot be used for a memory win. Fixed-rate
44+
runs require `oha`; the runner fails instead of silently substituting a
45+
saturating driver.
46+
47+
No baseline is checked in. A publishable baseline requires Bun 1.4.1 from the
48+
Pantry environment, `oha`, and dedicated Linux x64 hardware. Set
49+
`BENCH_DEDICATED=1` only on such a machine. Other runs are marked
50+
direction-only. The weekly GitHub Actions run is a regression signal and
51+
uploads its raw samples, but shared-runner numbers are not publishable.
3552

3653
## Flags
3754

3855
```text
3956
--targets comma-separated target ids
4057
--scenario routing scenario to load (default static-json)
4158
--driver oha | bombardier | autocannon | builtin
42-
--connections concurrent connections (default 50)
59+
--connections concurrent connections (default 64)
60+
--rate override every target's declared fixed request rate
4361
--load sustained-load seconds (default 60)
4462
--idle quiet seconds after load (default 180)
4563
--interval RSS sample interval in milliseconds (default 100)

0 commit comments

Comments
 (0)