Benchmarks: REA on Linux (Node 22, 24 and 26 startup, plus five local models)

[CassieFox23]

How fast REA's server starts on Linux, on Node 22, 24 and 26. I update it each release. A local model side test is in the comments.

It started as a comment on #1435 and moved here on 11 October, so it has its own page. Last updated for 6.4.0 and main at 13e03e9.

Startup: time to the tool list

Seconds, from process start. "Tool list" is the median of 5 runs, with the fastest and slowest run in brackets. "Init" is the median time to the first reply.

Target	Tools	Node	Tool list (min-max)	Init
6.3.0	139	22	1.90 (1.88-1.91)	1.68
6.3.0	139	24	1.59 (1.56-1.61)	1.39
6.3.0	139	26	1.38 (1.37-1.42)	1.21
6.4.0	170	22	2.16 (2.12-2.17)	1.84
6.4.0	170	24	1.75 (1.72-1.80)	1.48
6.4.0	170	26	1.55 (1.52-1.58)	1.31
main (13e03e9)	170	22	1.78 (1.76-1.80)	1.47
main (13e03e9)	170	24	1.47 (1.46-1.50)	1.21
main (13e03e9)	170	26	1.30 (1.28-1.33)	1.06

Main (13e03e9) against 6.4.0, time to the tool list (median):

Node	6.4.0	main	Saved	Faster by
22	2.16 s	1.78 s	0.38 s	17.8%
24	1.75 s	1.47 s	0.28 s	16.0%
26	1.55 s	1.30 s	0.26 s	16.5%

Saved and Faster by come from the exact times, so they can differ by a hair from the rounded cells.

• 6.4.0 is slower than 6.3.0 on all three Node versions, by 0.17 to 0.26 s. It also lists 31 more tools (170, not 139). Main is faster than 6.4.0 by 0.26 to 0.38 s, which is 16 to 18 percent less time. Main is also faster than 6.3.0, by 0.08 to 0.13 s.
• Main has four commits that 6.4.0 lacks. I did not time them one by one, so I cannot say which one changed the time. They are in the first fold.
• Node 26 is fastest and Node 22 is slowest for every target. Three more full runs, one in reverse order, moved no tool list median by more than 0.04 s.

First real job (Node 24)

Main saves 0.18 s at the tool list. The browser capture then takes 0.21 s longer, so the saving does not survive the first capture. The other three jobs keep it.

Seconds from process start to the job's reply, median of 5 runs. The job alone is in brackets (job reply minus tool list reply).

Job	6.3.0	6.4.0	main (13e03e9)
Browser capture	1.76 (0.49)	1.89 (0.50)	1.91 (0.71)
Script analysis	3.35 (2.10)	3.95 (2.51)	3.63 (2.46)
Android app scan	failed	2.03 (0.66)	1.79 (0.64)
Ghidra overview	7.49 (6.31)	7.62 (6.30)	7.45 (6.36)

Browser capture, 6.4.0 against main:

Step	6.4.0	main	Change
Tool list reply	1.38	1.20	-0.18
Capture alone	0.50	0.71	+0.21
Capture reply	1.89	1.91	+0.03

The Change column comes from the exact times.

• The browser saving does not survive. Main starts 0.18 s sooner, then spends 0.21 s more on the capture. The reply comes about 0.03 s after 6.4.0, within the run-to-run spread of about 0.05 s. A reverse-order pass gave -0.22, +0.19 and -0.03. I did not trace where the 0.21 s goes.
• For the other three jobs, the saving stays. The job alone takes the same time on main and 6.4.0.
• 6.3.0 fails the Android app scan. The second fold has its error message.
• The box ran faster for this pass than for the first table (6.4.0 tool list: 1.75 s there, 1.38 s here). So main saves 0.18 s here, not 0.28 s. I do not know why.

What changed between releases

Newest first. Time to the tool list, median.

Step	Node	Before	After	Change
6.4.0 → main	22	2.16 s	1.78 s	-17.8%
6.4.0 → main	24	1.75 s	1.47 s	-16.0%
6.4.0 → main	26	1.55 s	1.30 s	-16.5%
6.3.0 → 6.4.0	22	1.90 s	2.16 s	slower
6.3.0 → 6.4.0	24	1.59 s	1.75 s	slower
6.3.0 → 6.4.0	26	1.38 s	1.55 s	slower

• main (13e03e9): it has four commits that 6.4.0 lacks (first fold). perf(mcp): keep Playwright and type-only imports off MCP startup #1691 says it speeds up startup, but I did not time it alone. The first browser capture takes the saving back.
• 6.4.0: it lists 31 more tools than 6.3.0 (170, not 139). 6.3.0 fails the Android app scan, and 6.4.0 passes it.

Not run

Startup timing (the first table)

• macOS and Windows.
• A cold disk cache. Every timed run follows a warm-up run, so the files are already cached.
• The npx launch path. I started each server as node plus the package script, so the Node version is certain.
• Other Node versions.
• Any run with a client config, or with Ghidra, Hopper or IDA set up.

First real job (Node 24 only)

• Node 22 and Node 26.
• A second capture in the same session.
• Newer main. Main has moved on since 13e03e9, and I did not time the newer commits.
• macOS, Windows, a cold disk cache and the npx launch path, as above.
• The whole Stretchly app folder. Its reply was over the 10 MiB limit, so I used one file and raised the limit.
• A direct test of what Ghidra keeps between runs. The five timed runs sat within 0.4 s of each other.

The four commits main has that 6.4.0 lacks

• 0ecf3f2 (perf(mcp): keep Playwright and type-only imports off MCP startup #1691)
• 57c5aca
• 4353089 (style(website): kbd, install-node width and nested collapsible indentation #1699)
• 13e03e9 (perf(evidence): fuse streamed JSON token handling #1694)

#1691 says it should make MCP startup faster. Its own numbers: client.connect went from about 490 ms to about 400 ms. My numbers fit that claim, but I did not time it alone.

More on the first real job

• Script analysis: the job alone takes 2.46 s on main and 2.51 s on 6.4.0. Android app scan: 0.64 s and 0.66 s. Ghidra overview: 6.36 s and 6.30 s.
• The browser result fits what #1691 says: it moves the Playwright load from startup to the first capture. I did not time a second capture in the same session.
• 6.3.0 fails the Android app scan. Its reply says "Artifact path collision: res/2f.xml differs only in case from res/2F.xml". Both 6.4.0 and main read the same app file and succeed.
• 6.4.0 is slower than 6.3.0 on the script analysis. The job alone took 2.51 s against 2.10 s. The reverse-order pass shows the same gap: 2.37 s against 2.00 s.
• Each run had its own fresh home and temp folder. Nothing was left in them after the run.

Other test jobs ran on this box while I timed. The load average was 3.0 to 7.4 in the main pass and 0.8 to 4.8 in the check pass. The load average counts programs running or waiting for a CPU, over 1 minute. The box has 24 threads. The tool list times here are lower than in the first table (6.4.0 was 1.75 s there, 1.38 s here). The earlier timer gives 1.36 s for 6.4.0 and 1.17 s for main on this box now, at a load of 0.3 to 0.5. So the two timers agree, and the box ran faster that evening. I do not know why, and I did not test it. Main saves 0.18 s here, not 0.28 s. Every row has 5 timed runs, all ok except the failed 6.3.0 scan.

How to rerun

Install the target, then run the timer. The timer starts the server and sends the three requests. It does 1 warm-up and 5 timed runs. It prints one line of numbers in ms: init median, min and max, then tool list median, min and max. After those come the tool count and the Node version.

mkdir -p /tmp/rea-bench/6.4.0
npm install --no-save --prefix /tmp/rea-bench/6.4.0 rea-agents@6.4.0
/path/to/node mcp-time.mjs /tmp/rea-bench/6.4.0/node_modules/rea-agents/scripts/rea.mjs /tmp/rea-bench

For main, build a plain copy of the commit first. I used Node 24 for the build.

mkdir -p /tmp/rea-bench/main-src
git -C /path/to/rea-clone archive 13e03e943 | tar -x -C /tmp/rea-bench/main-src
cd /tmp/rea-bench/main-src && HUSKY=0 npm ci && npm run build
/path/to/node mcp-time.mjs /tmp/rea-bench/main-src/scripts/rea.mjs /tmp/rea-bench

My wrapper scripts check the load before each row and add one row to a CSV file. They start only if the load is 2.0 or less. The load average counts programs that are running or waiting for a CPU, averaged over 1 minute. This box has 24 threads.

The timer, mcp-time.mjs:

// Usage: <node> mcp-time.mjs <rea.mjs path> <work dir>
// Runs rea mcp 6 times (first is a warm-up), prints one line:
// init_median,init_min,init_max,tools_median,tools_min,tools_max,tool_count,node_version  (ms)
import { spawn } from "node:child_process";
import { mkdirSync, rmSync } from "node:fs";
import path from "node:path";

const [bin, work] = process.argv.slice(2);
const RUNS = 5;
const ms = (a, b) => Number(b - a) / 1e6;
const fail = (m) => { console.error(m); process.exit(1); };

async function once(n) {
  const home = path.join(work, "home-" + n);
  rmSync(home, { recursive: true, force: true });
  mkdirSync(home, { recursive: true });
  const env = { HOME: home, PATH: path.dirname(process.execPath) + ":/usr/bin:/bin", TERM: "dumb" };
  const t0 = process.hrtime.bigint();
  const child = spawn(process.execPath, [bin, "mcp"], { stdio: ["pipe", "pipe", "pipe"], env });
  let buf = "", err = "", seq = 1;
  const waiting = new Map();
  child.stderr.on("data", (d) => (err += d));
  child.stdout.on("data", (d) => {
    buf += d;
    let i;
    while ((i = buf.indexOf("\n")) >= 0) {
      const line = buf.slice(0, i); buf = buf.slice(i + 1);
      let m; try { m = JSON.parse(line); } catch { continue; }
      if (m.id !== undefined && waiting.has(m.id)) { waiting.get(m.id)({ m, t: process.hrtime.bigint() }); waiting.delete(m.id); }
    }
  });
  const exited = new Promise((r) => child.on("exit", r));
  const call = (method, params) => new Promise((res, rej) => {
    const id = seq++;
    const timer = setTimeout(() => rej(new Error("timeout on " + method + "\n" + err.slice(0, 500))), 60000);
    waiting.set(id, (x) => { clearTimeout(timer); res(x); });
    child.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
  });
  try {
    const init = await call("initialize", { protocolVersion: "2025-06-18", capabilities: {}, clientInfo: { name: "bench", version: "0" } });
    child.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
    const list = await call("tools/list", {});
    if (!list.m.result?.tools) throw new Error("no tools in reply: " + JSON.stringify(list.m).slice(0, 300));
    if (list.m.result.nextCursor) throw new Error("tools/list is paged (nextCursor set); count would be partial");
    return { init: ms(t0, init.t), tools: ms(t0, list.t), count: list.m.result.tools.length };
  } finally {
    child.stdin.end();
    const t1 = setTimeout(() => child.kill("SIGTERM"), 5000);
    const t2 = setTimeout(() => child.kill("SIGKILL"), 8000);
    await exited; clearTimeout(t1); clearTimeout(t2);
  }
}

const med = (a) => [...a].sort((x, y) => x - y)[Math.floor(a.length / 2)];
const f = (x) => x.toFixed(1);
try {
  await once(0); // warm-up, dropped
  const r = [];
  for (let i = 1; i <= RUNS; i++) r.push(await once(i));
  if (new Set(r.map((x) => x.count)).size !== 1) fail("tool count changed between runs");
  const I = r.map((x) => x.init), T = r.map((x) => x.tools);
  console.log([f(med(I)), f(Math.min(...I)), f(Math.max(...I)), f(med(T)), f(Math.min(...T)), f(Math.max(...T)), r[0].count, process.version].join(","));
} catch (e) { fail(String(e)); }

Inputs and rerun

Each run starts the server, sends the first two requests, then makes the job's call. The server is closed after each run. Each run gets a new empty home and temp folder. Each row has 1 warm-up run and 5 timed runs. The Ghidra row took about 8 s per run, so 5 was fine.

• Browser capture: a page of my own, with a heading, a button and one script. A local python3 -m http.server on 127.0.0.1 serves it. The call is capture_browser_scenario. It opens the page and clicks the button. The browser is headless Chromium from Fedora. The license is BSD-3-Clause, among others. The version is 154.0.8037.57. No sandbox flag was needed.
• Script analysis: analyze_javascript_application on a folder with one file. The file is app/main.js from Stretchly 1.22.1 (BSD-2-Clause). I set REA_MCP_MAX_RESPONSE_BYTES=268435456 on all targets.
• Android app scan: open_binary, then inspect_artifact, on Coffee 2.27 (Apache-2.0). The tool needs an open app, so the time runs through the second reply.
• Ghidra overview: open_binary, then binary_overview, on an ELF built from my own 3-line C file with cc -O0 -g0. Ghidra 12.1.4 (Apache-2.0) ran on Eclipse Temurin JDK 21.0.12.1+1. Temurin is GPL-2.0-only WITH Classpath-exception-2.0.
• Targets: 6.3.0, 6.4.0, and main at 13e03e9 (perf(evidence): fuse streamed JSON token handling #1694), built with Node 24.

Inputs, checked with sha256sum -c before each run:

be92de991fcfcc5a94342ec41c909828fa75c6068dc2e0c4ee3117ee73041cbc  tiny-page/index.html
8811aed177d8bda35d9ef8ae7f2a5995763291461b9cd39d4f6dbda247a9b5f7  stretchly-1.22.1.tar.gz
51aad8aa87a96a929a6894ae377641ebdbb87d3baadb14dbc6451b86cda048fa  stretchly-1.22.1/app/main.js
ae4688fe48e751517e7a1d56aed27bff10b87daac0d7e28993fb193a7dc8f223  coffee_47.apk
83a9c9f99c3d98719c94cfab6f7ea9b2b783bed2ab328d562c2b01de01611437  t.c

To rerun all 12 rows (about 5 minutes):

MAIN_REF=13e03e943 MAX_LOAD=20 /path/to/bench/run-firstjob.sh

Test setup

• Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
• CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
• RAM: 32 GB DDR4-3600
• Node: 22.23.1, 24.18.0, 26.11.1
• 6.3.0 is the npm package. Its integrity starts with sha512-1l3L22PYqlzW.
• 6.4.0 is the npm package. Its integrity starts with sha512-Vj1zWfSlna5b.
• 6.4.0 comes from the release commit a56f7bb (chore(main): release rea-agents 6.4.0 #1399).
• Main: 13e03e9 (perf(evidence): fuse streamed JSON token handling #1694), built with Node 24
• Each run used a new empty home folder and a small fixed environment (HOME, PATH, TERM).
• Load before each row: 1.23 to 1.99 (the 1-minute load average, explained in How to rerun).
• The release commit a56f7bb is an ancestor of 0ecf3f2 (perf(mcp): keep Playwright and type-only imports off MCP startup #1691).
• So perf(mcp): keep Playwright and type-only imports off MCP startup #1691 came after the 6.4.0 release. Main has it, and 6.4.0 does not.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

[CassieFox23]

A side benchmark: REA's CLI feeding free local models. A script ran three REA calls on a small test program of my own. Then each model read the same output and answered the same three questions. More on the setup is in my note on #1958.

The models

Model	Made by	File	Context	Licence
KAT-Coder-V2.5-Dev	Kwaipilot	IQ4_XS GGUF	32,768	Apache-2.0
Qwen3.5-4B	Qwen	Q5_K_M GGUF	16,384	Apache-2.0
Qwen3.5-9B	Qwen	Q5_K_M GGUF	16,384	Apache-2.0
Qwen3.8-27B	Qwen	UD-IQ4_XS GGUF	32,768	Apache-2.0
Qwen3.8-27B, fast	Qwen	GSQ-RCO IQ3_XXS GGUF, with MTP	32,768	Apache-2.0

They run on llama.cpp (MIT) behind llama-swap (MIT), one at a time, on a Radeon RX 6700 XT with 12 GB.

REA calls (Ghidra 12.1.4)

Call	6.3.0	6.4.0
inspect	8.46 s	7.69 s
decompile main	8.10 s	7.80 s
strings search	7.97 s	7.85 s
Total	24.53 s	23.34 s

One run each on a quiet box, all exit 0, so the gap is within noise. In 6.4.0, decompile with --filter-output raw_result now wraps its output in YAML.

How well each model read it

3 asks per model, 3 questions each. The secret word is not in the file, so "partly" is the best possible answer.

Model	Best possible (of 9)	Asks with a wrong number	Made-up answers
Qwen3.8-27B	9	0 of 3	0
Qwen3.8-27B, fast	9	0 of 3	0
KAT-Coder-V2.5-Dev	9	2 of 3	0
Qwen3.5-9B	9	2 of 3	0
Qwen3.5-4B	5	3 of 3	1 ("2155")

Speed, memory and power

Model	Writing (tokens/s)	Prompt (tokens/s)	Seconds per ask	Load (s)	VRAM (MiB of 12,272)	GPU power, median / peak
Qwen3.5-4B	79.6	1,328	5.1 / 2.4 / 4.1	4.0	6,072 to 6,108	174 / 185 W
Qwen3.5-9B	50.0	761	10.1 / 9.3 / 6.6	7.6	9,813 to 9,873	185 / 186 W
KAT-Coder-V2.5-Dev	26.7	307	16.7 / 6.4 / 6.4	14.8	11,707 to 11,814	44 / 178 W
Qwen3.8-27B, fast	11.3	185	44.8 / 29.9 / 30.5	17.3	11,838 to 12,055	68 / 186 W
Qwen3.8-27B	7.9	171	48.0 / 35.2 / 26.1	14.6	11,906 to 12,089	50 / 186 W

The first ask reads the whole prompt, about 2,800 tokens. The next two reuse the cache. Load times may be low, because the files were likely still cached.

The first KAT-Coder run, on a busy box

	Quiet	Busy
1-minute load average	under 1	44 to 47
Writing (tokens/s)	26.7	4.4
Prompt (tokens/s)	307	181
Seconds per ask	16.7 / 6.4 / 6.4	62 / 34 / 32
REA call (s)	about 8	11.4 to 13.1

The load average counts programs running or waiting for a CPU, and this box has 24 threads. My code editor's file search had run away during the busy run. Same model, same prompt, so the busy CPU was the cause.

Not run yet

• More asks, with a fixed seed.
• REA 6.4.0's output in the prompt. The models read 6.3.0's.
• Qwen3.5-4B on a fully quiet box. It started at a load of 2.29, and the others under 2.

Edit: on 12 October I reran on a quiet box and added four more models. The busy-box numbers are in their own table.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊