performance: a measured time budget, and the levers ranked against it

[jasonssdev]

Why this exists

"OpenKOS feels slow" is the most consistent piece of user feedback, and it is
currently unactionable — there is no measurement saying where the time goes, so
every proposed speed-up is a guess. This issue records a first measurement and
ranks the levers against it, so the individual optimisations can be judged
rather than assumed.

Measured, 2026-08-14

Machine: Mac, 48 GB unified memory, qwen3:8b, bge-m3, Ollama local.

One call, 4 KB of Spanish source:

with thinking:      47.5s
--think=false:      11.5s
client CPU time:     0.13s   (0% CPU over 47s wall clock)

One transcript ingested: 4m 28s.

A full curate session: 5m 10s, broken down by its own stage timers:

Stage	Calls	Model time
Identity	5	16s
Structure	7	21s
Metadata	6	14s
Contradictions	10	39s
Total	28	90s

Two different problems, wearing the same complaint

Ingest is inference-bound. 4m 28s is essentially all model time — roughly a
dozen calls, each generating structured JSON. Wall clock tracks tokens generated.

Curate is interaction-bound. 90s of the 310s was the model. The other ~220s
was the operator answering per-item prompts. No amount of inference optimisation
touches that number.

They need different fixes, and conflating them has been the reason "make it
faster" never got anywhere.

What is already correct

• Thinking is disabled ("think": False in llm/ollama.py). Worth 4.1x on
  this model. The largest single saving is already in place.
• The client is not the bottleneck. 0.13s of CPU against 47s of wall clock
  settles the recurring "should this be Go or Rust" question: the process is
  idle 99.7% of the time. A rewrite would change nothing measurable.

Levers, ranked by expected payoff over effort

1. Generate less. Wall clock tracks generated tokens almost linearly, so the
cheapest speed-up is not producing text that gets thrown away. In one ingest,
13 fully-written objects were discarded by the judge and the language gate
against 9 kept. See the two-phase extraction proposal in the "candidates are
generated in full" issue.

2. Fewer, larger windows. Each chunk is one call. Doubling _CHUNK_TARGET
halves the call count for a long source. Whether extraction quality survives is
an open measurement — see the chunk-blindness issue, which proposes the same
change for a different reason.

3. Smaller models for mechanical tasks. Edge typing, contradiction judging
and adjudication are classification, not composition. The per-task models:
seam already exists and is unused for them. A 3B model on the 10-pair
contradiction stage is directly measurable against the 39s recorded above.

4. Concurrency — worth doing, but expect less than it sounds. Most calls in
a run are independent (per-chunk extraction, per-pair contradiction checks,
per-edge typing) and execute serially. But on a single local GPU the concurrent
requests share the same compute, so throughput gains are sub-linear — plausibly
1.3–1.6x, not 2x. And it is unavailable on constrained hardware: at the current
32K context the model occupies 10 GB, which leaves no room for a second slot on
a 16 GB machine. See the num_ctx issue.

5. OLLAMA_KEEP_ALIVE. Ollama unloads after 5 minutes idle (ollama ps
showed UNTIL 3 minutes from now), so any pause longer than that pays a full
reload. This is documentation, not code — but it is the cold start behind the
historical first-file judge failures, and users should be told.

What to measure before building any of it

• Tokens generated per extraction call, and what share belongs to candidates
  that are later discarded. That number decides whether lever 1 is worth the
  restructuring.
• Extraction quality at 4 KB vs 8 KB vs 12 KB windows, on the same corpus. That
  decides lever 2, and it is the same experiment the chunk-blindness issue asks
  for.
• The contradiction stage on a 3B model, scored against the current one. That
  decides lever 3 on its own, without touching anything else.

Each is a harness run, not a refactor. None of the levers should be built before
its measurement exists — the same discipline ADR-0001 applies to model choice.

The user-facing side

Separately from making it faster, users need to know what to expect. Rough
figures from this session, at ~3,000 characters per page:

Document	Pages	Calls	Time
Short note or minutes	up to 6	1	~30s
Meeting transcript	~15	~11	~4 min
Chapter or report	~30	~23	~8 min
Long document	~100	~75	~25 min

A table like this in the documentation is worth more to a new user than any of
the optimisations above, because it converts "this seems stuck" into "this is
what four minutes looks like".