perf: parallel file indexing with per-thread arena allocators

[justrach]

Design from justrach/turbodb — adapted from src/parallel_index.zig per-thread arena pattern.

Problem

File indexing is single-threaded. On openclaw (4,950 files), indexing takes 42 seconds. The shared GPA allocator would cause lock contention if we naively added threads.

Evidence

Head-to-head benchmark on openclaw:

• codedb: 42,000ms indexing
• Competitor: 16ms (only walks files, no content indexing — but our content indexing should still be parallelizable)

Proposed Solution

Spawn N worker threads, each with its own ArenaAllocator. Partition files into chunks statically. Each thread indexes its chunk independently, then merge results.

Reference: justrach/turbodb parallel_index.zig — static work partitioning with per-thread arenas, zero allocator contention.

Design

Main thread:
  1. Walk directory → collect file paths
  2. Split into N chunks (N = min(cpu_count, file_count))
  3. Spawn N threads, each with ArenaAllocator

Worker thread:
  1. For each file in chunk:
     - Read content
     - Parse outline
     - Extract trigrams into local HashMap
  2. Return local results

Main thread:
  3. Join all threads
  4. Merge local indexes into global index

Failing test

test "issue-206: parallel indexing produces same results as sequential" {
    // Index 100 files with 1 thread and 4 threads
    // Compare outline counts, trigram candidates, search results
}

Expected impact

Metric	Before	After (4 cores)
Indexing	42s	~5-10s
Allocator contention	N/A (single thread)	Zero (per-thread arenas)

Files to modify

• src/watcher.zig — initialScan
• src/explore.zig — indexFileInner
• src/index.zig — trigram insertion merge

[justrach]

Related issues

• feat: Reduce Memory Overhead - Efficient Initialization Techniques #197 — reduce memory overhead (parallel arenas prevent shared allocator bloat)
• Perf discussion on the index #129 — perf discussion on the index (parallel indexing was mentioned as future work)
• perf: trigram index v2 — integer postings, delta compression, query planner #142 — trigram v2 (batch-accumulate pattern is a precursor to parallel accumulation)

[justrach]

Resolved on release/v0.2.57. Parallel indexing was already implemented (4 workers); bumped default to min(cpu_count, 8).

Stable benchmarks (openclaw, 13,867 files, 3-run average):

Metric	v0.2.56	Now	Delta
Cold speed	6.4s	2.4s	-63%
Cold RSS	3,678MB	597MB	-84%
Warm speed	—	1.2s	—
Warm RSS	—	423MB	—
Recall	52	52	intact

Worker comparison (openclaw cold):

• 1 worker: 2.90s / 636MB
• 4 workers: 2.47s / 608MB
• 8 workers: 2.42s / 575MB

More workers = slightly faster AND less RSS (smaller per-worker arenas).