Worker Pool

Worker Pool — Parallel Parsing

Parsing is CPU-bound (tree-sitter compiles native C++ parsers). To maximize throughput, codebase-vis spawns a pool of child processes via fork() to parse files in parallel.

Architecture

WorkerPool(size = CPU count - 1)
    │
    ├── worker[0] (fork → parse-worker.js)
    ├── worker[1] (fork → parse-worker.js)
    ├── ...
    └── worker[n] (fork → parse-worker.js)

How It Works

The WorkerPool class manages a pool of forked child processes:

class WorkerPool {
  #size       // Number of workers
  #free       // Array of idle worker processes
  #queue      // Array of pending tasks
  #pending    // Map of active worker → { resolve, reject }
}

Lifecycle

1. Initialization — new WorkerPool(size) spawns n workers, each running parse-worker.js
2. Task submission — pool.run(file) enqueues the file path and returns a Promise
3. Draining — #drain() assigns queued tasks to free workers via worker.send(file)
4. Processing — Each worker loads its own tree-sitter parsers, parses the file, and sends results back via process.send()
5. Collection — The main process captures results at the correct index (results[i] = result) to preserve input order regardless of completion order
6. Termination — pool.terminate() sends SIGTERM to all workers and clears state

IPC Communication

Main Process                    Worker Process
     │                              │
     ├── worker.send(file) ────────►│
     │                              ├── parseFile(file)
     │                              │   (loads own parsers)
     │                              ├── process.send({ id, dependencies, entities })
     │◄─────────────────────────────┤
     │                              │
     ├── (capture result at index)  │
     ├── #drain() ─── next task ───►│

Crash Recovery

If a worker exits with non-zero code or throws an error:

1. The pool rejects the pending promise for that worker's current task
2. The failed worker is removed from #free
3. A replacement worker is spawned via #addWorker()
4. Any unprocessed tasks in the queue are re-dispatched

This prevents a single corrupted file from stalling the entire parse.

Configuration

Setting	Default	Description
Pool size	CPU count - 1	e.g., 7 workers on an 8-core machine
--jobs flag	CPU - 1	CLI override for worker count

Worker Script (parse-worker.js)

Each worker is a self-contained Node.js process that:

1. Mirrors the GRAMMAR_MAP from parser/index.js to know which parsers to load
2. Maintains its own parserCache per-process (tree-sitter grammars are cached per-process)
3. Listens for process.on('message') — receives file paths from the parent
4. Parses the file and sends the result back via process.send()
5. Handles errors gracefully by returning { id, error: true } instead of crashing