Parallel Architecture Roadmap

[gwokhou]

OpenKB Parallel Architecture Roadmap

This document defines the roadmap for introducing parallel ingestion without weakening OpenKB's mutation safety model.

It builds on the locking and atomic-write foundation from PR #86, the serial mutation recovery work in PR #142, and the current codebase. The immediate target is not broad concurrent openkb add; it is a small mutation architecture for openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID>, followed by a worker-safe prepare path for directory ingestion.

This Markdown file is the source of truth. After changing it, refresh the visual companion: parallel-architecture-roadmap.html

Core Model

Parallel work may prepare disposable data. Only the serial mutation owner may change official KB state.

Rollback-critical official KB state includes:

• .openkb/
• raw/
• wiki/
• registry/config files
• mutation journals
• PageIndex local state

Generated reports, operation logs, and chat session files are side effects unless a future PR explicitly makes them rollback-critical official state.

Worker code may create private staging output and return metadata. It must not publish, register, commit, or treat a prepared name as final.

The implementation path is intentionally narrow:

1. make the existing serial add paths crash-safe;
2. move those add paths behind a shared serial coordinator;
3. split worker-safe prepare from serial commit;
4. add bounded parallel prepare for directory ingestion.

Architecture Terms

These terms are used throughout the rest of the roadmap. The key distinction is between disposable prepare output and rollback-critical official KB state.

• KB lock: cooperative lock around .openkb/ingest.lock; exclusive for mutations and shared for reads.
• Recovery drain: rollback of active mutation journals before the first exclusive lock holder mutates state.
• Atomic writer: temp file, file fsync, os.replace, and parent-directory fsync where supported.
• Mutation snapshot: journaled backup of the official paths a mutation may touch. Missing paths are recorded so rollback can delete newly-created artifacts.
• Serial mutation owner: the only path allowed to publish official KB state, assign final names, update registry/config, mark commits, and run post-commit hooks.
• Mutation coordinator: shared orchestration for snapshot, publish, compile/write, registry update, commit, rollback, and cleanup.
• Commit point: the point after which recovery must not roll back official KB state. Current add paths use snapshot.mark_committed() after registry update.
• Prepare worker: bounded worker that inspects inputs and writes private staging output only.
• PreparedDocument: worker handoff object with private staging paths, source identity, conversion result metadata, and commit inputs. Handoff data may be exposed directly or through a nested conversion result.
• Deterministic commit queue: serial commit loop that commits PreparedDocument objects in input order, even when preparation finishes out of order. It may be implemented as a streaming queue or as collect-all-then-commit.

Roadmap

The roadmap is intentionally staged: first make serial mutation recoverable, then make ownership explicit, then introduce worker prepare without letting workers write official state.

Review Checkpoints

• Stage 3: coordinator extraction must preserve existing add/cloud-import behavior and keep non-add workflows out.
• Stage 4: prepare workers must stop acquiring kb_ingest_lock() and must not write official .openkb/, raw/, or wiki/ state. Private .openkb/staging/prepare output is allowed.
• Stage 5: --jobs is acceptable only after commit order and final names are deterministic.
• Adoption Notes: review as migration contracts, not as required work for the next PR.

Stage	Status	Outcome	Scope
Stage 1: cooperative locks and atomic state	Done	Shared KB files have cooperative read/write locking and atomic core writes.	Existing CLI mutation/read boundaries, config, registry, and watch-triggered openkb add. Query/chat/session files remain side effects until explicitly promoted.
Stage 2: serial crash-safe openkb add paths	In review	Interrupted add/cloud-import commands leave either the previous stable KB or a recoverable journal.	openkb add <file>, openkb add --from-pageindex-cloud <DOC_ID>, local/cloud PageIndex artifacts, malformed journal handling.
Stage 3: add mutation coordinator	Next	Existing add paths share one serial transaction owner with explicit touched paths, commit point, rollback, and post-commit hooks.	Existing local-file add and PageIndex Cloud import only.
Stage 4: prepare/commit split	Planned	Worker-safe prepare is separated from serial commit.	Directory openkb add <dir>, URL/download prepare, local PDF extraction or private PageIndex candidate work, cloud fetch/name candidate, metadata extraction.
Stage 5: bounded parallel prepare	Planned	openkb add <dir> --jobs N can prepare in parallel and commit deterministically.	Directory ingestion and watch bursts; optional future batch PageIndex preparation.
Later: speculative compute	Later	Expensive LLM/PageIndex candidate work can run early only when outputs are validated before commit.	Candidate summaries, concepts/entities, link suggestions, cache warming.

Current Implementation

This section separates what is already implemented from what still blocks parallel prepare.

Area	Status	Code reality
Cooperative locking	Done	openkb.locks.kb_lock() provides shared read and exclusive mutation locks. It supports same-thread nesting and rejects read-to-write upgrade. watch itself is unlocked, but triggered adds go through add_single_file() and take the exclusive lock.
Atomic persistence	Done	Config, registry, init seed files, and many compiler/lint wiki rewrites use temp+replace helpers. Logs and some reports remain direct writes or append-only effects, so they stay outside rollback-critical commit state.
Recovery journal	In review / present locally	openkb.mutation snapshots touched paths, writes active/committed/rolled-back journals, rolls back interrupted mutations, handles malformed journals, and bounds repeated rollback failures.
Staged publish	In review / present locally	openkb add <file> can convert into .openkb/staging/add-*, then publish staged raw/ and wiki/sources/ artifacts by atomic rename with EXDEV copy fallback.
Hardlink-backed rollback	In review / present locally	Large directories can be hardlinked and restored by inode diff. This is safe only while writers in those trees use atomic temp+replace or append-only behavior.
Parallel openkb add workers	Not implemented	There is no ThreadPoolExecutor, --jobs, batch doc-name reservation, or parallel prepare queue. convert_document() still takes the KB mutation lock.

Stage 2 Baseline

PR #142 establishes the baseline that later parallelism must preserve:

Concern	Current design	Implication for later parallelism
Recovery timing	kb_ingest_lock() drains active journals on first exclusive acquisition.	Recovery is command-agnostic. Future mutations start with a clean base state.
Snapshot scope	openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID> snapshot hashes.json, PageIndex files/db paths, doc summary/source/images, concepts/entities, index, log, and final raw/ and wiki/sources/ artifacts.	Future workflows need explicit touched-path declarations instead of broad directory copies.
Publish semantics	Staged raw/ and wiki/sources/ files are atomically renamed into final locations; cross-device publish falls back to durable copy. PageIndex JSON/summary artifacts are written inside the snapshot window but are not yet written through the same atomic helper.	Worker output should publish without byte-copying on the normal same-filesystem path. PageIndex artifact writes should either move through atomic helpers or remain explicitly covered by snapshot rollback.
Hardlink snapshots	Large append-only or atomic-write trees can be hardlinked and restored by inode diff.	Any new writer into wiki/concepts, wiki/entities, or .openkb/files must use atomic replace or preserve append-only semantics.
Commit semantics	Registry update precedes mark_committed(). Logs and journal cleanup happen afterward as best-effort hooks.	Future coordinators must make the commit point explicit and keep non-critical effects post-commit.
openkb add --from-pageindex-cloud <DOC_ID>	prepare_cloud_import() fetches cloud data and resolves a name without writing; _write_long_doc_artifacts() writes after snapshot.	This command already separates fetch from write, but still executes serially.

Stage 3: Add Mutation Coordinator

Stage 3 is the next reviewable slice. It extracts the current openkb add mutation lifecycle into one serial coordinator without adding concurrency or migrating unrelated writers.

Stage 3 review contract:

• route the existing openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID> mutation lifecycles through the coordinator without changing external behavior;
• expose a touched-path declaration format for add-like workflows;
• make the commit point, rollback boundary, and post-commit hook boundary visible in code;
• leave remove, recompile, lint --fix, reports/logs, chat session files, and chat slash-command migrations out of the first coordinator PR;
• add tests proving that interrupted openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID> behavior is unchanged from Stage 2.

Do not introduce openkb add <dir> --jobs N, worker pools, final name reservation, or broad workflow rewrites in Stage 3. Stage 3 makes mutation ownership explicit before parallel prepare exists.

Core add workflow	Current state	Stage 3 target
openkb add <file>	Uses staging, snapshot, publish, compile, registry update, committed journal.	Move orchestration into a coordinator while preserving current behavior.
openkb add --from-pageindex-cloud <DOC_ID>	Separates cloud fetch from artifact write; snapshots before writing.	Treat cloud fetch as prepare and artifact write/register as coordinated commit.

The coordinator should own this lifecycle:

1. Confirm recovery drain has already completed before the mutation body runs.
2. Assign final doc names and paths under the serial owner.
3. Create the touched-path snapshot.
4. Publish staged artifacts.
5. Run the compile/index/write phase.
6. Update registry/config state.
7. Call mark_committed().
8. Roll back on failure before commit.
9. Run post-commit best-effort hooks.

Stage 4: Prepare/Commit Split

convert_document() cannot run in a prepare worker as-is: it acquires kb_ingest_lock() and publishes official artifacts. Split it before adding openkb add <dir> --jobs N. The worker-safe conversion path may perform snapshot-free read-only config or registry lookups, but it must keep all produced artifacts private until serial commit.

Prepare Worker Boundary

Prepare workers may:

• hash content;
• read config or registry data for explicitly allowed snapshot-free checks;
• inspect input type;
• convert into private staging under .openkb/staging/prepare;
• extract images;
• produce local PDF/PageIndex candidates only when no official KB paths are touched;
• return metadata and staging paths.

Prepare workers must not:

• acquire the KB mutation lock themselves;
• write official raw/, wiki/, or .openkb/ state, except for private .openkb/staging/prepare output;
• update registry/config state;
• write journals;
• assume the final doc_name.

Serial Commit Boundary

The serial commit path may:

• resolve final names against the current registry;
• snapshot touched official KB paths;
• publish staged artifacts;
• index/compile or validate candidates;
• update registry state;
• call mark_committed().

The serial commit path must not:

• commit workers out of order;
• let workers decide official KB state.

PreparedDocument should contain only handoff data:

• source path or source identity;
• content hash and original filename, either directly or through a nested conversion result and source path;
• private staging directory;
• detected document type and long-doc decision;
• candidate source/raw artifact locations inside staging;
• optional PageIndex/cloud metadata fetched without official writes;
• cleanup helper or method for failed or skipped prepare output.

Stage 5: Bounded Parallel Prepare

Once prepare has no official KB side effects, directory openkb add <dir> can use bounded workers. The initial batch implementation may keep one outer exclusive ingest lock across prepare and commit while ensuring worker functions do not acquire the lock themselves.

1. The main process enumerates inputs in deterministic order.
2. Workers prepare into private staging directories.
3. Completed preparations are collected for a serial commit loop ordered by input order, not completion order.
4. The serial owner resolves final names at commit time.
5. Failed/skipped preparations discard staging without touching official KB state.

This is the first stage that should introduce user-visible parallelism such as openkb add <dir> --jobs N.

The ordering contract is testable:

• input enumeration order is canonical for a batch;
• prepare completion order must not affect final commit order;
• failed or skipped inputs do not let later inputs claim names earlier than they would in the serial path;
• final names are resolved under the serial owner against the registry state at commit time;
• an openkb add <dir> --jobs 1 run and a bounded parallel run should produce the same official KB state for the same successful inputs.

Adoption Notes

These workflows can adopt the coordinator after the add path is stable. They are specified here to avoid later ambiguity, but they are not part of the first coordinator PR. Read this as future design, not current scope.

Adoption rule: a workflow should adopt the coordinator only when it mutates rollback-critical state or needs all-old/all-new recovery. Logs, reports, and UI/session files should stay post-commit side effects unless correctness depends on them.

Workflow Adoption

Workflow	Adopt when	Boundary	Commit point
remove	After the add coordinator is stable and PageIndex cleanup order is pinned down.	Resolve the target document and removal plan before mutation; delete generated artifacts, prune concepts/entities/index, clean scoped links, and clean PageIndex local state inside one coordinated mutation.	Registry removal by hash. Everything before this must be retryable while the registry entry still preserves doc_id and paths.
recompile	When recompilation needs all-old/all-new recovery for generated wiki output.	Resolve target docs first; for each affected doc, snapshot generated outputs and rerun compile from existing wiki/sources/ or wiki/summaries/.	Successful rewrite of all generated outputs selected for that doc. Registry identity does not normally define the commit point.
lint --fix	Only when a fix mode touches multiple rollback-critical wiki paths or becomes part of another coordinated mutation.	Build the valid-target index before mutation; rewrite only selected wiki pages under the coordinator.	Successful completion of scoped rewrites.
Reports and logs	Only if a report becomes rollback-critical output. Most log/report writes should remain post-commit hooks.	Reports that are official outputs can be written through atomic helpers under the mutation lock; operation logs should run after commit.	For official reports, atomic report write. For logs, no commit point: they are best-effort post-commit effects.
chat	Only for slash commands that mutate rollback-critical KB state.	/add should call the coordinated add path. Future mutating slash commands should declare the same touched paths as their CLI equivalents. Chat session JSON remains separate UI/session state by default.	Same as the underlying coordinated mutation. Chat session persistence has no KB commit point unless promoted to official state.
watch	When watch should benefit from the coordinated add implementation.	The watcher only detects files and calls the add path. Each detected file is its own coordinated add unless a future batch mode explicitly groups them.	Same as openkb add <file> for each triggered file.

Rollback Surface And Exclusions

Workflow	Snapshot surface	Do not
remove	hashes.json, doc raw/ artifact unless --keep-raw, wiki/sources/, wiki/summaries/, per-doc images, affected concepts/entities, wiki/index.md, PageIndex local DB/files for local long PDFs.	Do not contact PageIndex Cloud for pageindex_cloud docs. Do not make operation log append part of rollback.
recompile	Affected summaries, concepts, entities, wiki/index.md, and optional wiki/AGENTS.md backup/refresh output when --refresh-schema is active.	Do not re-run PageIndex, re-convert raw files, or change registry identity.
lint --fix	Pages selected by restrict_to, or all eligible wiki pages for explicit global openkb lint --fix.	Do not snapshot reports, sources, or unrelated pages for scoped fixes. Do not silently sweep unrelated dangling links during another command.
Reports and logs	Official report path if promoted to rollback-critical output. No snapshot for normal operation logs.	Do not let a failed log append roll back a successful KB mutation. Do not add report/log handling to the add coordinator unless the output affects correctness.
chat	Same as the underlying command for mutating slash commands. No snapshot for normal chat session JSON.	Do not make every chat turn a KB transaction. Do not roll back chat history when a KB mutation rolls back unless chat sessions become official state.
watch	Same as openkb add <file>.	Do not give watch independent mutation semantics. Do not make file-system event batching change add commit order without an explicit batch contract.

Worker Boundary

Use this table when deciding whether code belongs in prepare or commit. If correctness depends on current KB state, keep the write under the serial owner.

State or action	Prepare worker reads	Prepare worker writes	Serial owner writes
Source inputs outside the KB	Yes	No	No, except normal source handling outside this roadmap.
Private .openkb/staging/prepare staging directory	Yes	Yes	Yes, for publish, cleanup, or discard.
.openkb/ registry/config/journals metadata	Read only for explicitly allowed snapshot-free lookup.	No	Yes.
Rollback-critical raw/ and wiki/ artifacts	Read only for explicitly safe inspection; correctness must not depend on stale worker reads.	No	Yes.
PageIndex local DB/state and .openkb/files	Avoid live worker reads by default; allow only proven read-only candidate extraction whose outputs stay private.	No	Yes, and affected paths must be in the rollback surface.
Generated reports, operation logs, and chat session files	No worker writes that affect correctness.	No	Outside the core add transaction unless explicitly promoted to rollback-critical state.
Final doc_name, registry identity, and commit decision	No final authority.	No	Yes.

PageIndex and LLM Boundaries

PageIndex and LLM work can be expensive, but cost alone is not enough to parallelize it. Candidate work is safe only when it stays private and can be discarded before commit.

Subsystem	Parallel-safe candidate work	Serial-only official work
PageIndex local PDFs	Page count, local extraction, candidate page/source artifacts in staging, metadata preparation, or PageIndex work against private worker storage.	Publish official KB raw/ and wiki/sources/ artifacts, mutate .openkb/files or PageIndex DB paths, register doc_id, include affected paths in rollback.
PageIndex Cloud	Fetch cloud tree/pages and resolve candidate metadata without writing.	Write summaries/source JSON/images, compile, register synthetic hash, commit journal.
LLM compile	Later: draft summaries, candidate concepts/entities, and candidate links when they can be discarded.	Validate against current KB, write accepted wiki pages atomically, update index/registry, commit journal.

Keep LLM speculative work out of the next parallelism PR unless invalidation is explicit. The safer near-term win is CPU/I/O prepare parallelism with serial official writes.

Stage Review Matrix

Use this as the default review checklist. The detailed failure and verification tables below remain the source for targeted test planning.

Stage	Review focus	Must not regress
Stage 2	Serial add/cloud-import remains crash-safe around publish, registry update, and mark_committed().	Recovery must not roll back committed state; malformed journals and EXDEV publish remain bounded and recoverable.
Stage 3	The coordinator owns touched paths, publish, registry update, commit, rollback, and post-commit hooks for existing add paths.	No behavior change for openkb add <file> or cloud import; no worker pools or non-add migrations.
Stage 4	Prepare produces only private staging and PreparedDocument handoff data.	No worker-acquired mutation lock, registry/config/journal write, or official raw//wiki//.openkb/ write.
Stage 5	Bounded workers prepare in parallel while the serial owner commits in canonical input order.	--jobs N must match serial official KB state for the same successful inputs; worker failure leaves no official artifacts.
Later	Speculative PageIndex/LLM candidates are validated against current KB state before commit.	Stale candidate work must be discarded or recomputed, not published.

Failure Model

Each stage owns specific failure cases and relies on earlier-stage guarantees.

Failure	Required behavior	Primary stage
Process exits before commit point	Recovery rolls back touched official KB paths to the previous stable state or leaves a bounded recoverable journal.	Stage 2
Process exits after commit point but before log/journal cleanup	Recovery must not roll back committed official KB state; cleanup/log work is best-effort.	Stage 2
Malformed or partially written journal	Recovery handles it deterministically, reports it, and avoids unbounded rollback retries.	Stage 2
EXDEV during staged publish	Publish falls back to durable copy without weakening rollback semantics.	Stage 2
Coordinator body raises before commit	Coordinator rolls back touched official KB paths and does not run post-commit hooks as committed work.	Stage 3
Prepare worker fails	Its private staging is discarded; no official KB state changes.	Stage 4
Prepare worker succeeds but becomes stale before commit	Serial commit validates or resolves final state again before publishing.	Stage 4
Multiple prepares finish out of order	Serial commit loop preserves canonical input order.	Stage 5
Speculative LLM/PageIndex candidate becomes stale	Candidate is discarded or recomputed; validation is required before it reaches official KB state.	Later

Verification Matrix

Verification should prove the boundary introduced by each stage, not retest every command in the project.

Stage	Must prove	Example checks
Stage 1	Locks and atomic writers protect shared KB files without unnecessarily blocking safe reads.	Concurrent command smoke tests; temp+replace persistence tests; lock nesting and upgrade rejection tests.
Stage 2	Serial openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID> paths are crash-safe and recoverable.	Kill before/after publish and before/after mark_committed(); malformed journal; EXDEV fallback; hardlink rollback with atomic writers.
Stage 3	Add coordinator preserves existing behavior while making lifecycle boundaries explicit.	Existing openkb add <file> and openkb add --from-pageindex-cloud <DOC_ID> tests unchanged; forced exception before commit rolls back; forced post-commit hook failure warns without rollback.
Stage 4	Prepare has no official KB side effects.	File-system assertions that prepare writes only private staging; no worker mutation lock acquisition in prepare; no registry/journal writes.
Stage 5	Parallel prepare produces deterministic official KB state.	Compare serial openkb add <dir> vs openkb add <dir> --jobs N output; ensure commit order follows input order; worker failure leaves no official KB artifacts.
Later	Speculative candidates cannot corrupt official KB state when assumptions go stale.	Invalidation tests for changed registry/content; validation before accepting generated concepts/entities/links.

Non-Goals

• Do not revive a broad concurrent add pipeline before the coordinator exists.
• Do not let workers write official KB state.
• Do not let workers update registry, config, journals, PageIndex DB files, or logs.
• Do not make prepare-time doc names final.
• Do not combine openkb add <dir> --jobs N with new transaction semantics in one large review.
• Do not use hardlink snapshots for a directory unless every writer into that directory is atomic replace or append-only.

Maintenance Notes

• Update this roadmap before updating the HTML companion or DOT graph.
• When PR feat: add serial crash-safe mutation recovery #142 merges, change Stage 2 from "In review" to "Done" and retarget Stage 3 as the next implementation slice.
• If the locking backend changes again, keep the description at the abstraction level: cooperative shared/exclusive KB locks, not a specific OS primitive.